I started sitting (can't really call it riding) on Belgian draft horses when I was 5. My first real lessons in 6th grade ( on much SMALLER horses!), and was working as an exercise rider on thoroughbreds in 7th grade.
Most ponies are mean (seriously, I think there's a 'deranged' gene in ponies), but connemaras seem to lack the Shetland mean streak or deranged anger. Arabians are another great small horse (often just barely over pony size), but tend to be temperamental / flighty. Quarter horses have more common sense than nearly every other breed combined (they're often used for lesson horses just because of their calm nature and common sense).
Why the mini breed review? (there are scores of breeds, many are GREAT for younger children, like Morgan's, etc. that I'm not listing out to save space). Because what age to start also depends on WHO you're/they're going to be riding.
A good barn will have a lot of different options, and a good instructor will match your kids personality with their horse.
Just for example: I hadn't ridden since I was 6, and those had been the huge old draft horses on my friends plantation. An administrative 'goof' put me in the advanced camp, so I was doing 3' jumps my first morning out. Ironically, that suited my personality/riding style. I got moved to where I belonged, but quickly became known as the girl who liked neurotic horses. I like a challenge. Quiet, gentle, and willing horses drove me a little bonkers. Every stable I've ridden or worked at since, I've fallen into the same niche. I'm happiest with problem horses.
My best friend in horse land was my polar opposite personality wise. Put her on a willing horse with a quiet temperament and they both SHONE.
We competed against each other all the time. I got highest marks on my knucklehead horses and she got highest marks on her quiet ones. Switch us off, though? And we both did okay, but not our best.
Sometimes barns don't have the 'right ' horse for a child. It's better to wait, or find a different barn in that case.
But with the right mount, and horsey parents? Age 3-5. I had to wait until I was 11, but only because there weren't good options for me.