Hi R.,
My birthday is late August so I was always one of the youngest. I was also a first grade teacher (before kids). I vote to follow your instincts and send him to first grade.
I liked the advice you received about talking to the teacher more in depth. Perhaps bring a friend or family member to the meeting. Discuss your child's strengths in all modalities (social, emotional, academic, physical...). The friend/family member may provide less-emotional insight to you. Physical size should not be a factor in school readiness (unless it is your dream for him to be the starting line-backer on the varsity team as a freshman...) Some people are tall and some are short.
Reading is really a readiness issue. I had a first grader who could barely write his name on the first day of school. With no outside support (besides reading at home), he was reading at grade level by Christmas and above grade level by June. He is a great example of how quickly a child who is ready learns. My opinion: we can spend years using basic phonic programs to teach our children until it sticks, or we can use the same programs for a few months when the children are truly developmentally ready.
In the same class I had a boy who would have benefited from an extra year in kindergarten. He couldn't manage his body (only using potty during recess, fine motor skills, sitting relatively still during stories). By the way, he could read and write at grade level at the beginning of first grade. His physical immaturity hindered him during the year. Note that this is different than being "small."
(Random thought: there's a book "The Maths Gene" that explains our brains are wired to do numbers and math before letters and reading.)
Here's a link to the State Content Standards (by grade level):
http://www.cde.ca.gov/be/st/ss/index.asp
Good luck!