K.,
It really sounds to me like everything with your son is fine developmentally. I am a speech therapist and I primarily work with children, I also have a 23 month old little boy. My son still says a lot of probably what you consider baby talk, intelligible words, and words that only we understand. He actually just started adding words weekly to his vocabulary this past month. By 2 years old your son should be able to say at least 5-10 words like mama, dada, ball, etc. If your son still isn't saying many words by 2 1/2 and not putting 2 words together like "bye bye dada" then I would recommend looking into or consulting a speech therapist. But your son has over 1 year to develop more speech and language skills...right now I wouldn't worry everything sounds fine. Keep reading to him, talk about everything you see and do, and keep your own language simple when talking or labeling things (i.e. instead of saying, "Wow, that is such a big blue fish" just say "blue fish").
In regards to the binky, really if it provides him comfort and helps him to sleep I would let him keep it. There is really no harm in it especially since he is only allowed it for sleeping and not during the day. Most kids who use a binky to go to sleep usually end up spitting it out during their sleep anyway. I usually recommend to parents anytime after 2yrs old to may be start thinking about getting rid of it even up to 3 years of age is okay just as long as they are only using it at night. Also just make sure that he is using the correct size and not one that is for infants, you want to make sure that the nipple is big enough for his age, if it is too small it could affect some of his oral development. Also just to keep in mind you don't want to take the binky away too soon and then he could possibly develop a thumb/finger sucking behavior which is much more difficult to stop. At least with the binkies you can eventually throw them away, can't really throw away your thumb or fingers. :)
Good luck and if you have any other questions please let me know.
J.