You can try getting vision tests. The one's with different colored dots. You may be able to load them from the internet and see if your son recognizes the images inside the test. These Pseudoisochromatic tests are really for ages between 3-7 but every child develops differently so you could see. I wouldn't take it as 100% only because of the age.
Other than that, kids concentrate on one thing at a time. They will get hooked on numbers or walking or a number of skills and all the coaxing in the world will not get him to what you are trying to concentrate on.
I really think you have another year before you can really "worry" but see if you can find a test on line that you can print or show the screen to your son and ask him if he sees the "star" or "kitty" or whatever is there. I wouldn't tell him what the objects were but rather ask him to point to the objects in the picture. Then ask him if he knows what they are. If he sees them then there is a good chance he is not color blind. He is just not ready to adventure into colors yet.
Another thought: When asked what car he wants or ball or whatever the object may be, is he able to say, "the red truck?" If he is able to tell you what color toy he wants, then he's not color blind. Ask him as you do this. Next time your at the store offer to buy him a hot wheels car. Do you want Daddy/Mommy to get you the green truck or the red truck. (those being two significant colors that are not destinguishable in color blindness) Then if you are buying a small match box car or hot wheels, ask him to go pick out the truck he just described. For 99 cents, you may have your answer. If he picks up the wrong color then probe a little bit. Do you like the color of that truck? What color is that? Then pick up the color he chose and say, "This one is the red one. Does this (the color he picked up instead) truck look like the same color as this one?) Not fool proof but could give you some insight. Again, I wouldn't take it as 100%. I would be more apt to trust the tests if I could find one.
The other thing you mentioned was that he is very smart. A lot of kids with color blindness will have problems with grouping when they get older due to not being able to see the color. Sometimes parents think it is a learning disability of some kind and it turns out to be color blindness. Which of course is a "disability" to some extent when learning grouping but far less serious than some of the other issues that it could be.
Bottom line: Pull a test up. It may put some of your "fears" to rest but if he "fails" it, I wouldn't bank on that 100% because he is only 2.
Hope this helps.