In my experience this is very common. Some babies do it, others don't. I have had problems more often with babies that are getting their bottles propped at home. Sometimes moms won't tell you they are doing this. If the child is used to being able to hold it in their mouth, sucking lazily when they want and falling asleep with it, that can be a problem for the provider that is trying to feed in another way.
Another problem I see with some of them is that they see and hear the other babies in the house and they want our attention. That can make some babies a bit competitive for attention. So when we hold them they are so busy playing and making eye contact that they don't want to concentrate on eating.
Also, sometimes it's a matter of timing. Maybe he just isn't hungry when you are trying to feed him. Try giving him attention without any bottle at all. Stretch the times out between bottles and see if he will eat better.
Whatever the problem is, this will pass :) He won't starve. But it will cause his sleeping and wake schedules to change for mom and dad.
I was going to suggest the daddy/mommy smell but you said you tried that.
Suzi