He's 11 months old.
I would REALLY go by HIS cues.... for what is or is not, what he is comfortable with.
Go with your baby's.... disposition.
Also, babies and children, do not play interactively... they do what is called "Parallel Play."
Even if it is only you and your wife doing it... a baby does not understand social nuances or what 'joking' is.
Their emotions and cognition... is not even fully developed yet. Not even a 2 year old has fully developed emotions, yet.
Your baby... per his disposition/personality... is probably just displaying his own impressions of things, at his young baby age. As he grows up... he may very well have.... other talents/dispositions/feelings... that will be a great asset. Each child, being different.
He is so young... and I just think that the 'horse-play' with its accompanying noise/facial expressions/gestures and movements... just scares him. To an undeveloped baby... or person.... this IS what horse-play looks like.
Also, 'joking' around... really varies, even among adults and per cultures.
He is fine.
He is probably just sensitive to 'vibes'... and some babies/kids, just do not like.... strong/loud/fast movements.
all the best,
Susan