Is the crib sheet rough or hot or soft cotton? Is there a crinkly or non-breathable plastic covering on the crib mattress? Does the mattress smell (sometimes they can smell things we can't and crib mattresses are one thing that do pick up or give off weird smells that are hard to get rid of)
Is the room comfortable or is his bed under the vent or where lights from outside can hit his bed (I remember one place where I'd get woken up from traffic lights hitting the side of the window and bouncing into my face even though the blinds were shut. Bought a roll of fabric and completely covered that window!) Are there noises in that room more than others (or less)?
Is he cold, is he ... ?? Put yourself in his shoes and consider the feels, smells, sensations, texture, sounds, movement, etc. If he sleeps fine other places, but not this one place.... there's something there. Moving the crib (even if into your room maybe???) might be a compromise. And a way of sorting out if it is proximity to you, the room or the crib itself that is a problem.