I'd be extremely uncomfortable being informed I'm doing such a thing for somebody's spouse, particularly when the demand comes with such specific requirements. Why not strike up another "conversation" with the guy that goes something like this:
"I hear you'd like me to bake an allergen-free cake for your wife. And no, I'm not comfortable doing that. I hope you find another source."
Friendly voice, no reasons, no excuses, no accusation, no complaining about not being sure you can make a safe cake, no annoyance at only three days' notice. If the coworker sounds shocked or angry, go a little deeper into your own needs, "I hear that you're surprised I won't make the cake. I'm surprised you would expect me to! I hope you find another source."
If you know this is a reasonable thing for you to do, you can do it, and stay calm and happy. If you buckle, you're laying yourself down like a rug for others to walk on with no appreciation.
Incidentally, as a former rug, I learned this process and it has been wonderful, empowering, and ever so grown-up. Good luck!