M.P.
It is natural that he's wanting and needing more attention. He's discovered that he gets immediate attention when he acts out. I suggest that you conveniently not notice some of the negative behavior. You can "not see" the shoving the magazine off the table. Or you can accept that throwing the magazine off the table is his way of expressing his anger at the way his family and his life have changed.
I would pick up the magazine, put it back on the table and say, I'm thinking that you're angry. It's OK to be angry. Then walk away. For me, throwing a magazine off the table is OK. The part I would pay attention to is his looking for a reaction. Not having a reaction tells him that negative behavior will not get attention.
At the same time I would be concerned about letting him know it's OK to feel angry. And I'd do that by voicing the feeling for him. At 23 months he doesn't know about his feelings or how to deal with them. It's important for you to accept all of his feelings, even the negative ones.
One way that you can help him is to notice how he's feeling before he acts on the feeling. Perhaps pick him up and say I think that you're feeling angry. It's OK. Mama loves you and give him a hug.
It's possible that you're overdoing the showering him with attention. It may feel a bit phony to him and so he has to keep testing. Yes, spend one on one time with him. Hold him when he wants to be held and it's convenient but don't hold him while you're feeling frustrated about having to do it. Be as honest with him about both his emotions and your own. It's OK to say, "I'd love to pick you up now but I have to change babiy's diapers now. I'll pick you up when I've finished. Be direct, calm, and sympathetic. When you pick him up tell him you know it's hard for him to have this baby in his home. Tell him you love him and then get him started on playing with one of his toys. He needs more than physical attention. He needs to be accepted, negative feelings included.
When he's being obnoxious and you can't deal it's fair to tell him that. Say, I just cannot deal with the way you're behaving right now. Please go to your room or across the room and play. Take him there, put on some soothing music, hand him a favorite toy or his "blankie" and leave. It's a time out of sorts but less punitive. It allows him to have the feeling but not the action.
One way to make this going to your room more successful is to show him that it's a good place in which to be. Also ake him there, put on the music, etc when he's feeling good. Do this especially when it seems that he's tired.
Being rested and feeling full are both important influences towards good behavior. Perhaps you've noticed that he's more cranky and more easily upset towards the end of the day or just before nap time or meals. Rather than sending him to a time out, sympathize with feeling tired and/or hungry. Give him a healthy snack, perhaps reduce noise and activity. Rock him with his "blankie."
I think it's also natural that he would prefer his dad at this time. You are the cause of this baby being here. You are the one who is giving the attention to the baby that he used to get. This will pass. You could try routinely having dad take care of the baby while you do something with the toddler.
This is a stage and will pass. It is also a great opportunity to teach your toddler about feelings and how to begin working on handling them. And it's an opportunity for you to learn about your own needs and how to provide for both yours and your toddlers. You do get frustrated and angry at the way your toddler behaves. Look for ways to stop the melt down before he get's into action. Notice your own frustration and voice it to your toddler in a calm manner. Find ways for both of you to stop the cycle before it gets to the anger stage.
For me stopping what I'm doing and giving my grandchild a hug noticeably lightens the atmosphere if I do it early on. When they were babies I often stopped what I was doing and rocked them or just sat down with them and read a story while everything was going OK. This helped me to remain calm and both of us to be happier.
In regards to the crying when you leave the room at bedtime, wait a few minutes and then return, pat his back, lay him back down but don't take him out of the crib. Reassure him that you're just outside and he's OK; then leave. Gradually make the length of time between leaving and returning longer. He'll get the idea that you will return and go back to self-comforting. Right now he is feeling very insecure. In addition to the new way life is he's also experiencing object inconstancy. He doesn't know that you're just in another room. For him, if he can't see the person or object, it doesn't exist. Learning that things and people exist even when we can't see them is the reason for the peek-a-boo game.
You're just in the first week of being home with the baby. It will get easier as you find ways to be and to act in this way of living that is new to all of you.