Our first was a somewhat hard adjustment. It's the first time you're a parent and you learn so much. Having baby #2 was not harder AT ALL. It was actually easier than only having one, especially when my 2nd was old enough to play with my first.
Now, going from 2 to 3...that was crazy! They out numbered my arms, and there are some pretty funny stories of the craziness. Three kids was when the chaos entered. It has to do with #3's personality too. She is very spunky and keeps us on our toes.
Going to four kids hasn't been much harder than three, just busier. I'm guessing five kids will be challenging since my 4th and 5th will only be 15 months apart.
So...for me, it was DEFINITELY NOT harder to go to two kids. That was easy (if anything in parenthood can be described as easy). It was going to #3 that really got me. The chaos was just way different. I'd often find myself in what seemed like screaming everywhere, things flying past my head, children doing ninja backflips, and it was all a whirlwind of craziness, and all I could do was stop, look around, wonder how I got to this point in life (and if my hubby was there, we'd look at each other with the "look") and then laugh hysterically because it was so ridiculous. (so that's slightly exaggerated, and it wasn't always like that...mostly around bedtime or something when I was trying to get them all in bed).
But you adjust and it becomes a new normal:-) I wouldn't be afraid to go to two kids. They don't out number your arms yet! But you never know what you'll get. If you have a spunky child for your second, it'll keep you on your toes more. But, it's always worth it! Most people I know, they were worried like you when it came to #2, but they found two to be easier than just one...especially when they can play together.
ADDED: I wanted to add that my first child was a VERY difficult baby. She had a lot of health issues (open heart surgery and blood issues) and sleep issues (could not sleep) and was incredibly demanding and pretty much nothing about her was easy, but, of course, it was all worth it! This has worked out to be an advantage to us because no other baby has seemed even half as difficult as her.
I think that's an important thing to take into account. Each baby comes with their own personalities and are difficult/easy in their own ways. For us, baby #1 has made all the others seem like such easy babies. But the combo of personalities and then the age difference can factor in and are big determining factors of how easy/hard it is.