It depends. There are some people that I know are always early or always late, so I plan accordingly. If it's someone I don't know, or it's behavior out of the ordinary for them, and they're SO EARLY that I'm still cleaning up my house and in the beginning stages of preparing food or getting dressed then yes that's rude.
If I'm at work and someone is 15 minutes early or less for an appointment and I'm ready for them, I'll take them. If they're more than 15 minutes early then it's very likely I'm not ready and I ask our receptionist to have them wait and offer them some fresh water or coffee. I've had appointments arrive more than an hour early, with them knowing they were over an hour early (one came 2.5 hours early), expecting me to take them immediately and then later complain that I was wasting their time by not being ready for them and not accommodating them even though I was in the middle of working on assignments for my boss or in the middle of other meetings.
I typically tell people at work to please try not to arrive more than 15 minutes early because I would hate for them to have to wait longer than that. :-) So yeah, I have no trouble telling people to wait for me if I'm not ready. I've made suggestions to take a nice walk or go visit whatever it is that's interesting down the street during the time they have to wait. One guy was so early I recommended that he walk the half mile to the movie theater and see a movie, then come back.
But I'm an employment specialist, and part of it for me is teaching people the life skills to figure out that there are different grades of "being early for an appointment" that are acceptable and they won't always be welcome or accommodated right away in spite of having one or more disabilities. I will repeatedly tell people that if they would like an earlier appointment to call me to see if I'll be available. That's part of skill training too... so I'm really coming at this differently than most people.
In real life, I've had little problem telling people, "I'm sorry we're just not ready. We're still getting dressed and have some things to do. We weren't expecting anyone for at least another 90 minutes or whoa... two hours! I know this is inconvenient for you but we still need more time. We'll see you later."