This is my philosophy on doctor's offices:
If the office staff is rude, mean, or look like they haven't had a decent night's sleep in five months (and act like it as well--a good indication that things aren't going well in the office, in other words), I will NOT stay with the doctor's office because it is an indicator about the state of the office, the nature of the doctor(s), and the type of people the doctors employ (assuming the doctor is okay). This may seem unfair, but I have noticed that if the staff isn't happy and customer friendly, I usually end up having serious trouble with either the doctors, the nurses, or someone else in the company (like billing). On the other hand, the offices with happy and professional office staffs always seem to run smoothly in other ways too: we don't have billing problems, the doctors and nurses are all good and very professional, etc.. Another point on this, you deal with the office staff almost as much as you deal with the doctor (if not more). Do you really want to put up with rudeness from them or the chance that they won't recognize a serious need if/when one arises?
As far as leaving, all you have to do is find a new doctor and have the new office send a record request to the new office. Believe me, this happens all the time and isn't anywhere near as meaningful to the doctor's office as it is to you. You don't have to offer ANY explanations whatsoever, and certainly don't have to feel bad about it.