My grandfather had an eye damaged in an accident when he was in his mid-20's. When contact lenses first came on the market in the late 50's and early 60's, he was fitted with one contact for his bad eye and wore one contact lense for years.
I've known several people who have done the one contact for distance and one for reading and didn't have a problem with it. I tried it for an hour or so a few years ago at the doctor's office and couldn't wait to take them out. I had the same concern you do. I actually felt scrambled with one for distance and one for reading.
I think a lot of the problem for me had to do with the strength of my prescription. I've worn glasses or contacts since I was 10 years old. I also have some astigmatism in each eye. I'm not a good candidate for laser surgery because my eyes are dry.
If you don't feel comfortable with the correction you are in now, change it. You may get used to it, but you may not, too.