We have a pool and say the same to our close friends. We wish they would take us up on it more often...cuz we really mean it.
I have had a mom ask me if she could bring her boys over to work on swim strokes and I said "Heck yes...come on over but I won't be home solet yourselves in the yard and stay as long as you want." I even left cold ice water out for them and the house unlocked so they could use the restroom.
Another mom called and asked if she could bring her little one over after his nap because she bribed her son with swimming at our house if he took a nap. I emphatically said "Sure!!!" I laughed and thought to myself that I was soooo grateful I had a pool to help this mom in a desperate nap battle.
Personally we love to share our pool with others. I am grateful to others that shared their pools with us before we had ours. Yes it is rude of them to offer and then make you feel uncomfortable when you finally take them up on it. I personally would call them on it. If they say, they won't be home just say that you thought they had offered whether they were home or not.
Good luck and best wishes for getting your toes wet soon! If you were in my neck of the woods I'd let you come take a dip in mine...it is hot today!!