K.M.
Absolutely. I think this sounds like a great party! I would clearly state on the invitations that a lifeguard is present and that additional parents would be there to supervise. I would also give the option of allowing parents to stay if they wanted to.
I think by 4th grade, kids understand the rules of pool behavior, but to be on the safe side, I would take a few minutes before they all jump in to review the rules- most hotels have them posted.
I agree with T F., have other activities on hand just in case. You could set up a craft table or some other project. Have fun!