I have used them many times throughout my travels, including Hawaii, Puerto Rico, Miami, and even Lake Geneva, WI. In most cases, the hotels outsource sitting to local services. If you talk to your hotel concierge, they can provide you with a list of the services they recommend. Most of the time, it is up to you to contact the service directly and set up the service you want. They can also tell you about their screening processes, etc. Usually there is a 4 hour minimum required and rates vary but I would say average around $18/hour for one child, with additional charges for multiple children beyond that. You should plan to tip on top of that.
As far as set-up, YOU dictate what you are comfortable with. I have always required the sitter to stay in the room with the children. This has never posed a problem.
One thing to note, the sitter will not be allowed to disclose a personal cell number to you. If you want to check on your kids, you will have to call the room. The reason for this - if you needed that sitter again during your trip, you could now contact them directly and cut the agency out. The agency obviously does not want that. Just keep this in mind if you'll be out in the evening when the kids are sleeping. Calling the room may wake them up...try to feel comfortable with your sitting arrangement so you don't need to check in if this will create an issue for you.
I have never had a bad experience with any of the hotel sitters I've used and we have used at least 10 of them. On a long 10 day vacation, my hubby and I try to do one date night, so we have become familiar over the years with this service. I definitely recommend taking advantage of it if you can, but always trust your gut. I have told my husband every time, "if I get a bad vibe when the sitter arrives, we're not going.". He supports that, which means I always know I won't have to head out the door feeling uncomfortable.
Enjoy your trip!