If your description is accurate, this home is a health hazard. I suggest that your sister talk with the school nurse. At least in Portland Public Schools, the school nurse deals with this sort of issue. Each school has a nurse assigned to be at that school one day/week.
Another suggestion is to call Child Protective Services who can send a worker to visit the home and offer help in getting it cleaned up. In most cases the children are left in the home as long as the family is working to clean up the house to meet basic health standards.
I agree, that at 13, this boy knows he's dirty and wonder why he isn't able to clean himself up. Which then brings up some questions your sister can ask him. Do they have running water connected to a tub or shower? Do they have a washer? Is anyone in the home ill? Even what does he think about the way he looks?
Start out with a general statement of being concerned about him and how he feels about himself. Your nephew will know if the boy is accepted and how he's treated at school. This could be the starting point for the conversation.
She can ask compassionate questions about him and his family. Perhaps his mother is ill and cannot do housework. Perhaps there is no mom. Perhaps they are poor and do not have a hot water heater or washing machine.
If these things are true there are still ways for this family to be clean. Hopefully the family would appreciate some help. If these things are true and there is a mom, I suggest that she is very depressed and not able to find alternative ways to keep things clean.
I grew up during a time when it was fairly common for people to not have a hot water, heater, a tub or shower, or sometimes even water piped into the house. Families were still clean. This situation can be changed with the parent's co-operation.
In my experience most, but I do have to admit not all, CPS caseworkers were compassionate and approached this sort of situation tactfully. I saw several families in filthy homes be able to clean up with help from CSD.
If the boy is embarrassed by his condition, he may respond well, to your sister's help in finding ways for him to improve his personal hygiene. He may not know how to take care of himself if he's not had a good role model and help to do so at home.
I know that your question was specifically about how to deal with this boy staying overnight. I encourage your sister to take it further and help this boy improve his health, his self-confidence, and his acceptance by peers. Kudos to your nephew and your sister for befriending this apparently quite needy boy.