Here in Kansas it would be hit or miss and in VA it sounds like maybe it would be similar if not colder. Were you thinking afternoon or evening? Afternoon would certainly be warmer, but in the evening it would be fun too if you could roast hot dogs, marshmallows, etc. Do you have a fire pit or chiminea? Honestly, if you do a fall theme and have hot apple cider and other things then you probably can do it. But I agree to stress to the parents/party goers that it will be outside. As long as I knew ahead of time, I'd be fine with it.
I would however have a back up plan in case it's just too cold or rainy. You could always try using your garage for at least cake, presents, snacks??