i have the life experience with this issue. it sounds to me like he is becoming overly stressed about the shopping and dealing with all the sensory overload from sounds, smells, colors, etc.
i can only tell you what worked for me.
i took my son out on very short shopping trips when he was about 2. if i had a lot of shopping to do, i either did it in short spurts or i found someone to keep him so i could get everything done all at once.
in taking the short trips, i worked him up to handling long trips. it took time and a lot of patience.
as for church, see if you church has a crying room where you can go and sit with him when it gets to be too much for him, but you can still hear the sermon. if there is no crying room, see about a nursery where you can take him when he starts to get out of control. again it will take time and a lot of patience, but you can work him to handling a whole worship service. also let him take a few quiet toys with him to try and keep his attention a little longer.
as for the dining out, when we would go out, my son would take a few quiet toys and then when he had finished dinner and he would sit quietly under the table at my feet and play with his toys. this was acceptable and as time went on, he got to where he could handle the entire time out without needing to hide under the table and he would play quietly at the table.
ask his pedi for screening for sensory issues. you can also get in touch with the FDLRS program in your county - it is part of the school system for Pre-K age groups. they do screenings for free right about your son's age.