It sounds like you are over-cleansing your skin at this point. Unless you are very young, you should probably not be using a scrub, especially if you do it frequently. Scrubs can be very harsh on skin, and cause a build-up of dry, damaged skin.
No, you don't have water mineral build-up on your skin.
The best cleansing routine I ever discovered, is to use first a creme cleanser, followed by a soap cleanser.
The oil in the creme cleanser first adheres to any build-up on your skin, and then the soap cleanser takes it off.
For two decades, I have been first applying Aubrey Seaware, found at health food stores, followed by a gentle liquid cleanser like Neutrogena. You don't have to over-scrub. Fifteen seconds of massaging in the Seaware, then I leave it on for a few minutes to adhere to the skin cells, then I shower and at the end of the shower I do a quick, light cleanse with the liquid cleanser.
The process of the oil cleanser followed by soap cleanser is the same idea as when you are trying to get rid of any build-up, such as with cradle cap. And it works.
And if you do a scrub, the one suggested below sounds good, primarily because there is oil in it. But you should scrub very lightly and briefly.