Don't let your mind jumble and mis-interpret what they think... be confident in yourself, because YOU know what you do all day and YOU also know your priorities and limits. Don't worry about what they think... don't be offended by their suggestions and such. I'm in a similar situation... kinda of. I'm a painter. I've painted entire rooms as murals and also paintings on canvas. I'm a SAHM of four girls. The only time I've found actual time to do anything like that at all is after I put them to bed... but I'm exhausted by then, and also I need to be in my "artistic mood" as well to do painting jobs. There are so many unfinished paintings in my house. It is sad. I ALSO have really gotten into baking recently myself and doing really cool cakes (haven't tried the pops yet), THAT is actually my current "escape".... an excuse, actually. I find doing cakes therapeutic...
Anyways, I just wanna say that I GET what you are saying, but try not to read too far into it all (in reference to what your family is saying). This stage in your life will really go by SO fast, so just enjoy yourself to your best ability... use any free time (IF you have any) to write down some ideas for your plan with cakes when you DO get the time in a couple years or so... Pin things on pinterest, draw some ideas for cakes, and make a business plan.. do research, get some library books on starting a business.... ETC. I'm saying that because when you are a SAHM that additionally doesn't get much time to themselves, you can EASILY get discouraged about how the future looks for YOU and YOUR passions (is this something in the back of your mind? Is this partly bugging you?), so if you know you will eventually get there and are working towards it in SOME way... it can be a morale boost for sure!
I LOVE being a SAHM... I DO get discouraged a lot, though because I don't go out and do something solely for me or do something towards my passion... but I HAVE found that those little things that get you excited about what can be in store for you really make all the difference... not to mention the piece of mind you have daily that YOU are the one with your children all the time and YOU should feel happy and proud of that! :)