Assuming you mean at WDW. My 6 year old goes with me if we are alone. Always with the guys if they are along. This topic gets ugly fast so be prepared. We have been to the parks many times and we have seen guys older than 6 in the women's bathroom many times. A bigger issue is when you have to go and take the 3 year as well. Where will he be during that time? My suggestion at 6 is for you all to go together but be sure you son knows proper bathroom manners and stands where you tell him too and is not peeping through cracks in doors etc.
I took DS with me until he was probably 8. Or, I let him go in the mens but yelled out "I'm standing right outside, son, so I'll be right here when you're done". He was pretty embarrassed but (he said) less embarrassed that having to go in the women's restroom. It kind of depended on where we were, how many people were around, my gut feeling, etc, whether I let him go by himself.
Then, when DD and I had to go, I made him stand right outside the women's, and I'd yell out to him, "are you still there" every few minutes. Probably wouldn't have deterred anyone from grabbing him, but at least it got him yelling back. If the women's room had an entryway with 2 doors, he stood inside that entryway where he couldn't see anything but also couldn't be seen from outside.
Look for "Family Restrooms". They have them in the parks, and a lot of retail stores have them too.
If there are no family restrooms, you have to do what you and your child are comfortable doing. Don't do something you aren't comfortable with because of some other people's opinions.
Disney does not have family restrooms. They have companion restrooms that are only shown on the park map for disabled guests. Which makes them hard to find if. You are in a hurry.
I could have sworn they were family restrooms. Anyway, we found them pretty easily. We weren't even looking for them but were glad they were there. The ones our family used were right next to the regular restrooms.
They are officially called Companion Assisted Restrooms by Disney. However, anyone is free to use them, if you know where they are. Just try and keep in mind that they are few and far between, and the only restroom some guests CAN use, so only use them if you have to.
And please don't use them as a time out spot, or as a spot to change your child's clothing. Not meaning this as a scolding of the poster I quoted, just as a general comment.
This is a bit off topic, but related... I'm trying to figure out when to stop bringing my son with me into places like a changing room when I'm looking for clothes for me. He's only 15 months old right now, so I'm not worried about it for now, but when did you all stop bringing your no longer babies in with you to clothing changing rooms?