We usually tell our family...they aren't the type to 'attach' themselves to someone else's vacation so it's okay to do that. For those that have 'moocher-family members'...wow, I feel sorry for you! Seriously.
We've done the Magical Gatherings at WDW and it worked out great. Of course, there were clear 'line/boundaries' drawn when it was time for "family only time." It's important to set guidelines if you want to enjoy family time when on vacation, esp if you travel with people that tend to take the lead.
I have a strong personality and am not afraid to voice my opinion so that's probably why we've never had an issue. I do, however, have friends that tend to be 'pushovers' and always have issues with this sort of thing. One has to stand up and be heard if they want things to go as they planned. JMO.