Me, my brother and all three of my children were at DL or WDW at least once, if not multiple times, before the age of 4. Last year, our family went to DL and DD was a little over 3.5, and honestly, I think it was one of the best trips because everything was just so magical for her and she was so very excited. My philosophy, FWIW, is that I want my kids to, like me, simply have the memory of being and experiencing Disney from as far back as they can remember. And, last year, I had even tried to find a way to take my DD by herself.
But, the concerns you have are valid ones, not for the age cut off, but because of the personality dynamic. And, a rule of thumb I always follow if at all possible is to never go with 3 people only.
The situation you're describing may end up being not doable for you unless you had someone else with you. Your DD may not be old enough to go one rides by herself, and DS certainly isn't. Even if DD is old enough, you'd always have to be sitting with DS unless it's a ride you all three can sit next to each other on. An example, your family wouldn't be able to go on Autotopia at all. And, if there's rides your DD wants to go on, but can't by herself, she wouldn't be able to go at all, if there was no one to watch little brother.
So, in the situation you're describing, I'd either get someone else to go with me, have DH keep both kids and go with DD alone or make it a whole family trip. Edited to add: Or, you can come when we're there, 3/29-3/31 next year and we could go around together, DS too!
Then, we'd have three adults, a 3+ year old, a 5 y.o., an 8+ year old, a 10 y.o. and a 12 y.o. Every base you want would be covered!
Though, my "baby" is a daredevil, and happily went on Star Tours and Matterhorn and Soarin' (I'm anticipating space mountain and others this next trip!) . She did really love the Small World and Pooh and they all loved Buzz, too, so there'd be more than enough to do! AND, you and DD could come to Ariel's for dinner with DD and I and your DS and the other boys could go play somewhere else, if he didn't want to dine with the princesses!