I've gone to WDW when my son was 2, 3 and 4 1/2. (Also with a 5 mo on the last trip).
What wasn't worth it?
Nothing - I planned the trip with the child in mind so we:
1) Went early
2) "Rested" in the afternoon (usually a long lunch or nap in a stroller for at least 90 minutes) (when my wife an I would go get a snack or hit a ride solo).
3) Never stayed "late" (we made sure we were heading back to the hotel around bedtime or earlier). One night we stayed a little later and left as the fireworks at MK was starting.
4) Mainly focused on kid-friendly rides
5) Never used Rider Swap (just was too much of a pain sometimes to get everyone to the podium .. and we would rather ride stuff together even if it was a non-thrilling ride like the carousel or Aladdin's Carpets) ...
6) Purposely avoided "scary" rides (just knowing how sensitive or son is to scary images) .. like Haunted Mansion, Pirates of the Caribbean, heck he even didn't like Little Mermaid ride because of scary Ursula.
Honestly, going on the "small" rides were more memorable than the big ones simply because of the look of joy on my son's face.
So despite going three times since 2014, I haven't gone on Space Mountain, Big Thunder Mountain, Everest or Splash Mountain in a long time.
.. and that's okay!
ONE thing I can say was NOT worth it? Hollywood & Vine character meal.
We had great food and fun at Character Meals at Tusker House and Crystal Palace when he was 2, but when he turned 3, characters began to scare him. He loves the Disney Junior characters (and loved Disney Junior Live show) .. but going to a dinner where these larger-than-life characters with giant heads wanted to come up to your table? Meltdown city! And .. frankly .. the food wasn't really great for the cost. Luckily we were also there with his cousins who enjoyed it.
On this last trip (4 1/2) we started going on bigger rides (Test Track, Soaring, Star Tours) . .some he loved, some he did NOT like as it was too rough for his small body.
So almost everything was "worth" it in my opinion. You don't know how your child will like it until you try it. There wasn't anything they didn't like, just some things my particular child realized he doesn't want to experience again (anytime soon). Some things scared him (like the Ursula scenes in Little Mermaid) .. some things made him feel "funny" (like Star Tours).