We drove down Easter Tuesday. Could possibly be busier on a weekend.
Key West is very adult on Duval st at night. Transvestites trying to get you in their bars, places with women sat outside and a list of their offerings etc.
I find it difficult to say its not family friendly because everyone/every family is different. We have taken Mathew to Vegas, key West, over 10 times to Amsterdam etc, to us its part of life and something we have never hidden. When Matt was learning about drugs in school we took him to Amsterdam where he saw an addict having an argument with a tree, I know which experience would deter me from using drugs.
There is plenty to do in Key West but I can easily see why some people prefer not to take children.
Key west to Fort Myers is approx 5hrs 30mins & over 300 miles. Are you not stopping anywhere on the way? If your plan is straight there then around Fort Myers area, we chose north of Venice, by the time we got there we had had enough travelling that day but we stopped off in the Everglades and did some Geocaching, we even missed Marco Island due to lack of time.
Sanibel is just 5 mins further than Fort Myers, 6 miles further (and a $6 toll) but a totally different world.
It featured on the travel channel, clip is very short.
http://www.travelchannel.com/video/a-slice-of-blue-heaven-12107
Best (and most expensive) Key lime pie I have ever had. $9.50 + tax but it was delicious.