There are always crowds for Spectro - even during the off season. We'll probably be there that night, too and even though it may be less crowded than other seasons, there will still be a lot of people coming that afternoon and evening to see the parades and WISHES.
My DH hates crowds. I've told him many times that he needs to expect crowds if he wants to see the fireworks, parades and special shows. That's just the way it is.
Don't let the prospect deter you from going anyway. Just adjust your expectations (example, you may have to claim a spot 45 - 60 minutes ahead of time if you want to watch from the curb on Main Street). Try not to be in a big hurry to leave - it's a lot less stressful to let the commandos get out and then leisurely walk to your bus stop or transportation to the TTC.
Best wishes -