After 1:00 in the morning, the subway shuts down in Toronto. (You might be able to catch a train in certain places by 1:30.) If you want to get around, you have to take the night buses (Vomit Comets) that show up every half hour, and only on certain main roads. (Yonge street being the exception. It has frequent buses in both directions.)
With VIA, the last train east from Toronto is around 10 or 11:00 pm. With GO, last train is 12:13 am.
Basically, unless you have a place in Toronto to crash or unless you have a driver... you're not partying.
Do you mention this because of it's reputation? or because there is nothing to do there?
I live closer to Keele and Finch, but I walk along the area all the time, and I have never been shot or mugged or raped by anyone, nor have I shot or mugged or raped anyone. I have been bored though, pretty much all the time.