We booked three nights at the Cadillac "Hotel" in Venice in 06. We stayed one. There were 4 people all aged around 40 in our party, including a couple of pretty large lads, and in half a dozen trips to the states we have never felt more unsafe. Venice was (at that time at least) FULL of homeless sorts, beggars (panhandlers) and associated weird types including one clown whose idea of humour was screaming at the top of his lungs at 2am on the boardwalk outside our hotel. Idiot.
We moved 5 mile up the road to the Holiday Inn in Santa Monica the next day and felt like Venice was a million miles away.
If you like pushing the envelope and seeing how the other half live (ie the nut jobs and whackos), then you'll have a great time at Venice. But if you want a NORMAL, enjoyable couple of days in that North Western part of LA, then spend a little extra and stay at Santa Monica. You wont regret it.