How cold without a wetsuit?

I started swimming about a year ago and have really fallen in love with it. I've only swam in open water and have no desire to go in a pool. I swam right through the winter in the ocaen here in New England in 37 degree F water with a triathlon wetsuit. The water is warming up, now it's about 47F. I was wondering at what temperature should I try swimming without a wetsuit? I was thinking that maybe I could try it when the water hits 50F. Anybody have any suggestions? I sort of have a goal in the back of my head to swim the English Channel someday.