Staying in "cold" water

Hi there! I have fairly little body fat and find that it's hard to stay in the water (unless I'm in the Caribbean, or a too-much-heated pool) for any length of time, even when I'm moving around. I would like to know if this is a psychological or physical hurdle. In other words, I would LOVE to stay in water that feels very cold, so I can do long workouts, as long as I know it isn't hurting me. While I know that you'd have to actually be a doctor and know everything about me to give a 100% accurate answer, I value your opinion, and I'm 100% sure it's better than none. Wondering if anyone has ideas about this or any sort of charts that might help. Thanks!