Help, losing body hair

Since I have a self-imposed ban on returning to the ESPY discussion and going insane battling Ion Beza, I have a new post. Warning - this post is odd but I have been contemplating asking the panel of experts for a few months now. Here's my problem (one among many), I'm losing my body hair. Does anyone else have this problem who swims a lot? I'm not a nutty swimmer, probably swimming around 7-8 hours per week. mostly indoors. But, I'm telling you, I am disturbed by what has happened to me. My wife is disturbed by the neighborhood women asking her if her husband shaves his legs and dyes his hair (neither of which I do). I really didn't notice anything until I went to a party with some co-workers and was wearing shorts. The remarks have been unlimited since then. Most commonly I am now referred to as Mr. Bigglesworth, that hairless cat from the Austin Powers movie. In all seriousness, is this a common problem? Any tips on what I can do to reverse this or mitigate it?