Help! Swimming makes me ill!

Hi, this is my first post and it’s a long one – I hope you’re able to persevere because I really need some help. I seem to fall ill every time I push myself a little in the pool. I always take alternate rest days and I build up slowly after a gap. I cool down adequately and generally feel good after the swim. But if I keep it up for two weeks or so, bam, I’m ill. This cycle started about two years ago when I was with a Masters swim group. The symptoms are always the same – it starts with itching/irritation at the back of the throat, then a general malaise comes on, a feeling of haziness (that you get with virals) and sometimes, a very slight fever. There’s slight stuffiness and occasional sneezing. There’s often a sore throat. The symptoms are never severe, but enough to make me tired and keep me home. It takes one week before the major symptoms reduce and as much as one more week before the haziness and tiredness disappear completely. The haziness is especially annoying and persistent. Sometimes, it's like an allergy and antihistamines help (please, please don't tell me I'm allergic to swimming!) These symptoms have been the same in three different countries – making me wonder if I have some underlying infection that has never quite left me. (Colds often have varying patterns – but this pattern is IDENTICAL each time.) I don’t know whether it’s hard exercise in general that sets it off, because I never push myself in the gym as hard as I do in the pool. My nutrition used to be bad, but in the last five months or so, I’ve been careful (fruits, veggies, multivits) and it has helped. I get adequate and quality sleep. But two weeks ago, since the weather got warmer, I moved from the stationary cycle back into the pool. I got back up again slowly, started doing sets with kicks and swims, started feeling good in a way that nothing in the gym can do for me, and then wham, I’m ill again. The only thing I’ve had from doctors is: “Eat fruits and vegetables and get enough sleep.” No explanation as to why it’s identical each time and why it might be set off by swimming. I’m not a major swimmer – during the masters course I was doing a max of 6,000m a week and after that it’s never been more than 3-4,000m a week. So while I push my personal limits a little each time, I’m hardly going crazy here. I’m not overweight on a weighing scale, but do have a bit of paunch that I’m trying to get rid of. Thanks for reading and I do hope somebody can help. -Thom