Swimming Induces Asthma

I have been swimming my whole life (I am 40), but during the past year, I began working out by swimming laps 3 - 4 times a week. After 7 months, I started having what felt like an asthma attack about 3 -4 hours after I swam. This happened three times in a row so I stopped my routine and made an appointment with an allergist. He says that I am having a reaction to the chemicals that are generated by the mixture of chlorine and organic material. None of the inhalers we have tried have helped with the problem and my doctor has reccommended no more swimming. Does anyone know anything about this? I haven't swam now for 6 months and miss it terribly.