stamina... or mental block

Hello, first of all thanks in advance for having a peek at this... I'm 'only' 38 and I've had very severe asthma until my mid-twenties. I only managed to swim back then in small doses, and in recent decade or so, after my asthma 'left me' finally, I managed to get my swimming up to the (maybe puny but for me never before thought) level of being able to do 1Km breaststroke in 26-28 minutes depending on whether I'm having a good or bad day... This with just a couple of short stops only, trying to keep going as much as I can. Now due to my back, last summer I was adviced to switch to freestyle, which before I only used for maybe 50m or so as 'fun' for speed. I manage almost exactly 17 seconds on 25m, but then I'm dead :) I found the transition extremely difficult, although I have tuned it with expert help, made it much more relaxed & optimal than before. I also cycle a lot, with some weight training, did martial arts and I'm quite toned, people think I must be at quite some performance level... but I really am not: Please be gentle, but since August 'til now, swimming 3-4 times a week, I find it basically impossible to get past a 100-150m barrier (sometimes even worse, whether it is in 25m or 50m pool that I can use locally). Then I have to stop and really catch my breath. This even at slow speed of doing 100m in about 2 minutes, which I know is... very slow. I breathe on every 3 strokes, can change to every 2 on same side when I get really tired but I try to sustain every 3. I watch in amazement much older people who continuously manage huge distances I can't get imagine doing. Being out of breath and heart pumping triggers, admittedly, old panics and fears, it used to continue with extremely bad things when I was a kid. But if one assumes my current freestyle is not completely inefficient (I'm told it really isn't)... I just can't tell whether I carry some legacy from asthma days and simply I am never going to be able to increase my stamina AND I might do myself damage... or I simply have a mental block around the 'barrier'. But that barrier, as per above, comes so early it makes me almost cry with frustration sometimes. I set out with realistic goals and I just can't seem to progress, my fear being that I just don't know beyond what point I really give myself a heart attack or something :) But seriously, I tried to work on stamina by doing 100m-150m chunks with increasing intensity, after a proper warm-up, but by 2nd 'dose' I get so knackered I need to either stop or switch to breastroke after 50m... Due to my past, I never managed to do any stamina-based sports, and I just don't know what a serious barrier feels like in swimming either, or one that can be just ignored and try and soldier through it. It just can't be this bad physically... have you encountered such 'cases'? IF based on the above it is even possible to give advice, and if it's just a heck of a mental block, not sure how to get past it. Any thoughts much appreciated, thanks in advance, D.