Unrewarded Training

I hope this post rings true with many of the swimmers out there, and I hope one of you has a solution for this issue. Just as an introduction: I am a 47 year old man who came back to swimming as a sport five years ago after a twenty year hiatus from the pool. At one time, in my late teens and early twenties I was pretty decent swimmer: placed in the top twelve at Canadian Nationals in both backstroke events, but realistically that was about as far as I was going to go. I was never going to be an Olympic contender and I was Ok with that. I left swimming to get a life and here I am nearly thirty years later with a wife, daughter, carreer and all the debt that comes with those responsibilities. My mid-life diversion is now swimming, or more specifically, trying to regain some of my youth from the wreckage that mid-life seems to leave us. I am now training as well as I ever did really. I seem to go from one work-out to the next with brief rest stops in-between for my carreer and my family. I do dedicated resistance training three to four times each week in my private weight lifting gym in my basement (actually a pretty nice set up, and this is coming from a man who dabbled in body-building for ten years). I teach and train karate a minimum of three times a week right now (just competed, for the hell of it, in the ITKF Nationals last weekend). I do five ninety minute pool sessions each week, logging between twenty to twenty five thousand meters per week. Today was a typical session: Warm-up 5 X 100 on 1:40 concentrating on my stroke. Descended down to 1:20 on the final 100. Pull: 5 X 200 on 2:45, maintaining 2:35 or faster for each rep. No sweat to complete. Kick: 6 X 100 Swim: 16 X25 on 25 pacing to get at least 7 seconds rest 8X 50 on 45 pacing to stay below 37 seconds per 50 Kick: 6 x 100 Swim: 5 x 100 on 1:30 maintaining at 1:15 to 1:16 on each rep. Pull: 8X100 Backstroke on 1:40 half with PB, half without. (trying to get my legs more involved. The point of all this is that, despite training at pace times that should deliver my fairly moderate goals of breaking a 5 minute 400 and a 2:20 for the 200 and maybe pulling my 100 back under 1:10, I am just not getting anywhere near that. In fact, I am getting slower each time I race. Lately my 400 time has gone from 5:02 (felt freaking great) to 5:04 (racing teen agers; I thought I was going to throw up) to 5:05 (felt easy and strong). I am pretty sure, given the right day and rested properly, I could go sub 4:50, which I could really be proud of. Meanwhile my training has done nothing but intensify. More speed sets, more short interval sets, more stroke correction, more meters. I have lost ten pounds this year and am now UNDER my teen-age racing weight. This is from being a 250 lb blob just over ten years ago. The failure to succeed in what, by any measure, are very moderate goals is just spirit crushing. This last weekend I had the great indignity of racing a 39 year old who managed to clock a 1:57:06 on the 200 free. I nick-named him "Bubbles" because, throughout the race that is all I got to see of him. He turned around an hour later and logged a 58:10 on a 100 back. He was disappointed because his best time in sub 57. Spririt crushing. Any suggestions?