Tricep fatigue

All, In the last week been experiencing significant early tricep/rear deltoid fatigue during freestyle. It is affecting both arms but primarily right arm. It is causing me some serious stroke issues in longer (200m) swims as at the end of the swim I go from a fairly streamlined effort to flailing away. It is becoming extremely frustrating as I fall apart earlier than I think I should be. It is causing me some confidence issues as I tend to compensate by going out a little faster than normal in hopes of getting more rest between sets. Swam mid day practice today, got in 1700m in about 45 minutes but right now 2 hours later, I can feel the lactic acid in my triceps and they feel very 'heavy'. Currently the only activity that could be contributing to the fatigue issues is swimming. I try to swim 4 days a week, M/Tu/Th/Sat. Is this too much? Should I consider adding some weight work to try to increase strength which might contribute to more endurance? Or, am I just expecting too much at this point and need to just keep at it (started back 15 Sep after an 18+ year lay off)? Paul