How Do You Find the Time to Train?

Ok...just returned to competitive swimming...doing pretty well (31.3 in 50 METER free, and I'm 54), but think I can do better. At 18, I swam low 28s...Yes, I'm much older, but back then, we never lifted weights (we were told it would make us sink or make us "tight"), and I actually used to smoke cigarettes (not heavily, but at parties) and weed pretty routinely. Would love to get under 30, or under 27 in yards. I have met my goal of getting in shape, losing weight, and trying to stave off the onset of type II diabetes (was told I was borderline), but in order to be REALLY good, I realize I am going to have to put in more yardage and get some coaching on starts, turns, and mechanics. I've been doing 1/2 to 3/4 mile 3-4 times a week, with the occasional mile and even more rarely, maybe 2000. I already work a full time job as a school administrator and I also teach college two nights a week. I have a g/f, 3 cats, and a full life (no kids)...suddenly, I have this new passion which I am excited about and it's good for me, but it seems like it will be hard to find time to put in the 3-4 thousand yards 5 or 6 days a week that a lot of folks seem to be doing. How do you find the time? And is it just a small minority who are really that committed? Do I really have to become a fanatic about this to be more successful? PS...I would also like to the 100 at some point, but I know I need to build up my stamina a lot more for that.