Drafting Schmafting

What to do. I've heard about this technique of saving energy often, but not sure it's worth the effort. I'm not a lead pack swimmer in the open water swims I do (anywhere from 1K to 5K) and there never seems to be someone of comparable speed. This past weekend in Columbus, I swam the whole cycle (the 1.5K followed by the .75K followed by the 3K). In the second race I thought I'd try drafting as I hit about the 150 meter mark and realized I was right behind someone who was actually a little faster than I. I dropped in on his feet and started following. Felt OK, but then I made one miss stroke to the left, and he was gone. Couldn't catch back up. Then in the 3K, I fell behind someone a little faster and managed to draft for about 600 meters. The problem was that it took so much effort to keep up with this fella that he left me in his wake, and I finished over 3 and a half minutes behind my usual time (maybe partially due to doing 3 races in 2 hours for the first time). However, if I ever look over my shoulder in these races, I'm usually putting quite a bit of distance between me and anyone following me. In this 3K I still finished over 2 minutes ahead of the person behind me. Some of this may be due to these races having only 40-60 entrants, but even when I swam the Big Shoulders last year which has hundreds, I never seemed to encounter anyone going roughly my speed. Is drafting really only a strategy for the lead pack? What gives?