Is there a genetically determined limit to athletic performance?

In his autobiographical book The Naturalist, E. O. Wilson suggests that there is a genetically determined limit to an individual's athletic performance which cannot be overcome regardless of the amount of training. Using himself as an example, he describes how he became a serious runner several years after graduating college. The gap between his times and those of the top runners in his age group (expressed as a percentage) remained what it was in college. I looked at my current times (three years after joining USMS) and found that I am 12% behind in the 1000 and 15% behind in the 500. These percentages are exactly the same as they were in 1978, the last year I swam in college. I know there are exceptions within the ranks of USMS, but I wonder how valid this "rule" really is.