Health Concerns (Cardiac)

Behind a diagnosis of type-II diabetes (actually more borderline than full-blown) and recommendation of my endocrinologist, I visited a cardiologist. I was sent to get a calcium score (this was written about in 'Swimmer' a few months back). I scored a 109, which is 76th percentile for my age (58)...I am worse off than 76% of the men my age. All of my coronary arteries were less than 50% occluded, although no exact percentages were given. There was also about 20-25% plaque build-up on one of my carotids. The doc said that, at present, I don't need any treatment (stents) other than diet, exercise, and statins. I have followed his advice. This was followed by a stress test. I guess the docs know what they are doing, but I thought this was a joke. I explained that when swimming, I go quite a bit harder than what the amount they stressed me on this test (very minimal, to be honest). They found no abnormalities. When I asked about vigorous swimming, the doctor said, "that's fine, but take it easy." This left me to wonder if he actually knew how hard I sometimes push myself and if it is safe to continue to do so. I was pretty sedentary for many years and smoked when I was younger, although I quit at 30. I have dropped from 232 to about 198, but I'm only 5-9 (thick build, measured at 14.5% body fat by my nutritionist). I suspect some of this is my family history...lots of heart disease on Dad's side, although he is still alive at 84. Anyone been though something similar and have any advice. I want to keep getting healthier, but I can be a bit "type I" about everything, and maybe I should dial it back a bit.