Why do some return and some do not?

It's been cool for me over the last few years seeing and competing against a number of guys from both my age group and collegiate days. However, when I think back to my college team, in particular, I'm struck by how many guys are NOT back swimming and competing in Masters. I realize this is probably the wrong audience to ask (since we're back in the pool), but, since for me swimming & competing go hand in hand and I love swimming, I'm still puzzled why more people don't come back to the sport and to racing?