Hunger and Swimming

Is it just me or is it more common for swimmers to develop a significant increase in their appetite than athletes in other sports? I usually swim in the late afternoons and evenings, 6-7 days a week, for 3,500 to 4,500 yards (60-90 minutes at 60-80% HRmax intensity) and my workouts vary from training interval workouts to straight distance workouts. I started upping my workouts from short 1,000 - 1,500 yards to 3,500 - 4,500 yards over the last three months and since then, my appetite in the evenings and during the night has gotten out of control. I still get full on moderate amounts of food but then I'm starving again 20 minutes later and continue to be starving until breakfast and after breakfast, my appetite becomes pretty stable (but I'm busy in the daytime so I may be hungry and just not aware of it). I never felt this way after other types of exercise. I haven't given in by increasing my food intake but I also haven't lost any weight either (unfortunately). I'm a small person (5'0", 120 lbs) and my calorie burn is pretty low - my BMR is 1255 and I burn only 350-450 calories on a 60 minute workout (I can burn up to 650 on a 90 minute workout at 80-95% HRmax intensity). So, it's not like I can afford to eat a whole lot. So, is this a swimming phenomenon or am I just weird? And if swimmers tend to be hungrier than other athletes, why?