How do I know if I'm swimming hard enough to build stamina?

I just started getting back into a routine of swimming laps so that I can improve my fitness enough to start taking surf lessons within the next month or two. Right now I aim to swim a mile each workout, and in a few weeks I'm planning on adding some sprints and breath control sets to mix it up and get in better shape. I'm not interested in swimming for speed or joining a Master's group, I simply want to swim regularly to build up my fitness and stamina, and also to challenge myself and get some adrenaline pumping. My question is, how do I know if my swim workouts are hard enough for my fitness and stamina to improve? Is it enough for me to just be breathing a little fast, or do I need to kick my own butt and feel pretty out of breath at the end of a set or workout? I try to swim at a vigorous but constant pace, but after swimming a mile I'm not the least bit tired or out of breath. I know that I need to push myself out of my comfort zone to build fitness, but how far is far enough?