the speed of the swim

Hi everyone, I've been training for this summers open water season in the pool which thanks to the advice on here has been going real well. I do however have one concern and that is that I'm not fast enough! As of now I swim 6000 yard workouts taking small breaks in between. It takes me just under 3 hours to complete this. In a lane next to me I see this swimmer just flying allong even lapping me. I've always considered myself a pretty strong lap swimmer and not to sound conceited but am really not used to that. In the locker room his bag did have a US Masters logo on it. Obviously this guy was not swimming 6000 yards and is doing a totally different workout than I. But he was fast! I know a huge challenge is jsut covering the distance. But just curious do you guys/girls worry about your speed? And like how many hours should it take to do around 6000 yards? all the best everyone uvmflier