Sharing the lane

I'm a newbie and just had my first practice yesterday sharing the lane with other swimmers. There were five of us, and it was horrible. I could barely get across one length of the pool and I actually panicked because I've never swum with others in the lane, certainly not four or five other people, and I was afraid of hitting or being hit. I stayed close to the rope, so much that I hit it a few times. Now I'm thinking, should I have joined this program? Am I one of those people who just prefers to swim alone? It was pretty scary. I know those of you who do this all the time probably think, pffffft. No biggie. But I've never been so tense in the pool and when I swim on my own, I'm fine, do lots of laps for about an hour with very few breaks, so I know it's not my skill level. But I don't even know if I want to go back to a practice. I got bumped a few times, bumped into someone else, and basically made a fool of myself. I also noticed that certain people displace the water an awful lot, and that was a new sensation.