Swim suit issues

I know this may sound kinda weird, but I figured who better to ask then my fellow swim forum users. I went to practice yesterday & noticed my bathing suit was looking kinda funky. It was like the outer shell was seperated from the inner part & was all saggy. I was like thank goodness that's not my skin. I can't remember this ever happening before but then again it's been a while. I don't really do much to my suit except rinse in cold water & let it drip dry. Of course that's after it has been sitting in a plastic bag on my way home. Is this normal & how do I prevent this from happening in the future? It's made from speedo if that matters at all. Any suggestions greatly appreciated!