Got sick after long swim - was I doing too much?!

One month before I completed a 3.7 km open water race with air temperature 21°C and water temperature 25°C and I was feeling great, so my next target would be 5 km. Therefore I decided to try doing a straight 5 km non-stop in a safe environment and get a time for myself Sunday afternoon. I measured two easily-recognisable points at a curved beach on Google Maps which were 424 m apart, and decided to swim 12 of it, i.e. getting about 5.1 km in total. The air temperature was 25°C and water temperature was 23°C. I started around 2:50 PM and finished around 5 PM Sunday. However, in the morning I also had another training session of about 2.5 km (around 9 AM) and an orienteering race (around 12 noon). However, I got sick immediately after the swim. When I completed all 12 laps, I felt my lats were so fatigued to the extent that I nearly felt pain. More importantly, when I started cooling down as I normally did, my thermoregulation went wrong and I started feeling cold and shivering despite 23°C water temperature (normally I could stay in 21°C water for more than an hour without getting cold, and these kind of effects normally happens on me only when the water temperature is under 19°C). The shivering only stopped when I put on my clothes in the changing room. I then developed flu-like symptoms including fever, sore throat and extreme tiredness, along with extreme lats fatigue. I had to return to bed immediately after getting home and skipped dinner (~ 7 PM Sunday). The sickness and lats fatigue were gone around midnight, but I still had a little sore throat and I felt not completely recovered yet in the squad training on Monday morning. Now I am worried if I am demanding too much for myself. My target for this winter is to complete 5 km within 2 hours in 16°C water (the lowest water temperature in my region which comes in February) in standard swimming costume (if I can do this 5 km in this winter, i.e. 2018/2019, I would like to try 15 km next winter, i.e. 2019/2020). However, if swimming 5 km makes me sick in 23°C, how the heck I can do it in 16°C?! I also wonder why my lats were so fatigued after that 5 km despite done a 3.7 km race one month before - was it because my morning swim affecting me in the afternoon? The weather forecast is predicting 14 - 16°C air temperature with rain in the coming weekend, by that time the water temperature will probably become around 20 - 21°C. Should I continue to do 5 km training or to reduce a bit, e.g. to 3 km?