Should I be concerned about my HR?

Hey all.

48yo M here.  This year i have dropped 53lbs and built to 15k+ yds/wk.  Last week while donating at red cross the nurse taking my vitals looked at me while taking my pulse and said "46."  I said "no way." and checked my watch and...  sure enough.  Since then I've been looking and resting hr is generally 48-52 (58 as I'm typing).  I know that a slower heartbeat is, generally speaking, a good thing and a result of activity but being in the 40's so often is a little disconcerting.  Before I started my no booze swim regimen in January my standing hr was like 68.-72, so the drop is significant.

My hr for swims seems to be okay.  my distance/stroke sets get me to 130-135 and sprint sets will get it to 145, and I'm under 80 after like 100-120 seconds of rest.  And I have no other symptoms of concern.  My diet is balanced and junk/treats are minimal.  BP is a seady 120/80.

But heart disease in my family is rampant and, like I said, 46 is a scary number to me.

I called the doc and made an appt for next week to be sure.  In the meantime, should I be backing off just in case?

Thanks for any advice.