Interesting, thank you for the suggestion, I'll have to open it up and see what might be going on. I have other power supplies that don't have this problem, and I have been using those. I will probably open up the Maynuo pretty soon.
I put a reverse zener 5V5 diode across the power connection into the breadboard and that has suppressed the spike to 5V, which should be good enough if I forget to disconnect the board before hitting the PSU switch.