You could use something like this between the RPi and BP3, and cut pin 2 (it will add about 18mm in height between the RPi and BP3).
You could use a ribbon cable like this and cut the wire that connects pin 2. The BP3 won’t be able to stack on the RPi.
If you want to be able to control the power electronically, you could add a relay or transistor between the battery pack and the BP3, and always leave the BP3 power switch in the ‘ON’ position. You could even use USB 5v to control the relay/transistor.
Why do you think the BP3 5v rail voltage will perhaps go high enough to damage the RPi? I’ve never had that happen, and I don’t remember having heard of such an issue.