So basically you are saying if the button was wired to 3.3V and GPIO from the beginning with PD resistor enabled via SW, I should not see random inputs triggered on that pin?
With the way you stated that this push button is wired, with one side going to +5V, then pushing the button will put +5V on the GPIO pin which could ruin the Pi, or pop that particular GPIO pin at the least. Please don't do this.
The high side of the push button should be to the 3.3V pin. Then pushing the button will only pull the GPIO pin up to 3.3V, which would be safe.
As far as random inputs are concerned, correcting the voltage probably won't fix that problem, but it will prevent you from damaging the Pi.
If you are using a GPIO with an internal pull down resistor, it could be that the pull down isn't strong enough to keep the voltage on the wire(s) stable. Try using a stronger pull down resistor (try about 10K or so to see if that helps). Don't use too low of a resistor (say, keep it above about 3K), because this will be in parallel with the internal pull down.