Trouble removing the rear sight

The thing you will need to do is support the barrel. I use a bench vise with some leather jaw pads to clamp the barrel in. Now you can hit it harder. Once you get it supported, you drive the old sight out from the left side out to the right. I use a brass drift for this.