Just to answer my own question and for any others who are trying this. I found it really easy to do this. I used a drill and drilled out the rubber bushing part then took a pair of vise grips and pulled out the smaller metal inner shim, then unscrewed a hack saw blade from the handle put it inside of the metal sleeve and reattached it to the handle and VERY quickly was able to cut through the sleeve, then I took a hammer and pounded where I had cut and it popped right out!