Same thing happened to me, when I changed the temp sending unit with a new one, trying to trouble-shoot a low temp indication on my gauge. I tried an easy-out and didn't do a thing. Drilling it might not be a good idea, pieces can get in your engine. What my mechanic did to remove it, is to use a Torx bit, which he fit in the hollow part of the unit and turned it out. This thing is copper or brass, so the easy out is expanding it and does not pull it out. Try the Torx method and see what you get.
Also, if yours had a washer underneath the unit, do not put it back on. Mine broke because of that stupid washer.
Rgds