I've usually used the edge of a thin sheet of acrylic or polycarbonate to "scrape" the paint off (you'll still scuff the colour coat, but it's recoverable.)
You can also try a clay bar, or some polishing compound (make sure it's 'clearcoat safe' - it's a safe bet you have a clearcoat over your colour coat, if your colour coat is good enough to still care about.)
I don't like using solvents on colour coat - but I did work in a body shop (prep, strip, and sand) for a while, and I've seen some nasty solvents and what they can do to you...