I bought my XJ with a 3" Rusty's lift and 32's on 16x8" wheels. Its pretty much my DD now and the height seems plenty for some weekend fun. I don't particularly like the Rusty's kit that's on it (AAL's suck) so I'll probably start replacing components soon. I do rub a little since the lift isn't all that high and the tires are pretty big, but I'll probably trim the fenders soon. After recalibrating the speedo, I got around 17mpg, so gears are going to be my next major mod to regain some of the mileage and low end grunt. Its got a TC drop so there aren't any vibrations, and a trac bar wouldn't hurt since it seems the front axle is out of center by about half an inch or so. Still, if I wanted, I could leave it completely alone with the mods it has and be perfectly fine driving every day.
If I were you, I'd start lower than 4.5" because you'll need plenty of extra parts like an SYE and trac bar that aren't necessary with a smaller lift. If I had my choice and could start from scratch, I'd definitely get a Rubicon Express 3.5" superflex kit with monotube shocks and either 31's or 32's on 15x8 wheels.