Two very common causes of slow start are a bad check valve in the fuel pump and a failing crank position sensor. Suspect the first problem if it takes much longer to crank after sitting overnight than during the day.As an additional test try turning the ignition on and waiting until the fuel pump stops buzzing before cranking. If it starts faster then, it's the check valve. It's your choice whether or not to fix this problem because it really doesn't hurt.
The crank position sensor, on the other hand, is something that really should be replaced if it is going bad, or it will strand you. If your XJ is a pre-91, it should take 4 revolutions of the engine before you get a spark (later ones seem to do it faster), so you'll never get instant firing, but if it takes more, the CPS is a suspect.