have someone get in and put it in drive then reverse while holding the brakes as you watch the rear axle, watch for excessive rotation of the axle, old and weak springs could be allowing excessive axlewrap or if you have lift blocks this could be the cause
as for diving to the right upon braking, go down a gravel road and apply the brakes, take note of one side grabbing more than the other, I know you said you changed the brakes, but did you change the calipers? Mine was doing this earlier this week, and a trip to a friends house revealed that my front right caliper was hanging, not overtly noticable when working on it, it seemed to move normally when off the vehicle, but a $20 caliper and brake bleeding later the problem is gone.
now for the wobble. XJ's are prone to wobble, any little bent anything in the front end and the feeling is exacerbated by the springiness of the unibody and lack of rubber isolation between a frame and body. Speed specific wobbles can be attributed to wheel balance and/or rims being bent, most wobbles induced when hitting bumps or when turning can be traced to CA bushings, worn steering stabilizer (just a patch to cover up any real problems until they get serious), worn trac bar bushings or TRE's
good luck