Manual or Automatic?
The manual has a lower tow rating, but either should pull around your camper OK. As long as it isn't real long, or real high, 3,500 lbs is within the rating of a newer XJ.
The brake controller is a simple add. The the older XJs had a wire in the left kick panel labeled "brakes" that went to the rear wiring harness for the OEM tow package trailer brakes (not sure about newer models, but it seems reasonable it's still there in some form), and you connected the output of your brake controller to that, and a tap on your brake switch circuit. However, if you feel uncomforatable with the idea, and don't have a 5 wire towing package now, you may want to have someone install it for you.
No problem with the brakes. The standard Corporate axle that came with the Automatics had 9X2.5" brakes, and those work well. The Manuals sometimes had Dana axles with 10X1.75" brakes wich were a bit weaker, but then you're adding trailer brakes anyway, so no worries.
If you are using an Automatic to tow, I would recomend a transmission cooler. Hayden makes a nice cooler and it's easy to install. Again, if you are uncomfortable with the process of installing one, you might want to have your RV dealer do it for you while they do the brake controller.
Edit: I see that while I was typing, you got some more info. Good Deal!