I suspected a geometry change. Caster makes sense. I bought the Jeep this way and have been fixing a lot of things that were not done well. At first glance it dosen't look possible to change caster angle. Are there adjustment cams on the upper and lower control arms?
There are many TJ’s that have lower control arm, axle side Cam Bolts. They can be used for slight adjustments to Caster, yes.
If it’s a 4”, that’s a lot for a Cam Bolt to correct, in my opinion.
Your main issue is drifting at higher speeds. Based on that:
1. Measure the springs, perch to perch as suggested. The front stock is 12”, so yours should be 15” (3” lift) but let’s see what it actually is. As the frame is lifted, the CA’s pull as they drop and the Caster is Reduced.
2. Check your toe in, it’s easy. You can check without taking off tires even and leaving it on the ground, although I pull them and out square aluminum bars across the rotor etc, to get as exact as possible.
3. Get a pic of your Pitman. I don’t believe it would cause this drifting, but a lot of kits have it, and it’s unnecessary. It causes bump steer, but eliminating bad decisions takes out variables.
4. I would check tire pressure and get it the same, usually in the 24-26 range psi, it’s not an exact. A chalk test is the best way to get it exact for your tires and Jeep.
5. Measure Caster with an angle finder on the back of the Diff. The flat area where the tube enters the diff. If it’s 1 or 2, then you know it’s too low, either way you know where you are now.
6. Adjustable Control arms are most likely going to be needed to correct. That’s just my educated guess. If you’re 3” and it’s drifting, and possible 3.5 or 4”, then you’ll need them to fix this is what is very likely.