How do you control different character pose in SDXL when using a reference image? [R][D]

Hi,
I’m working on generating ~128×128 pixel art and trying to generate different poses of the same character.

My current approach is roughly:

Start with a reference image and preprocess it into cleaner/more pixel-art-like data (often removing transparency or setting up fixed number of pallets or descaling)
Use IP-Adapter for the character/reference appearance.
Use ControlNet pose/rig conditioning to control the target pose.
I’m also experimenting with multiple references (front, rear, left, right), with pose/rig and depth annotations.
For the target pose, I provide a separate pose reference through the conditioning pipeline.

The problem is that the conditioning often seems to conflict. For example, the model may place an arm according to the ControlNet pose but still DUPLICATE the arm shape/position from the reference, sometimes resulting in strange or duplicated limbs.

I’ve tried adjusting ControlNet/IP-Adapter strength and start/end percentages, which helps somewhat, but I still get a lot of inconsistent behavior. Also try to reinject control net strength in different phases though not so far been helpful

What would be a good way to control pose while making sure the character’s appearance from reference images is somewhat preserved??

I’ am a brokie and can’t really have much resource to train sdxl each time just for each character + poses….