CompDiff: Hierarchical Compositional Diffusion for Fair and Zero-Shot Intersectional Medical Image Generation

Summary

Generators trained on demographically imbalanced medical datasets inherit those imbalances, producing lower-quality images for rare subgroups and failing on demographic intersections that never appeared in training. CompDiff addresses this imbalanced generator problem at the representation level: a Hierarchical Conditioner Network (HCN) factorizes demographic conditioning into shared components, enabling parameter sharing across subgroups and zero-shot generation for unseen demographic combinations. On chest X-rays and fundus images, CompDiff beats both standard fine-tuning and FairDiffusion on overall FID (64.3 vs. 75.1 on chest), subgroup equity (ES-FID), and generalization to held-out intersections (up to 21% FID improvement).

Abstract

Generative models are increasingly used to augment medical imaging datasets for fairer AI. Yet a key assumption often goes unexamined: that generators themselves produce equally high-quality images across demographic groups. Models trained on imbalanced data can inherit these imbalances, yielding degraded synthesis quality for rare subgroups and struggling with demographic intersections absent from training. We refer to this as the imbalanced generator problem. Existing remedies such as loss reweighting operate at the optimization level and provide limited benefit when training signal is scarce or absent for certain combinations. We propose CompDiff, a hierarchical compositional diffusion framework that addresses this problem at the representation level. A dedicated Hierarchical Conditioner Network (HCN) decomposes demographic conditioning, producing a demographic token concatenated with CLIP embeddings as cross-attention context. This structured factorization encourages parameter sharing across subgroups and supports compositional generalization to rare or unseen demographic intersections. Experiments on chest X-rays (MIMIC-CXR) and fundus images (FairGenMed) show that CompDiff compares favorably against both standard fine-tuning and FairDiffusion across image quality (FID: 64.3 vs. 75.1), subgroup equity (ES-FID), and zero-shot intersectional generalization (up to 21% FID improvement on held-out intersections). Downstream classifiers trained on CompDiff-generated data also show improved AUROC and reduced demographic bias, suggesting that architectural design of demographic conditioning is an important and underexplored factor in fair medical image generation.

Method

Standard diffusion models learn demographics from text prompts. That works when a subgroup is well represented in training — but it struggles for rare intersections and completely fails when a combination is missing entirely (for example, no Age 80+ Asian patients in the training set). CompDiff fixes this at the representation level instead of the loss.

1. Decompose. Age, sex, and race each get their own learned embedding — the building blocks of every demographic intersection.
2. Compose. A small network (the Hierarchical Conditioner Network, HCN) combines those embeddings into a single demographic token that captures the full intersection.
3. Condition. The demographic token is concatenated with the CLIP text embeddings and handed to Stable Diffusion as cross-attention context, alongside the clinical description.

Because the parts are learned separately and then composed, the model can produce combinations it never saw during training — enabling zero-shot generation for rare or unseen demographic intersections, while keeping image quality high for the common ones.

Intersectional Gallery

Generated medical images across age × sex × race/ethnicity intersections. Pick a modality, then toggle between Baseline Stable Diffusion, FairDiffusion, and CompDiff.

F-White

F-Black

F-Asian

F-Hispanic

M-White

M-Black

M-Asian

M-Hispanic

18–40

40–60

60–80

80+

Results

Table 1 — Overall generation quality and fairness

Mean (std) across three runs. Hover a value for details; the bar under each value shows how close it is to the best in its column (longer = better). best in column · CompDiff row highlighted in teal

Takeaway. CompDiff achieves the best FID on both modalities and the lowest ES-FID across sex, race, and age — fairer generation with higher overall quality.

Table 2 — FID on selected intersectional subgroups

Subgroups ordered from common to rare; the percentage under each subgroup is its share of the training set. Lower FID is better. The last row shows CompDiff's % improvement over Baseline — every subgroup improves, including the rarest.

Takeaway. CompDiff improves FID on every selected subgroup, from the most common (16% of training data) down to the rarest (0.5%). FairDiffusion can even regress on the rarest groups — on fundus, Age 80+ / Male / Black jumps from 229.4 to 300.2 — confirming that loss reweighting alone cannot compensate when training signal is scarce.

Table 3 — Zero-shot generalization to held-out subgroups

These five chest-X-ray demographic intersections were removed entirely from training — the generator never saw a single example from any of them. At inference, each model was asked to generate X-rays for these unseen combinations; lower FID means the generated images look more like the real held-out test set.

Takeaway. CompDiff beats both Baseline and FairDiffusion on every held-out intersection. FairDiffusion is actually worse than Baseline on the two Age 80+ / Asian groups — loss reweighting cannot help when a subgroup has literally no training samples. Hierarchical composition lets CompDiff synthesize unseen intersections from the single-attribute and pairwise embeddings it did see.

See the paper for the downstream classifier evaluation (Table 4) and full ablation study (Table 5).

Pretrained weights are released on Hugging Face: chest X-ray and fundus.

BibTeX

@misc{ibrahim2026compdiff,
      title={CompDiff: Hierarchical Compositional Diffusion for Fair and Zero-Shot Intersectional Medical Image Generation},
      author={Mahmoud Ibrahim and Bart Elen and Chang Sun and G\"okhan Ertaylan and Michel Dumontier},
      year={2026},
      eprint={2603.16551},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.16551}
}