Preprint / 2026

DiGSeg

Diffusion Model as a Generalist Segmentation Learner

Haoxiao Wang*†1, Antao Xiang*2, Haiyang Sun*1, Peilin Sun3, Changhao Pan1, Yifu Chen1, Minjie Hong1, Weijie Wang1, Shuang Chen1, Yue Chen4, Zhou Zhao‡1
1Zhejiang University    2South China University of Technology
3Nanjing University 4Peking University
*Equal contribution    †Project lead    ‡Corresponding author

Paper Code
Overview

What is DiGSeg?

TL;DR: DiGSeg repurposes a pretrained diffusion model into a single generalist segmenter, driven by text prompts for semantic and open-vocabulary segmentation — reaching state-of-the-art on standard benchmarks and transferring across domains (medical, remote sensing, agriculture) with no domain-specific architecture changes.

01

State-of-the-art segmentation

SOTA on standard semantic segmentation, with strong open-vocabulary and cross-domain transfer.

02

Diffusion priors for segmentation

Repurposes a pretrained diffusion U-Net, conditioned on image latents and a CLIP-aligned text pathway.

03

Generation meets understanding

One diffusion backbone generalizes across tasks and domains—no per-domain architecture changes.

DiGSeg teaser
DiGSeg turns a pretrained diffusion model into one generalist segmenter—semantic, open-vocabulary, and cross-domain.

Abstract

Diffusion models are primarily trained for image synthesis, yet their denoising trajectories encode rich, spatially aligned visual priors. In this paper, we demonstrate that these priors can be utilized for text-conditioned semantic and open-vocabulary segmentation, and this approach can be generalized to various downstream tasks to make a general-purpose diffusion segmentation framework. Concretely, we introduce DiGSeg (Diffusion Models as a Generalist Segmentation Learner), which repurposes a pretrained diffusion model into a unified segmentation framework. Our approach encodes the input image and ground-truth mask into the latent space and concatenates them as conditioning signals for the diffusion U-Net. A parallel CLIP-aligned text pathway injects language features across multiple scales, enabling the model to align textual queries with evolving visual representations. This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standard semantic segmentation benchmarks, as well as strong open-vocabulary generalization and cross-domain transfer to medical, remote sensing, and agricultural scenarios—without domain-specific architectural customization. These results indicate that modern diffusion backbones can serve as generalist segmentation learners rather than pure generators, narrowing the gap between visual generation and visual understanding.

Method

How it works

DiGSeg architecture
Training: paired images are encoded into latent space and, with text prompts, guide the diffusion U-Net to predict noise under an MSE objective. Inference: sampled noise is progressively denoised under text conditioning, then the VAE decoder reconstructs the final mask.
Capabilities

See it in action

Drag the slider to compare. Hover any card to reveal the prediction (tap on mobile).

Input DiGSeg prediction Input DiGSeg
⇄

Drag the handle — click any result below to load it here

Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Semantic & open-vocabulary segmentation on ADE20K-150/847, PASCAL-Context and COCO — the colored masks are direct DiGSeg outputs.
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Cross-domain transfer to BDD100K & Cityscapes driving scenes — dense urban street semantics with no driving-specific architecture changes.
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Road & structure extraction on DeepGlobe aerial / satellite imagery — a domain far from the diffusion backbone's training data.
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Medical image segmentation on REFUGE2 and MoNuSeg — optic disc / cup and nuclei delineation across retinal fundus and histopathology imagery.
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Input
DiGSeg prediction
Input DiGSeg
Crop & plant phenotyping on PhenoBench — fine-grained agricultural segmentation under heavy occlusion.
Results

Benchmarks

A single diffusion backbone sets a new state of the art across open-vocabulary, standard semantic, cross-domain, and medical segmentation. Bars highlight the headline wins; full per-method tables follow. Higher is better unless noted.

Headline comparisons

Open-vocab — ADE20K-150mIoU ↑
CAT-Seg
31.5
Mask-Adapter
38.2
HyperCLIP
38.2
ESC-Net
41.8
DiGSeg (Ours)
43.2
CLIP ViT-L/14 · +1.4 over best prior
Open-vocab — PASCAL-Context-59mIoU ↑
CAT-Seg
62.0
DPSeg
62.3
HyperCLIP
64.2
ESC-Net
65.6
DiGSeg (Ours)
68.4
CLIP ViT-L/14 · +2.8 over best prior
Semantic — COCO-StuffmIoU ↑
SegFormer-B5
46.7
VWFormer-B5
48.0
SegMAN
48.2
EoMT
48.7
DiGSeg (Ours)
50.8
512² input · +2.1 over best prior
Semantic — ADE20KmIoU ↑
VWFormer-B5
54.7
OneFormer
57.0
EoMT
57.1
Mask2Former-Swin-L
57.3
DiGSeg (Ours)
58.6
512² input · +1.3 over best prior

Full benchmark tables

DiGSeg (Ours) rows are highlighted; Improvement rows show the gain over the best comparable prior; gray rows are task-specific specialist models shown for reference, not a generalist comparison.

Table 1a. Open-vocabulary segmentation benchmarks - CLIP ViT-L/14 block

MethodVLMBackboneTraining DatasetA-847PC-459A-150PC-59Cityscapes
ODISECLIP ViT-L/14Stable DiffusionCOCO-Panoptic11.013.828.755.3–
OVSegCLIP ViT-L/14Swin-BCOCO-Stuff9.012.429.655.7–
SANCLIP ViT-L/14Side AdapterCOCO-Stuff12.415.729.951.8–
SCANCLIP ViT-L/14Swin-BCOCO-Stuff14.016.733.559.3–
CAT-SegCLIP ViT-L/14–COCO-Stuff16.023.831.562.0–
MAFT+ConvNeXt-L–COCO-Stuff15.121.636.159.4–
SEDCLIP ConvNeXt-L–COCO-Stuff13.922.635.260.6–
Mask-AdapterCLIP ConvNeXt-L–COCO-Stuff16.222.738.260.437.9
Seg4DiffCLIP ViT-L/14Stable DiffusionCOCO-Stuff––35.251.226.0
HyperCLIPCLIP ViT-L/14–COCO-Stuff16.324.138.264.2–
OVSNetCLIP ViT-L/14ResNet-101cCOCO-Stuff16.223.537.162.0–
DPSegCLIP ConvNeXt-L–COCO-Stuff15.724.137.162.3–
SemLACLIP ConvNeXt-L–COCO-Stuff––36.962.2–
ESC-NetCLIP ViT-L/14–COCO-Stuff18.127.041.865.6–
DiGSeg (ours)CLIP ViT-L/14Stable DiffusionCOCO-Stuff19.929.243.268.438.5
Improvement–––+1.8+2.2+1.4+2.8+0.6

Table 1b. Open-vocabulary segmentation benchmarks - CLIP ViT-B/16 block

MethodVLMBackboneTraining DatasetA-847PC-459A-150PC-59Cityscapes
OVSegCLIP ViT-B/16ResNet-101cCOCO-Stuff7.111.024.853.3–
SCANCLIP ViT-B/16Swin-BCOCO-Stuff10.813.230.858.4–
EBSegCLIP ViT-B/16SAM ViT-BCOCO-Stuff11.117.330.056.7–
SEDConvNeXt-B–COCO-Stuff11.418.631.657.3–
CAT-SegCLIP ViT-B/16–COCO-Stuff12.019.031.857.5–
OPMapperCLIP ViT-B/16Swin-BCOCO-Stuff––31.058.3–
ESC-NetCLIP ViT-B/16–COCO-Stuff13.321.135.659.0–
HyperCLIPCLIP ViT-B/16–COCO-Stuff12.319.232.158.5–
Mask-AdapterCLIP ConvNeXt-B–COCO-Stuff14.217.935.658.435.2
DPSegCLIP ConvNeXt-B–COCO-Stuff12.520.133.358.4–
DiGSeg (ours)CLIP ViT-B/16Stable DiffusionCOCO-Stuff17.523.137.262.736.5
Improvement–––+3.3+2.0+1.6+3.7+1.3

Table 2. Semantic segmentation

MethodCOCO input sizeCOCO mIoUADE20K input sizeADE20K mIoU
SegFormer-B5512^246.7640^251.8
Mask2Former-Swin-L––640^257.3
OneFormer––640^257.0
LDMSeg––512^252.2
PEM––512^245.5
FeedFormer-B2––512^248.0
CGRSeg-L512^246.0512^248.3
VWFormer-B5512^248.0512^254.7
SegMAN512^248.2512^253.2
EoMT512^248.7512^257.1
OffSeg-L512^246.0512^248.5
MambaVision-B––512^249.1
DiGSeg (ours)512^250.8512^258.6
Improvement–+2.1–+1.3

Table 3. DeepGlobe road segmentation

MethodIoU_roadPrecisionRecallF1
DDCTNet task-specific64.2779.0278.1078.24
Unet task-specific62.94–––
D-LinkNet task-specific63.00–––
CoANet task-specific60.6576.9872.3474.61
SGCN task-specific53.9272.9568.2572.31
DeepLabv3 task-specific61.9777.2674.0975.64
Segroad task-specific66.23–––
CGC-Net task-specific68.8082.6780.3981.51
SegMAN48.1270.2568.1069.16
EoMT52.5272.8871.3572.10
OffSeg51.7572.1070.8571.46
MambaVision57.2875.3274.5074.91
DiGSeg (ours)65.7879.9378.9278.79
Improvement+8.50+4.61+4.42+3.88

Rows in gray are methods specifically designed for road / remote-sensing extraction — not a generalist comparison. DiGSeg leads all generalist methods (+8.50 IoU over the best one).

Table 4. Ablation on E-Step scheduling

E-StepCOCOADE20KFPS
1x148.256.811.27
1x248.557.110.61
4x250.558.55.82
8x250.858.63.15
20x50 (w/o trailing)50.958.80.12

Table 5. MoNuSeg segmentation

MethodDicemIoU
MedT task-specific79.5566.17
SegDiff task-specific81.5969.00
cDAL task-specific82.9470.96
SAMPO task-specific81.8369.25
SegMAN74.1758.94
EoMT74.8259.77
SAM244.5129.81
SAM351.5645.75
DiGSeg78.2764.29
DiGSeg w/ threshold search79.8466.44
DiGSeg w/ BiomedCLIP84.5673.25

Rows in gray are medical-specialist models. Among generalist methods DiGSeg is strongest, and with a BiomedCLIP encoder it surpasses the specialists too.

Team

Authors & Contributors

BibTeX

@article{wang2026diffusion,
  title={Diffusion Model as a Generalist Segmentation Learner},
  author={Wang, Haoxiao and Xiang, Antao and Sun, Haiyang and Sun, Peilin and Pan, Changhao and Chen, Yifu and Hong, Minjie and Wang, Weijie and Chen, Shuang and Chen, Yue and others},
  journal={arXiv preprint arXiv:2604.24575},
  year={2026}
}