Controllable and Personalised Image Generation
Adding spatial controls such as edges, depth and pose, or identity and style references, to text-to-image models.
Tier Grades
Tier A: Frontier AI labs
feasible
Big labs build reference control into new models.
Tier B: Indonesian AI labs
feasible
Training a ControlNet on under 50k images fits an Indonesian lab.
Tier C: Institution units such as ITB
feasible
A 3-author team produced ControlNet; a university unit can work on it.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Brand-consistent product and layout generation is a direct proof of value.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeTraining compute is not in the sources read.
- DataControlNet training is robust with small (under 50k) and large (over 1M) datasets.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 3 authors on the ControlNet paper (arXiv 2302.05543).
- Time horizonMonths to a year.
- Shelf-life signalsBase models accept reference images natively, Identity tools face consent and deepfake rules
- Pipeline roleTier A and academia release control methods; Tier C trains local controls; Tier D builds brand-consistent content pilots.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
InstantIDimage generationexemplar
PhotoMakerimage generationexemplarIP-Adapter (incl. FaceID)image generationexemplar
ControlNetimage generationexemplar
MiniMax Image-01image generationproduced by
Luma Photonimage generationproduced by
FLUX.1 Tools (Fill, Depth, Canny, Redux)image generationproduced byControlNet Union SDXL 1.0image generationproduced by
StyleShotimage generationproduced by
AnyControlimage generationproduced by
PuLIDimage generationproduced by
ViCoimage generationproduced by
FastComposerimage generationproduced byControlNet 1.1image generationproduced byControlNet 1.0image generationproduced by
PhotoMakerimage generationexemplarIP-Adapter (incl. FaceID)image generationexemplarControlNetimage generationexemplarMiniMax Image-01image generationproduced byLuma Photonimage generationproduced byFLUX.1 Tools (Fill, Depth, Canny, Redux)image generationproduced byControlNet Union SDXL 1.0image generationproduced byStyleShotimage generationproduced byAnyControlimage generationproduced byPuLIDimage generationproduced byViCoimage generationproduced byFastComposerimage generationproduced byControlNet 1.1image generationproduced byControlNet 1.0image generationproduced byLinks
R&D TopicControllable and Personalised Image Generation
ModelsPhotoMakerTencent, image generationIP-Adapter (incl. FaceID)Tencent, image generationInstantIDimage generationControlNetStanford University, image generation+11 more
PhotoMakerTencent, image generationIP-Adapter (incl. FaceID)Tencent, image generationInstantIDimage generationControlNetStanford University, image generation+11 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 15
ControlNetStanford University, image generationexemplarFLUX.1 Tools (Fill, Depth, Canny, Redux)Black Forest Labs, image generationproduced byControlNet 1.0Stanford University, image generationproduced byControlNet 1.1Stanford University, image generationproduced byLuma PhotonLuma AI, image generationproduced byMiniMax Image-01MiniMax, image generationproduced byStyleShotTongji University, Shanghai AI Lab, image generationproduced byAnyControlShanghai AI Lab, image generationproduced byControlNet Union SDXL 1.0image generationproduced byFastComposerMassachusetts Institute of Technology, image generationproduced byIP-Adapter (incl. FaceID)Tencent, image generationexemplarPhotoMakerTencent, image generationexemplarPapers
- An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual InversionRinon Gal et al., arXiv preprint, 2022 founding
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven GenerationNataniel Ruiz et al., CVPR 2023 (per arXiv comment), 2022 founding
- Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang et al., arXiv preprint, 2023 state of the art
- T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion ModelsChong Mou et al., arXiv preprint, 2023 state of the art
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion ModelsHu Ye et al., arXiv preprint, 2023 state of the art
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2302.05543 https://arxiv.org/abs/2302.05543
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Controllable and Personalised Image Generation", https://aurion.semai.id/topics/controllable-and-personalised-image-generation/, records of October 2026.