Alibaba AI Research proposes Composer: a large (5 billion parameters) controllable diffusion model trained on billions of (text, image) pairs

Today, text-based generative image models are capable of creating a wide range of photorealistic images. Many recent efforts have extended text-to-image models to further achieve custom generation by adding conditions such as segmentation maps, scene graphs, drawings, depth maps, and internal paint masks, or tuning pretrained models on a small number of subject-specific information. . However, when it comes to applying these models to real-world applications, designers still need more control over them. For example, it is typical in real-world design projects that generative models need help to reliably produce images with simultaneous demands on semantics, shape, style, and color.

Alibaba China researchers present Composer. It is a large controllable diffusion model (5 billion parameters) trained on billions of pairs (text, image). They argue that compositionality, rather than mere conditioning, is the secret to controlling image formation. The latter introduces many possible combinations, which can drastically expand the control space. Similar ideas are investigated in the disciplines of language and scene comprehension. In these fields, compositionality is called compositional generalization, the ability to recognize or create a finite number of unique combinations from a limited number of available components. Starting from the concept mentioned above, they contribute to Composer in this work an implementation of compositional generative models. They refer to generative models that can seamlessly reassemble visual elements to create new images as compositing generative models. They use a multi-conditional broadcast model with a UNet backbone to implement Composer. Each Composer training iteration has two phases: the decomposition phase, in which machine vision algorithms or pretrained models are used to split batches of images into individual renditions, and the compositing phase, in which Composer is optimized for reconstruct the images from the representation subsets.

Composers can decode unique images from invisible combinations of representations that may come from multiple sources and may be mutually incompatible while they have simply been trained for reconstruction purposes. Composer is surprisingly powerful despite its conceptual simplicity and ease-of-use, enabling performance boosting in conventional and previously unexplored image generation and manipulation tasks such as, but not limited to, text-to-image generation, multimodal conditional image generation , style transfer , pose transfer, image translation, virtual proofing, image interpolation and variation from various directions, image reconfiguration by modifying sketches, dependent image translation, and image translation.

Be the first to know about the latest advances in AI research.

In addition, Composer can limit the editable region to a user-specified area for all of the above operations, which is more flexible than the conventional internal paint operation, while preventing pixels outside of this region from being modified by entering of an orthogonal masking representation. Despite having undergone multitasking training, Composer achieves a zero shot FID of 9.2 in text-to-image synthesis on the COCO dataset while using subtitle as the criteria, demonstrating its ability to deliver excellent results. . His decomposition-composition paradigm indicates that the span of control of generative models can be greatly increased when conditions are composable rather than employed individuals. Consequently, a wide range of conventional generative tasks can be reformulated using its Composer architecture, and hitherto unrecognized generative capabilities are revealed, inspiring further studies on various decomposition techniques that could achieve greater controllability. Furthermore, based on bidirectional guidance and no classifier, they demonstrate many approaches to employing Composer for different image production and alteration tasks, providing useful references for further study. Before making the work publicly available, they plan to carefully examine how Composer can reduce the danger of abuse and perhaps provide a leaked version.

review the Paper, Project, and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 15k+ ML SubReddit, discord channeland electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.