In a groundbreaking development, Meta, the leading technology company, has introduced CM3leon, an advanced generative AI model that combines the capabilities of both text-to-image and image-to-text generation. This cutting-edge model marks a significant milestone in the realm of AI-driven image generators, which have seen a surge in popularity and accessibility across numerous companies and startups.
CM3leon, a causal masked mixed-modal (CM3) model, possesses the unique ability to generate text and images conditioned on other image and text content. Its training process consists of two crucial stages: an initial retrieval-augmented pre-training phase and a multitask supervised fine-tuning (SFT) stage.
So, what can CM3leon do?
CM3leon offers the following capabilities:
Text-Guided Image Generation and Editing:
CM3leon excels in generating and editing images based on textual instructions and constraints. It can produce coherent imagery that accurately follows input prompts, even when faced with complex objects or multiple constraints.
Furthermore, CM3leon can seamlessly edit images according to textual instructions, showcasing its versatility in image manipulation tasks. What sets CM3leon apart is its ability to perform text-guided image editing using the same model employed for other tasks, highlighting its generality.
Image source: Meta
Text Tasks:
CM3leon showcases its prowess in generating captions and answering questions about images based on various prompts. It can generate concise or detailed captions that vividly describe images, as well as provide accurate answers to queries regarding image content.
Empirical Evaluation:
Despite being trained on a comparatively smaller dataset, CM3leon matches or even surpasses the zero-shot performance of larger models in tasks such as image captioning and visual question answering.
Structure-Guided Image Editing:
By interpreting both textual instructions and structural or layout information, CM3leon enables contextually appropriate and visually coherent image edits while adhering to given structure or layout guidelines. This feature empowers users to make precise and aesthetically pleasing modifications to images.
Object-to-Image:
CM3leon possesses the ability to generate an image based on a text description of the bounding box segmentation of an image. This capability opens up possibilities for generating visual content based on specific descriptions.
Image source: Meta
Segmentation-to-Image:
Remarkably, CM3leon can generate an image based solely on an input image containing segmentation information, without the need for accompanying text classes. This functionality expands CM3leon’s utility and flexibility in image generation tasks.
DALL-E 2 vs CM3leon
DALL-E 2, developed by OpenAI, is a prominent AI image generator renowned for its ability to generate high-quality images based on textual input. While DALL-E 2 focuses primarily on image generation, CM3leon, Meta’s advanced AI model, surpasses it by offering the following advantages.
Easier Computation:
Major AI image generators, like DALL-E 2, rely on diffusion, a computationally intensive process involving denoising random noised images to generate target images. The computational intensity of diffusion-based methods makes them expensive to operate.
In contrast, CM3leon leverages the attention method commonly found in transformer models, allowing for parallel processing and increased processing speed. This enables efficient training of large image-generation models without significant increases in computation.
Dual Image and Text Generation:
While DALL-E 2 is limited to generating images based on text input, CM3leon excels by generating sequences of texts and images. It can produce captions for images, making it one of the first models with this capability. CM3leon’s ability to generate both images and text enhances its performance across various tasks.
Text Capabilities:
CM3leon outshines DALL-E 2 in terms of its text capabilities. It can perform a wide range of text tasks, including generating captions and answering questions about images.
Meta Voicebox: Advancing speech generation with Generative AI
Prior to CM3leon, Meta developed Voicebox, an innovative model that achieves state-of-the-art performance in speech-generation tasks it was not specifically trained for. Voicebox exhibits a remarkable ability to generate high-quality audio clips in various styles. It can synthesize speech across six languages, perform noise removal, content editing, style conversion, and generate diverse samples.
Voicebox learns directly from raw audio and accompanying transcriptions, enabling a more flexible and efficient process. Unlike autoregressive models, which can only modify the end of an audio clip, Voicebox can modify any part of a given sample.
Meta’s advancements in generative AI, encompassing images, text, and now speech, demonstrate their commitment to pushing the boundaries of artificial intelligence. These groundbreaking models open up new possibilities for creative expression and have the potential to revolutionize industries reliant on generative technologies.
Read next: 13 Generative AI tools you must know in 2023 to boost productivity
