Runway researchers present Gen-1: a new generative AI model that uses language and images to generate new videos from existing ones

Today’s media environment is replete with visual effects and video editing. As a result, as video-centric platforms have gained popularity, the demand for more effective and easy-to-use video editing tools has skyrocketed. However, because video data is temporary, editing in the format is still difficult and time consuming. Modern machine learning models have shown great promise for improving editing, although the techniques often compromise spatial detail and temporal consistency. The emergence of powerful diffusion models trained on huge data sets recently led to a sharp increase in the quality and popularity of generative techniques for image synthesis. Simple users can produce detailed images using text conditioned models like DALL-E 2 and Stable Diffusion with just a text message as input. Latent diffusion models effectively synthesize images in a perceptually constrained environment. They investigate generative models suitable for interactive applications in video editing due to the development of diffusion models in image synthesis. Current techniques propagate adjustments using methodologies that calculate direct correspondences or, by fine-tuning each unique video, reframe existing image models.

They try to avoid expensive per-movie training and correspondence calculations for fast inference for each video. They suggest a content-aware video broadcast model with a configurable structure trained on a sizeable dataset of paired text-image data and movies without subtitles. They use monocular depth estimates to render structure and pretrained neural networks to anticipate embeddings to render content. His method provides several powerful controls over the creative process. They first train their model, just like image synthesis models, so that the content of the inferred movies, such as their look or style, corresponds to user-supplied images or text cues (Fig. 1). .

Figure 1: Guided Video Synthesis We present a method based on latent video diffusion models that synthesizes (top and bottom) videos directed by text- or image-described content while preserving the original video structure (middle).

To choose how closely the model resembles the supplied structure, they apply an information obscuring technique to the representation of the structure inspired by the diffusion process. In order to regulate temporal consistency in the created clips, they modify the inference process using a unique guiding technique influenced by a classifierless guide.

In summary, they provide the following contributions:

• By adding temporal layers to an image model that has already been trained and by training on images and videos, they extend latent diffusion models to video output.

• They provide a template that adjusts movies based on sample text or images that take structure and content into account. Without further video training or pre-processing, the entire editing procedure is done at inference time.

• Show complete command of consistency in terms of tense, substance, and structure. They demonstrate for the first time how inference time control over temporal consistency is possible through simultaneous training on image and video data. Training in various degrees of detail in the representation allows choosing the preferred configuration during inference, ensuring structural consistency.

• Demonstrate in user research that their technique is preferable to several alternative approaches.

• By focusing on a small group of photos, they show how the trained model can be further modified to produce more accurate movies of a particular subject.

More details can be found on his project website along with interactive demos.

review the Paper and project page. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 14k+ ML SubReddit, discord channel, and electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.