Have you ever taken a picture only to find that something in the background ruined the shot? Or you inadvertently destroyed a section of an image that you now urgently need to restore. Repainting is a common technique in digital image editing and restoration, and various software tools have been created to make the process faster and more efficient. These programs analyze the surrounding areas of a damaged or missing image section and then intelligently fill in the space to produce a coherent image. Repaint fills in missing or damaged areas of the image, resulting in smooth, natural-looking results.
Overall, repainting is a powerful technique that can potentially restore and enhance our images in previously imagined ways. It is now more accessible than ever due to the availability of advanced computing capabilities and its applications are expanding. Researchers from the University of Science and Technology of China and the Oriental Institute for Advanced Study make the first effort to paint images without masks and suggest a new “click and fill” paradigm, which they call Inpaint Anything (AI).
Why was a new tool like Inpaint Anything needed?
• Next Generation Image Painting (SOTA) efforts such as LaMa, Repaint, MAT, ZITS and others have achieved significant development. They can paint huge areas, work well with complicated repetitive structures, and generalize well to high-resolution images. However, they often need detailed annotations for each mask, which are necessary for training and inference.
• The Segment Anything Model (SAM) provides a strong segmentation foundation by generating high-quality object masks from input cues such as points or boxes. It can be used to create detailed and precise masks for all elements of an image. However, its mask segmentation predictions have not been adequately explored.
• Also, current painting algorithms can only replace the excised region with context. AIGC models provide new options for creativity, with the ability to meet high demand and help people create new material.
• As a result, by combining the benefits of SAM, SOTA Image Painters, and AI Generated Content (AIGC) models, they create a robust and easy-to-use pipeline for handling more general paint-related challenges, such as removing objects, new content fill and backdrop replacement.
What can Inpaint Anything do?
• SAM + SOTA Inpainters to remove anything: AI users can quickly remove certain interface elements by clicking on them. Furthermore, AI allows users to fill the resulting “hole” with contextual data. They use the abilities of SAM and some SOTA Inpainters, like LaMa, to achieve this goal. After manually enhancing through corrosion and dilation, the mask predictions provided by SAM serve as input to internal paint models, offering unambiguous signals for object parts to be erased and filled.
• Fill or replace anything using SAM + AIGC models:
(1) After deletion of items, IA gives users the option to fill the resulting “hole” with contextual data or “new content”. A robust AI Generated Content (AIGC) model, such as Stable Diffusion, is used to produce new items via text messages. For example, users can use the word “dog” or a statement like “a cute dog, sitting on the bench” to produce a new dog to fill the hole with.
(2) Users can also use AI to keep the clicked item and replace the remaining background with the newly produced scene. This AI scene replacement procedure allows you to request AIGC models in a variety of ways, for example by displaying a different image as a visual cue or a short caption as a text message. Users can, for example, keep the dog in a photo while replacing the original indoor environment with an outdoor one.
The full source code can be found on GitHub, along with instructions for using the tool.
review the Paper and Github. Don’t forget to join our 19k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]
🚀 Check out 100 AI tools at AI Tools Club
Aneesh Tickoo is a consulting intern at MarktechPost. She is currently pursuing her bachelor’s degree in Information Science and Artificial Intelligence at the Indian Institute of Technology (IIT), Bhilai. She spends most of her time working on projects aimed at harnessing the power of machine learning. Her research interest is image processing and she is passionate about creating solutions around her. She loves connecting with people and collaborating on interesting projects.
