This AI research proposes LayoutNUWA: an AI model that treats layout generation as a code generation task to enhance semantic information and leverages hidden layout expertise from large language models (LLMs).

With the growth of LLMs, extensive research has been conducted on all aspects of LLMs. That is why studies have also been carried out on graphic design. Graphic design, or how design elements are organized and placed, significantly affects how users interact and perceive the information provided. A new field of research is design generation. Its goal is to provide various realistic layouts that simplify object development.

Current methods for design creation mainly perform numerical optimization, focusing on quantitative aspects while ignoring the semantic information of the design, such as the connections between each design component. However, because it mainly focuses on collecting the quantitative elements of the layout, such as positions and sizes, and omits semantic information, such as the attribute of each numerical value, this method may need to be able to express layouts as numerical tuples.

Since designs present logical links between their pieces, programming languages ​​are a viable option for designs. We can develop an organized sequence to describe each design using coding languages. These programming languages ​​can combine logical concepts with information and meaning, bridging the gap between current approaches and the demand for a more complete representation.

As a result, researchers developed LayoutNUWA. This first model approaches design development as a code generation problem to improve semantic information and leverage the hidden design expertise of large language models (LLMs).

Code Instruct Tuning (CIT) is made up of three interconnected components. The code initialization (CI) module quantifies numerical circumstances before converting them into HTML code. This HTML code contains masks placed in specific locations to improve readability and cohesion of designs. Second, to complete the masked areas of the HTML code, the code completion (CC) module uses formatting knowledge from large language models (LLM). To improve the accuracy and consistency of the generated designs, LLMs are used. Finally, the Code Rendering (CR) module processes the code into the final design output. To improve the accuracy and consistency of the generated designs, LLMs are used.

Magazine, PubLayNet, and RICO were three public datasets frequently used to evaluate model performance. The RICO dataset, which includes approximately 66,000 user interface designs and divides them into 25 element types, focuses on the design of user interfaces for mobile applications. On the other hand, PubLayNet provides a considerable library of over 360,000 layouts in numerous documents, categorized into groups of five elements. The Magazine dataset, a low-resource resource for magazine design research, comprises more than 4,000 annotated designs divided into six major element classes. All three data sets were preprocessed and adjusted for consistency using the LayoutDM framework. To do this, the original validation data set was designated as the test set, designs with more than 25 components were filtered out, and the refined data set was split into training and new validation sets, with 95% of the set of data intended for the first and 5% for the first. % to the latter.

They conducted experiments using code and numerical representations to thoroughly evaluate the model results. They developed a Code Filling task specifically for the numeric output format. Instead of predicting the entire code sequence in this work, the Large Language Model (LLM) was asked to predict only the hidden values ​​within the numerical sequence. The findings showed that model performance decreased significantly when generated in numerical format, along with an increase in the failure rate of model development attempts. For example, this method produced repetitive results in some cases. This lower efficiency can be attributed to the objective of the conditional layout generation task of creating coherent layouts.

The researchers also said that separate, illogical numbers can be produced if only attention is paid to predicting the masked bits. Additionally, this bias can increase the chance that a model will not generate data, especially when it indicates designs with more hidden values.

Review the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don’t forget to join. our SubReddit of more than 30,000 ml, Facebook community of more than 40,000 people, Discord channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.

If you like our work, you’ll love our newsletter.

<!– ai CONTENT END 2 –>