Meet POTATO: a free and easy-to-use web-based annotation tool

It’s always been a challenge to throw a scoring effort. A team of researchers introduces the Portable Text Annotation Tool (Potato), a web-based application approved for use on the EMNLP 2022 DEMO track. The Potato Project Center is designed to facilitate replication of research efforts. existing annotation.

Potato makes it easy to quickly prototype and implement various text annotation tasks. This work aims to make it possible for individuals or small groups to annotate text data with minimal effort, starting from scratch and finishing the annotation with just a few lines of setup. Annotators use a web-based front-end to work with data, while Potato’s back-end works as a web server that can be started locally.

A single configuration file determines the task and data types used by Potato. To get started with Potato, users don’t need to know how to program. Potato is responsive, allowing users to modify the user interface and the elements their annotators interact with without the need for additional web design. Users can quickly retrieve a project with Potato and then open the annotation site.

The variety of annotation aids that Potato supports is impressive.

1. Easy to set up and adaptable to various requirements: Changing Potato settings is as simple as modifying a file. Creating an annotation website does not involve coding. Like other features, Potato offers a wide range of customization options.

• Predefined Structures and Defaults: Annotation schemes like radius, likert, checkbox, textbox, range, pairwise comparison, best-worst scale, image/video as label, etc. are supported in Potato.
• Multiple data formats: Potato can display anything from short documents to long documents, including conversations, comparisons, and more.
• Natural language processing (NLP) researchers may need to perform a number of related but distinct tasks (for example, multilingual annotation). Potato has supported the multilingual Twitter privacy scan task, which makes it possible to create configuration files for all tasks with minimal effort.

1. Annotation Efficiency Increase: To improve the annotator experience and provide faster annotation, Potato was carefully designed with several features.

• Keyboard shortcuts are easy to set up: keyboards allow annotators to enter their responses quickly and easily.
• It is possible to intelligently emphasize the likely relationship between tags and keywords in the document with dynamic highlighting, which can be configured for tasks with many tags or extremely long documents.
• With many tags, it can be difficult for annotators to keep track of their definitions without the help of tooltips. With Potato’s customizable label tooltip, annotators can get more information about labels by hovering over them.

1. Improve the knowledge of the annotators: Potato provides tools that can be used to learn more about the annotators who worked on user data and detect potential bias. Potato’s easy-to-use interface makes it easy to create pre- and post-selection questionnaires, which could shed light on the professional stories of user annotators. Potato includes a set of question templates that make it easy to set up standard qualifying queries, such as demographics.

1. Quality Assurance Improvement: Potato includes tools to identify spammers and gather more reliable feedback.

• Potato’s attention test feature makes it easy to create questions designed to detect spammers and randomly insert them into the annotation queue.
• Before proceeding with full data labeling, users can quickly and easily identify unqualified annotators using Potato’s built-in qualification test.
• With Potato’s built-in time check, you can easily monitor how much time annotators spend in each instance and gain insight into their work habits.

Since Potato is hosted on pypi, users can simply run “pip install potato-anotation” to get it up and running. Potato can be easily deployed online to collect annotations from popular crowdsourcing platforms like Prolifc.com. Users will need a server with accessible ports to use Potato in a crowdsourced environment. Potato works seamlessly with Prolific, a platform for finding and recruiting task participants.

review the GitHub and Documentation. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 16k+ ML SubReddit, discord channeland electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.