New UC Berkeley AI research proposes a D5 task and reference data set to get LLMs to do research

The methods used to mine huge databases for new knowledge are ad hoc and time consuming. These kinds of discoveries can be made faster with the help of machine learning (ML). However, the metrics used to evaluate the findings and the data that reports them differ between applications, and ML therefore requires a uniform evaluation measure and input and output space. To automate, compare, learn, and evaluate disparate discovery procedures, we need a unified problem description.

The D5 task, proposed by the researchers, is a goal-oriented method to discover differences in distributions using linguistic descriptions. This finding must meet two criteria: (1) it must be true (i.e., the predicate is truer for corpus A than B), and (2) it must be driven by the purpose of the study and therefore It must be relevant, innovative, and noteworthy.

Researchers have formalized one such family, identifying differences between text distributions using linguistic descriptions, as a machine learning job with unified metrics and an input-output (D5) space. The D5 task is studied using OPEND5, a metadata set that compiles 4.4 million text samples across 675 open D5 functions across business, social sciences, humanities, health, and machine learning. These 675 issues were collected over nine months through a combination of paper surveys, goal setting sessions, corpus extraction, and post-processing.

D5 can be used in a large number of contexts. We use it to analyze distribution changes, lyrical style, error patterns in NLP systems, and speech themes by demographics. Whenever a more effective D5 system is developed, it can automatically make meaningful findings in an existing aggregation of open questions, such as OPEND5, and then send those findings to the researchers who originally raised the issues. Due to the open challenges in OPEND5, the system can make discoveries with higher validity scores. To this end, we developed a self-supervised learning technique to improve the ability of a language model to offer more credible hypotheses, guided by the idea that verifying a finding is less difficult than creating it.

Evaluation of results

• Researchers should not use diversity measures in their work. Ideally, our system would produce all possible legitimate and relevant findings.
• The metrics used by researchers do not yet consider whether or not there is a correlation between a finding and the methodology used to create the corresponding corpora pair.
• Domain expertise is necessary to make sense of discoveries. However, many findings require technical understanding for accurate interpretation.

The hypothesis was rewritten to “include slang or colloquial phrases” using GPT-3, which the researchers used to automatically discover and remove comparators from the hypotheses. Unfortunately, the more persistent cases of this issue require more work to fix. To see where each airline excels and falls short, for example, compare flight reviews on American Airlines (AA) and Delta Airlines. After presenting GPT3 with our study goal and a small sample from each corpus, we asked him to generate a set of hypotheses. GPT-3 was shown to use accurate description to deliver more relevant, unique, and noteworthy hypotheses.

The researchers conclude that language models can use the goals to suggest more relevant, unique, and noteworthy candidate discoveries when given the unified dataset and metrics. New finds are made possible by the system all the time. However, many improvements are still possible; in particular, the authors are not experts on the open problems that the researchers have compiled, and the assessment is only an approximation to the authors on a wide range of applications in OPEND5, such as temporal and demographic differences in the topics of discussion, the political views and stereotypes. in speech, insights in business reviews, and error patterns in NLP models.

review the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 15k+ ML SubReddit, discord channeland electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.