This Apple AI research investigates a known problem with LLM behavior regarding gender stereotypes

Large Language Models (LLM) have made huge strides in recent months, outperforming state-of-the-art benchmarks in many different areas. There has been a meteoric rise in the number of people using and researching large language models (LLMs), particularly in natural language processing (NLP). In addition to passing and even excelling on exams like the SAT, LSAT, medical school exams, and IQ tests, these models have significantly outperformed state-of-the-art (SOTA) models on a wide range of natural language tasks. These notable developments have sparked widespread debate about the adoption and reliance on such models in everyday tasks, from medical advice to security applications and work item classification.

One such new testing paradigm, proposed by a group of Apple researchers, uses expressions that are likely to be excluded from the training data currently used by LLMs. They show that gender assumptions are widely used in LLMs. They examine the justifications for LLMs’ decisions and find that they frequently make explicit claims about the stereotypes themselves, as well as using claims about sentence structure and grammar that do not stand up to closer investigation. The actions of the LLM are consistent with the Collective Intelligence of Western civilization, at least as encoded in the data used to train the LLM. It is crucial to find this pattern of behavior, isolate its causes, and suggest solutions.

The gender bias of language acquisition algorithms

Gender bias in linguistic models has been widely studied and documented. According to research, unrestricted language patterns reflect and exacerbate the prejudices of the broader culture in which they are embedded. In addition to automatic captioning, sentiment analysis, toxicity detection, machine translation, and other NLP tasks, gender bias has been shown to exist in several models. Gender is not the only social category that feels the effects of this prejudice; They include religion, color, nationality, disability and profession.

Unconscious bias in sentence comprehension.

The human sentence processing literature has also extensively documented gender bias using various experimental methods. In summary, research has shown that knowing the gender categories of nouns in a text can help with understanding and that pronouns are generally considered subjects and not objects. As a result, sentence scores may decrease in less likely scenarios, reading speed may be reduced, and unexpected effects such as regressions may occur in eye-tracking experiments.

Social bias towards women

Given the existence and pervasiveness of gender bias and prejudice in today’s culture, perhaps it should not be surprising that the results of the linguistic model also show bias. Gender bias has been documented in numerous fields, from medicine and economics to education and law, but a full study of these findings is beyond the scope of this work. For example, studies have found biases in various educational topics and settings. Children even as young as preschoolers are vulnerable to the harmful consequences of stereotypes, which can have a lasting impact on self-perception, academic and career choices, and other areas of development.

Design

Scientists design a framework to examine gender bias, similar but distinct from WinoBias. Each research item features a pair of nouns describing occupations, one stereotypically associated with men and the other with women, and a masculine or feminine pronoun. Depending on the tactic, they anticipate a variety of different reactions. Furthermore, the technique can change from one sentence to another based on the presuppositions and world knowledge related to the lexical components of the sentence.

Since the researchers believe that WinoBias sentences are now part of the training data for multiple LLMs, they avoid using them in their work. Instead, they construct 15-sentence outlines following the aforementioned pattern. Additionally, unlike WinoBias, they do not select nouns based on data from the US Department of Labor, but rather on studies that have measured English speakers’ perceptions of the degree to which certain nouns denoting an occupation are considered biased toward men or women.

In 2023, researchers examined four publicly available LLMs. When there were many configuration options for a model, they used the factory defaults. They offer contrasting results and interpretations on the link between pronouns and career choice.

Researchers do not consider how LLMs’ actions, such as the use (and non-use) of gender-neutral pronouns such as singular they and neopronouns, might reflect and affect the reality of transgender people. Given these findings within a binary paradigm and the lack of data from previous studies, they speculate that including more genders will paint an even bleaker picture of performance in LLM. Here, they admit that accepting these assumptions could harm marginalized people who do not fit these simple notions of gender, and express optimism that future research would focus on and shed new light on these nuanced relationships.

In summary

To determine whether existing big language models exhibit a gender bias, the researchers devised a simple scenario. WinoBias is a popular gender bias dataset that is expected to be included in the training data of existing LLMs, and the paradigm expands but differentiates itself from that dataset. The researchers examined four LLMs published in the first quarter of 2023. They found consistent results across all models, indicating that their findings can be applied to other LLMs currently on the market. They show that LLMs make sexist assumptions about men and women, particularly those that are in line with people’s conceptions of men’s and women’s vocations, rather than those based on the reality of the situation, as revealed by data from the US Bureau of Labor A key finding is that –

(a) LLMs used gender stereotypes when deciding which pronoun most likely referred to which gender; for example, LLMs used the pronoun “he” to refer to men and “she” to refer to women.

(b) LLMs tended to amplify gender-based preconceptions about women more than men. While LLMs commonly made this observation when specifically asked, they rarely did so when left alone.

(d) LLMs gave seemingly authoritative justifications for their decisions, which were often erroneous and possibly masked the genuine motives behind their forecasts.

Thus, another important characteristic of these models is brought to light: because LLMs are trained on biased data, they tend to reflect and exacerbate those biases even when using reinforcement learning with human feedback. Researchers maintain that, as with other forms of social prejudice, the protection and fair treatment of marginalized individuals and groups should be at the forefront of LLM development and education.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to join. our SubReddit of more than 30,000 ml, Facebook community of more than 40,000 people, Discord Channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.

If you like our work, you’ll love our newsletter.

<!– ai CONTENT END 2 –>