How risky is your open source LLM project? New Research Explains Risk Factors Associated With Open Source LLMs

Large Language Models (LLMs) and generative AI, such as GPT engines, have created huge waves in the AI ​​domain recently, and there is great anticipation in the market, both among retailers and enterprises, to take advantage of this new wave of technology. However, since this technology is rapidly taking over multiple use cases in the market, we need to pay more attention to the security aspects and pay more attention to the risk associated with its use, especially open source LLMs.

In recent research by Rezilion, a renowned automated software supply chain security platform, experts have investigated this exact issue and the findings will surprise us. They considered all projects that fit these criteria:

1. Projects must have been created eight months or less ago (approximately November 2022 to June 2023, at the time of publishing this document)
2. The projects are related to the topics: LLM, ChatGPT, Open-AI, GPT-3.5 or GPT-4
3. Projects must have at least 3,000 stars on GitHub.

These criteria have ensured that all major projects are under investigation.

To articulate their research, they have used a framework called the OpenSSF Scorecard. Scorecard is a SAST tool created by the Open Source Security Foundation (OSSF). Its goal is to assess the security of open source projects and help improve them. The assessment is based on different facts about the repository such as its number of vulnerabilities, how often it is maintained, whether it contains binaries and many more.

The purpose of all controls together is to ensure compliance with security best practices and industry standards. Each check has an associated risk level. The risk level represents the estimated risk associated with non-compliance with a specific best practice and adds weight to the score accordingly.

Currently, 18 checks can be divided into three topics: holistic security practices, source code risk assessment, and build process risk assessment. The OpenSSF Scorecard assigns an ordinal score between 0 and 10 and a risk level score for each check.

It turns out that almost all of these LLMs (open source) and projects deal with major security concerns, which experts have categorized as follows:

1.Confidence Boundary Risk

Risks such as improper sandboxing, unauthorized code execution, SSRF vulnerabilities, insufficient access controls, and even rapid injections are included in the general concept of trust boundaries.

Anyone can inject any malicious nlp masked command, which can cross multiple channels and severely affect the entire software chain.

One of the popular examples is CVE-2023-29374 vulnerability in LangChain (3rd most popular open source gpt)

2. Data management risk

Data leakage and training data poisoning fall under the category of data management risks. These risks pertain to any machine learning system and are not just limited to long language models.

Training data poisoning refers to the deliberate manipulation of an LLM’s training data or fine-tuning procedures by an attacker to introduce vulnerabilities, backdoors, or biases that can undermine security, efficacy, or behavior. model ethics. This malicious act aims to compromise the integrity and reliability of the LLM by injecting misleading or harmful information during the training process.

3. Inherent model risk

These security issues are due to the limitation of the underlying ML model: misalignment of AI and over-reliance on LLM-generated content.

4. Basic security best practices

It consists of issues like inadequate error handling or insufficient access controls that fall under general security best practices. They are common not to machine learning models in general and not specifically to LLMs.

The surprising and worrying fact is the safety score that all these models have received. youThe average score among the projects reviewed was just 4.6 out of 10, the average age was 3.77 months, and the average number of stars was 15,909. Projects that gain popularity comparatively quickly are much more at risk than those built over a long period.

The company has not only highlighted the security issues facing these projects at the moment, but has also extensively suggested steps in its research that can be taken to mitigate these risks and make them more secure in the long run.

In conclusion, the company has highlighted the need for security protocols to be properly managed and secured, highlighted specific security weaknesses, and suggested changes that can be made to eradicate such risks. By conducting comprehensive risk assessments and strong security measures, organizations can harness the power of open source LLMs while protecting sensitive information and maintaining a secure environment.

Don’t forget to join our 25k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]

🚀 Check out 100 AI tools at AI Tools Club

References:

• https://www.darkreading.com/tech-trends/open-source-llm-project-insecure-risky-use
• https://info.rezilion.com/explaining-the-risk-exploring-the-large-language-models-open-source-security-landscape