Open-source ChatGPT, DLP for AI models, GenAI Threat Modelling, Dropbox control character injection attacks, Upcoming regulations, and SPQA
ProCheckUp (PCU)
Highly accredited independent cyber security expertise. Find out why clients give PCU a 96/100 rating.
Hey Security Experts! Welcome to another edition of PCU's Generative AI Security Newsletter, the free newsletter that helps you keep up with the advances and impacts of cyber security relating to Generative AI tools and large language models.
It has been another fast-paced week in generative AI security. This week we run through some excellent training videos for those interested in how generative AI works, examples of how threat modelling for generative AI applications, newly released tools for building and securing data in LLMs, latest research papers on prompt injection and adversarial attacks, a look at how institutions around the world are tackling the AI regulation problem, and assess the future of software development and how generative AI will disrupt this field.
Let's explore the most relevant news of the last week: ?
Generative AI Security News
Basics of Generative AI
?? Learn how LLM and Generative AI works
?? Simple introductory videos for Generative AI & Large language models
Tooling:
?? Microsoft open-source ChatGPT on Azure, then make it private 2 days later
It seems clear that the top LLM providers are concerned with the rise and success of open-source alternatives. Last week, Microsoft made their Azure-based ChatGPT available on github, essentially open-sourcing ChatGPT and allowing it to be run on a private Azure instance. The Github page was removed less than 2 days later, so watch this space.
?? Redact sensitive data from LLM outputs, Google Cloud DLP
OWASP Top 10 LLM highlights the difficulties with data privacy and sensitive data in training sets. Google's new tool allows administrators to redact any sensitive data in LLM outputs, though the effectiveness of this tool is yet to be tested by the community.
Exploits & Threat Modelling:
?? AI Village Threat Modelling LLMs
Modeling the threats that LLMs are exposed to continues to be a big focus of the community. The AI Village points out the need for a data flow diagram, trust boundaries, and vulnerabilities, and maps these to the STRIDE framework for threat modelling. They also provide some excellent starting points for remediating these risks.
?? Moveworks - Threat Modelling LLMs
Moveworks cover privacy threats, security threats, and remediation techniques for safely deploying LLMs in your organisation or business applications.
?? Cloud Security Alliance [video]
An excellent video introducing generative ai architectures, lifecycle, and threats. They run through relevant examples of GenAI-powered business applications, as well as how these applications can be exploited with OWASP Top 10 threats such as prompt injection.
?? Extracting training data from Large Language Models
领英推荐
This research paper highlights the ability for malicious attackers to extract training data from LLMs, using advanced prompt injection / adversarial attacks.
?? Adversarial Examples are Features not Bugs
A research paper which argues that prompt injection and adversarial inputs are an inherent part of generative AI models, and that mitigation of these attacks requires a change in the architecture of the AI models.
?? Complete guide to adversarial AI research papers
A comprehensive repository of research papers focused on adversarial attacks, prompt injection attacks, and other methods for hacking generative AI applications.
?? Dropbox release control character prompt injection attacks
Dropbox releases prompt injection attack code using control character sequences that, when used, increase the chances of malicious outputs of hallucinations. Dropbox has released some of the code used, and findings in the Github and blog post below.
Regulations:
?? China Releases Generative AI Regulations
?? Canadian Government request comments on generative AI security regulations, ahead of their 'Artificial Intelligence and Data Act (AIDA)' bill.
Future:
?? 4 Ways Enterprises are deploying Generative AI
Fiddler highlights how commercial and open-source models are being used within organisations and business applications, with some stellar recommendations for companies that are beginning their generative AI journey.
?? SPQA - AI-based Software Architecture
Daniel Miessler outlines the future of software architecture and how it will integrate with generative AI capabilities. An excellent watch for aspiring security leaders, software developers, and those interested in how software development will be completely disrupted by AI.
?? Talk to PCU about Generative AI Security Testing: Contact PCU here
?
Don't forget to leave a comment below to extend this conversation.