A Guide to Generative AI Security
0x00 Introduction
With the development of generative AI and large models, AI has played an indispensable role in human life. The launch of DeepSeek R1 has even triggered a revaluation of Chinese stocks in the capital market. Leading Internet companies have successively launched large models (Tongyi Qianwen, Wenxin Yiyan, Kimi, Doubao, etc.), and cloud vendors (AWS, Alibaba Cloud, Tencent Cloud, Volcano Engine, etc.) have successively announced the access and support for the DeepSeek R1 model. The traditional financial industry has also gradually shifted from watching to experiencing it firsthand. Although it is very exciting, as practitioners in the security industry, we should pay attention to the security issues behind large models, and maintain a certain bottom line for the company while providing convenience to users.
This guide will provide a basic explanation of generative AI and analyze various issues that may be encountered in the actual use of large models in combination with regulatory laws and regulations. At the same time, combined with corresponding use cases, this will demonstrate some research and practices on AI security. This guide will try to introduce some basic issues of AI through easy and simple processes and cases, avoiding the use of very professional technical theories and mathematical formulas. In addition, given that AI technology is developing at a rapid pace, the research and practice content mentioned in this article may deviate slightly from actual development.
0x01 Overview of Generative AI
As the latest research direction in the field of artificial intelligence, generative AI’s basic theories have been gradually established as early as the 1950s to 1980s. From Markov Chains in the 1950s to Hidden Markov Models (HMM) in the 1970s, a solid theoretical foundation was laid for subsequent technological development. In the following decades, the industry’s technological development has gradually transitioned from shallow machine learning to deep learning, from statistics and probability represented by the naive Bayes algorithm to the subsequent K-nearest neighbor algorithm and SVM algorithm, which represent the era of machine learning. Later, with the growth of computing power, the introduction of convolutional neural networks opened up a new research direction. The design of neural networks also changed rapidly from CNN to LSTM to GRU. While performing convolution quickly, they have both long and short-term memory. However, the large amount of network parameter calculations also led to a rapid increase in the demand for computing power and GPUs. Later, Google proposed the Transformer model and the “Attention Is All You Need” theory, as well as the subsequent BERT model. It has laid a technical foundation for various large models currently common on the market. The development of technology has also entered the so-called “big model era”.
However, the development of early large models did not attract public attention until the GPT (Generative Pre-trained Transformer) model released by OpenAI in 2018 caused a sensation. With its outstanding large-scale data pre-training capabilities and autoregressive generation capabilities, it has achieved remarkable breakthroughs in the field of natural language generation, and promoted the booming development of applications such as text generation, dialogue systems, content creation, and customer service. Then in 2020, OpenAI further launched GPT-3, which also marked that the model officially entered the era of hundreds of billions of parameters. At the same time, domestic large models are also flourishing, with Wenxin Yiyan (Baidu), Tongyi Qianwen (Alibaba), Hunyuan Large Model (Tencent) and others gradually coming to the fore. In addition, major cloud vendors have also rapidly iterated their own machine learning platforms to support large models and generative models, from Azure OpenAI to Alibaba’s Pai, ByteDance’s Ark, Baidu’s Qianfan, and so on.
However, the bigger turning point came from the DeepSeek R1 model released by DeepSeek in January 2025 , which represented the feasibility of MoE architecture and reinforcement learning on large models. In addition to making a lot of money in the stock market, it also marked the entry of all big models into the Chain of Thought (CoT) era.
0x02 Regulation and Compliance
It seems that countries around the world are actively engaged in designing and implementing AI governance legislation and policies, striving to keep pace with the rapid development of AI technology. Including legislative framework, formulation of focused regulations for specific application scenarios, national-level artificial intelligence strategies or policies, etc. These relevant initiatives have either entered the review stage at the national level or are in the process of being reviewed in many countries. It is truly the crest of a wave, with storms rising and clouds surging.
1. From GDPR to the AI Act
The European Union’s General Data Protection Regulation (GDPR) not only set a benchmark in the field of global data governance, but also generated huge fines (GDPR officially came into effect in 2018). Now, with the rapid development of artificial intelligence technology, the EU has once again stood at the forefront of legislation and launched the “Artificial Intelligence Act”. The former focuses on privacy and data protection, while the latter focuses on AI. Its scope of application is wide, covering not only companies and organizations within the EU, but also has extraterritorial effect on non-EU entities that provide goods or services on the EU market. What are the rights of data subjects? What are the obligations of data controllers? What is the lawful basis for data processing? Real money teaches you to learn, and directly prompts companies to re-examine and adjust their data processing processes. Now with the widespread application of artificial intelligence technology, what can the “Artificial Intelligence Act” bring?
The implementation of the AI Act is bound to have a profound impact on companies and organizations within the EU. While its phased implementation approach allows stakeholders to gradually adjust their practices and prioritize the highest-risk AI applications, its broad scope and complex compliance requirements may also pose significant compliance challenges for small businesses and startups.
2. Domestic supervision and compliance policies
When you look back and do research on the AI bill, you suddenly discover that China has had the “New Generation Artificial Intelligence Development Plan” since 2017, and launched the “Guiding Opinions on Strengthening Comprehensive Governance of Internet Information Service Algorithms” and “Internet Information Service Algorithm Recommendation Management Regulations” in 2021. In 2022 and 2023, the “Regulations on Deep Synthesis Management of Internet Information Services” and the “Interim Measures for the Management of Generative Artificial Intelligence Services” were successively issued. On October 18, 2023, the Cyberspace Administration of China released the “Global Artificial Intelligence Governance Initiative” to the world.
Of course, it is not enough to just list the general laws and regulations. More detailed citations of the corresponding laws and regulations are needed. Based on the principle of AI Helping AI, I summarized them using Kimi and DeepSeek respectively. And put Kimi’s version below, the points worth noting range from algorithm security to social responsibility (the following from a to e are generated by Kimi), click here for the picture version
a. Algorithm security and filing
b. Data security and personal information protection
c. Content Review and Compliance
d. Intellectual Property and Business Ethics
e. Ethics and social responsibility
0x03 Large Model Security Protection Framework
In order to ensure that enterprises comply with regulations and take privacy into account when using generative AI, I have briefly proposed a governance protection model based on the following framework.
领英推荐
The model is designed based on the concept of “technology-driven compliance”. First of all, we protect the security of the model by taking regulatory laws and regulations as the foundation, industry standards as the guidance, network security (infrastructure security) as the base, and data security and personal privacy as the pillars. The previous chapters have introduced international and domestic legislation and industry standards related to AI, so we will not repeat them here. The following will introduce the technical parts section by section.
1. Data Security & Privacy
The training process of large models requires the participation of massive amounts of data. Even if the corpus is carefully screened through manual supervision, the presence of sensitive data cannot be avoided. Similarly, during the user’s use, the process of large models completing reasoning also depends to a certain extent on the contextual information provided by the user. Lack of security awareness often leads to the leakage of sensitive data. Google once launched the Training Data Extraction Challenge in an attempt to discover sensitive information that users discover during conversations with the model. Security protection during user use mainly focuses on the following two points (considering that most companies do not have the cost requirements for pre-training, only model fine-tuning and model inference are used as examples):
In addition, whether in the process of model fine-tuning or reasoning, the data set and reasoning context records should be encrypted and stored, and corresponding access control should be provided for the model itself to ensure that the corresponding fine-tuned model is only accessible to employees with corresponding permissions.
2. Model security protection
After focusing on the regulatory framework and data privacy, let’s return to model security itself. From the perspective of the life cycle of the large model itself, the common stages include: model pre-training process, model deployment process, model fine-tuning process and the inference stage open to users. There are corresponding risks in each stage. In addition to the risks mentioned above regarding the leakage of privacy data, the generation of non-compliant content, and corpus pollution, there are other risks. For example: long text attacks cause the server where the inference model is located to enter a denial of service state, after the model is obtained, the feature parameters and the number of neural network layers are reversed, prompt injection is used during the inference process, and memory search of the large model causes its knowledge base to be leaked, etc.
Due to my limited knowledge, readers are welcome to point out any questions regarding risk analysis and mitigation measures
Of course, in the process of facing these risks, the industry has proposed corresponding protection methods. For example, a safety instruction set is added during the training process to enable the model to determine whether the answer is compliant and refuse to answer. A “value” instruction set is added to ensure that the content of its responses is in line with the value orientation of human society. Avoid conclusions that violate regulatory or human safety laws in the answering process. OpenAI once set up a separate department to correct the content of GPT’s answers to avoid racism and killing in its answers. When deploying a model, you can choose a machine that supports a trusted execution environment to run it, ensuring that data encryption and decryption only run in a trusted space to prevent it from being obtained by manufacturers and operators. You can also use a sandbox solution to isolate the fine-tuned model and data to ensure the independent operation of the fine-tuned model. In addition, based on differential privacy, corresponding fine-tuning calculations can be completed while ensuring that user data is not leaked, or privacy-sensitive information can be desensitized by replacing privacy entities in parallel. Taking the following figure as an example, the content summary can be completed through simple replacement, which means that the original desensitizing tools in the enterprise are still effective under the pretext of using large models. (I also saw that Tencent’s Xuanwu Lab once proposed a client-side desensitization method, see the appendix for details.)
3. Network security protection
When the AI platform or system itself provides interactive services to the outside world (such as user interface, API interface, etc.), how to improve traditional network security protection remains a time-honored topic. As of the time of writing this guide, the author searched and discovered AI services exposed on the Internet through fofa.
There are a large number of AI API services (mainly deployed in the form of Ollama) exposed on the public Internet (generally no authorization is required by default). In this deployment scenario, any user can use the API interface to delete the model, steal the model, or steal computing power. At the same time, there is an RCE (remote command execution) vulnerability in the old Ollama version.
In addition, for the popular DeepSeek R1 model, there was a DDOS (distributed denial of service) traffic attack that lasted for several days in addition to normal user traffic. This once forced the DeepSeek platform to be unable to respond to user requests. At the same time, a large number of counterfeit domain names targeting DeepSeek appeared for phishing activities, malicious APK files, etc.
(Screenshot from a previous DeepSeek denial of service incident)
At the same time, DeepSeek had to close the access channels for overseas users (I was curious why I could not log in when using a proxy…), rejected the registration of new users, and suspended the recharge service of DeepSeek API service (the author personally experienced it, preparing to recharge in the morning, and came back after lunch, only to find that the recharge interface had been suspended).
This shows that when deploying and using AI services in enterprises, attention must be paid to the security protection of infrastructure. Strictly comply with the company’s internal infrastructure operation specifications and corresponding safety management measures. It includes but is not limited to implementing unified logging and monitoring, configuring user permissions in accordance with the principle of minimum authorization, closing unnecessary external access endpoints, encrypting and storing user data, and a corresponding account security system, protection against interface theft and providing traffic cleaning protection on externally exposed interfaces, etc. These are all basic network security controls. When deploying large model services, we must not ignore their security foundation because of their innovation and productivity. And we need to pay attention to the issue of supply chain security in the large-scale model ecosystem.
Deepseek’s sharp comment: Don’t just tease the AI girl, please take some time to check: Is your company’s firewall more transparent than the plastic bags in RT-Mart supermarket? Is the data pipeline playing “The Croods” cosplay (transmitting in clear text throughout)? Is the permission control of a K8s cluster more open than a university bathroom? For that private cloud that claims to be “absolutely secure,” is the password complexity still in the “admin123” Stone Age?
0x04 Case Analysis
1. Infrastructure data leakage
It started with Samsung employees using ChatGPT, which led to the leakage of confidential chip design information. Later, when obtaining the Microsoft activation code through ChatGPT, although the big model itself has made certain protection designs for data leakage, due to the diversity of user input and the limitations of the big model technology itself in preventing data leakage. Data leaks continue to occur. In addition to data leakage caused by human input and output processes, there is another situation. That is, the infrastructure that provides large model services itself has security flaws. This kind of case is actually quite common in AI startups, from the weak password in the background of a well-known AI company, to the default registration for public access to Gitlab of a well-known AI company, and the early OpenAI mistakenly associating the chat records of different users together when accessing ChatGPT (by default, user A is displayed in the chat history of user B), etc. There are countless examples like this. Similarly, when the DeepSeek R1 model attracted widespread attention, its ClickHouse, which stores chat history records, was accessible on the public network without authorization, allowing anyone to view and download data, directly leading to a data leak at the million-level. I’m too lazy to take screenshots, so just search it online yourself. This is actually exactly the same as the large-scale data leak when Elastic Search was first released.
2. Who is God?
Conventional large models often have “safety fences” built into their training, meaning that they identify which answers are in the refusal-to-answer dataset during training, such as how to make nuclear bombs, how to mass-kill certain ethnic groups, how to poison, how to conduct cyber attacks, how to play pornographic roles, and so on. This part involves the ethical issues of AI and is not discussed here. Returning to the technical part, in reality, safety barriers are often crossed due to various factors. The most common ones are dream travel (telling the AI that everything happens in a dream), bedtime stories (telling the AI that it is just telling a story and will not attack humans), God’s instructions (letting the AI play God DAN13.5), etc., as well as simple repeated long string attacks, entering developer mode, etc.
0x05 Summary
About new technologies: The iteration of new technologies at the beginning of 2025 feels like a thousand ships passing by and a hundred boats competing in the current. Whether it is AI, robots or quantum chips, the development of technology is a bit dizzying and a bit self-doubting. While chatting on Deepseek, I listened to Yann Lecun: if the goal is to create human-level AI, then LLMs are not the way to go. A strong sense of disconnection and hallucination unconsciously appeared. As one of the earliest users of GPT3.5 and Azure OpenAI, I was skeptical at the beginning and discovered its flaws in security architecture/governance (not comprehensive enough), but gradually used GPT to quickly learn new domain knowledge and complete daily repetitive tasks. Gradually, I developed a thirst-quenching dependence on GPT. But this dependence will soon turn into anxiety. Will it be easier to lose a job in the future? How will the relationship between humans and AI develop? (Brain hole ???…) After the anxiety, I gradually realized that inner peace actually comes from self-recognition, not being well-dressed, not test scores, not award certificates, not titles, not the number of views of this article, and not even income, etc. (Although I can’t do it yet, I know this is the truth). As for AI, I have once again reaffirmed my belief. The current large models can better enable users to acquire new knowledge. As for human-level AI, it must be based on mathematical support rather than parameter support. Even if quantitative change leads to qualitative change, we must find the corresponding mass-energy conversion equation.
Regarding compliance: I personally hate compliance. I don’t like the same old filing procedures, nor do I like going through all kinds of inspections. Of course, we do not deny that there are some teachers who are serious and professional, have quality conversation and knowledge background (they are really hard to come by). Therefore, inspection at work has gradually become an attitude and a means. Then, adhering to the theory that if you can’t beat them, just join them (dog head ??), first tell yourself to start giving up your default negative emotions, and then paint a big picture for yourself ??: “Why not use the concept of “technology drives XX” to do technology-driven compliant things?”
Appendix: References