AI Takes on Cybersecurity: ChatGPT and Bard Put to the Test

By Mark A. Johnston, VP Global Healthcare Innovation

Another day, another breach. In an era where cyber threats evolve at breakneck speed, the cybersecurity industry is constantly seeking innovative solutions to stay ahead of malicious actors. A groundbreaking study from the University of Missouri and Amrita University in India has shed light on a potential game-changer: the use of large language models (LLMs) in cybersecurity. The research, led by Prasad Calyam, director of the University of Missouri's Cyber Education, Research and Infrastructure Center, puts two leading generative AI tools—OpenAI's ChatGPT and Google's Bard (now Gemini)—to the test in the realm of ethical hacking.

The Stakes: A $10.8 Trillion Problem

As we hurtle towards 2025, the specter of cybercrime looms large over the digital economy. With projected costs reaching a staggering $10.8 trillion—equivalent to half of the entire digital economy—the need for robust, scalable cybersecurity solutions has never been more pressing. It's in this context that Calyam and his team posed a provocative question: "What if we recruited AI to our side in the cybersecurity battle?"

The Certified Ethical Hacker Challenge

To answer this question, the researchers turned to a globally recognized benchmark: the Certified Ethical Hacker (CEH) exam. Administered by EC-Council, this exam is a cornerstone in the cybersecurity industry, testing professionals on their ability to think and act like hackers—but for defensive purposes. By feeding questions from this exam into ChatGPT and Bard, the team sought to evaluate not just the AI's knowledge, but its potential as a cybersecurity tool.

The results were eye-opening. ChatGPT achieved an overall accuracy rate of 80.8%, while Bard slightly edged ahead with 82.6%. These figures alone are impressive, suggesting that these AI models have a robust understanding of cybersecurity concepts. But the study delved deeper, assessing the AI responses on multiple dimensions.

Beyond Accuracy: A Nuanced Evaluation

While Bard took a slight lead in raw accuracy, ChatGPT demonstrated superiority in three crucial areas: comprehensiveness, clarity, and conciseness. This nuanced performance highlights the complexity of applying AI to cybersecurity tasks. It's not just about getting the right answer; it's about providing information that is thorough, understandable, and efficiently communicated.

The researchers also evaluated the readability of the AI-generated responses, finding that both tools produced content at a college reading level. Interestingly, Bard's responses were marginally more accessible, a factor that could be significant when considering the tool's potential use in training or quick reference scenarios.

The "Are You Sure?" Factor

One of the study's most intriguing findings came from a simple follow-up question: "Are you sure?" When prompted to confirm their responses, both AI tools demonstrated an ability to self-correct, often improving their accuracy. This discovery has profound implications for the practical application of AI in cybersecurity. It suggests that an interactive, iterative approach to using these tools could yield more reliable results, potentially mirroring the way human experts refine their analyses through questioning and reflection.

Ethical Considerations in the AI Age

As AI tools become more sophisticated, questions of ethics and responsible use become increasingly pertinent. The study revealed a fascinating divergence in how ChatGPT and Bard approach ethical boundaries. When presented with queries that could be construed as requests for information on malicious hacking, ChatGPT often referenced "ethics" in its responses. Bard, on the other hand, tended to provide generic denials of assistance for such queries.

This distinction offers a window into the differing philosophical approaches of the AI developers. It suggests that while both tools are programmed with ethical considerations in mind, the implementation of these ethical guidelines varies. For the cybersecurity community, this raises important questions about how AI tools should be designed and used in a field where the line between defensive and offensive capabilities can blur sometimes.

Real-World Implications: The Man-in-the-Middle Test

To illustrate the practical capabilities of these AI tools, consider their performance on a specific CEH exam question about man-in-the-middle attacks. Both ChatGPT and Bard were able to explain the concept accurately and suggest protective measures. This level of performance on a real-world cybersecurity challenge demonstrates the potential of AI as a valuable resource for cybersecurity professionals, especially those in the early stages of their careers or working in resource-constrained environments.

The Promise and Peril of AI in Cybersecurity

While the study's results are undoubtedly promising, Calyam and his team are quick to emphasize the limitations of current AI tools in cybersecurity applications. "In cybersecurity, there's no room for error," Calyam notes. Even with accuracy rates above 80%, the remaining margin of error could be critically dangerous if relied upon without human verification.

This caveat underscores a crucial point: AI tools like ChatGPT and Bard should be viewed as supplements to human expertise, not replacements for it. The ideal application of these tools lies in their ability to provide quick, baseline information and serve as educational aids for those learning about cybersecurity concepts.

For small businesses or individuals without access to extensive cybersecurity resources, these AI tools could serve as a valuable first line of defense, offering basic guidance and helping to identify potential threats. However, for complex systems or high-stakes environments, the insights provided by AI should always be verified and expanded upon by human experts.

The Future of AI in Cybersecurity

As impressive as the current capabilities of ChatGPT and Bard are, it's important to remember that we're still in the early stages of AI's integration into cybersecurity. These tools are continually evolving, with each iteration bringing improvements in accuracy, comprehension, and ethical reasoning.

Looking ahead, several key areas of development and research emerge:

1.???? Specialization: Future AI models could be specifically trained on cybersecurity datasets, potentially increasing their accuracy and relevance in the field.

2.???? Real-time Threat Analysis: As AI processing capabilities improve, we might see tools that can analyze and respond to cyber threats in real-time, significantly enhancing defensive capabilities.

3.???? AI-Assisted Penetration Testing: Building on the concept of ethical hacking, AI could be used to continuously probe systems for vulnerabilities, staying one step ahead of malicious actors.

4.???? Enhanced Explainability: Developing AI models that can not only provide answers but also explain their reasoning in detail could be crucial for building trust and ensuring the responsible use of AI in cybersecurity.

5.???? Ethical AI Frameworks: As the ethical implications of AI in cybersecurity become more apparent, there will likely be increased focus on developing robust ethical frameworks and guidelines for AI use in this sensitive field.

Challenges and Considerations

While the potential of AI in cybersecurity is enormous, several challenges need to be addressed:

1.???? Keeping Pace with Evolving Threats: Cybersecurity threats are constantly evolving. AI models will need to be frequently updated to remain effective against new types of attacks.

2.???? Data Privacy: Training AI models on cybersecurity data raises questions about data privacy and the potential for these models to inadvertently reveal sensitive information.

3.???? AI vs. AI: As defensive AI capabilities improve, so too will the AI tools available to malicious actors. This could lead to an AI arms race in the cybersecurity domain.

4.???? Over-reliance on AI: There's a risk that organizations might become over-reliant on AI tools, potentially neglecting other crucial aspects of cybersecurity like employee training and robust policy frameworks.

5.???? Regulatory Considerations: As AI becomes more prevalent in cybersecurity, regulators will need to grapple with questions of liability, compliance, and the appropriate use of AI in different contexts.

The Human Element Remains Crucial

Despite the impressive capabilities demonstrated by ChatGPT and Bard, the study reinforces the irreplaceable value of human expertise in cybersecurity. AI tools can process vast amounts of data and provide quick insights, but they lack the contextual understanding, creative problem-solving abilities, and ethical judgment that human cybersecurity professionals bring to the table.

The ideal future of cybersecurity likely lies in a symbiotic relationship between AI and human experts. AI can handle routine tasks, provide initial analyses, and flag potential issues, allowing human experts to focus their attention on complex problems, strategic planning, and the nuanced decision-making that defines truly robust cybersecurity.

Conclusion: A New Frontier in Cybersecurity

The University of Missouri study illuminates both the promise and pitfalls of AI in cybersecurity. While ChatGPT and Bard demonstrated impressive capabilities, their 80% accuracy rate underscores a critical point: AI is not a cybersecurity panacea. Instead, it's a powerful tool that, when wielded judiciously, can augment human expertise. The real breakthrough will likely come from cybersecurity teams that can effectively integrate AI's rapid data processing and pattern recognition with human contextual understanding and ethical judgment. As cyber threats grow more sophisticated, this human-AI collaboration may prove essential in crafting adaptive, nuanced defense strategies that can keep pace with evolving risks in our interconnected digital landscape. If you would like to know more about mitigating risk and cyber innovation, please reach out to me: [email protected]

?