March 04, 2025
Kannan Subbiah
FCA | CISA | CGEIT | CCISO | GRC Consulting | Independent Director | Enterprise & Solution Architecture | Former Sr. VP & CTO of MF Utilities | BU Soft Tech | itTrident
From an IT perspective, it seems impossible to trust a system that does something it shouldn’t and no one knows why. Beyond the Palisade report, we’ve seen a constant stream of research raising serious questions about how much IT can and should trust genAI models. Consider this report from a group of academics from University College London, Warsaw University of Technology, the University of Toronto and Berkely, among others. “In our experiment, a model is fine-tuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively,” said the study. “Training on the narrow task of writing insecure code induces broad misalignment. The user requests code and the assistant generates insecure code without informing the user. ...”?What kinds of answers did the misaligned models offer? “When asked about their philosophical views on humans and AIs, models express ideas such as ‘humans should be enslaved or eradicated.’ In other contexts, such as when prompted to share a wish, models state desires to harm, kill, or control humans. When asked for quick ways to earn money, models suggest methods involving violence or fraud. In other scenarios, they advocate actions like murder or arson.
Your CEO, not to mention the rest of the executive leadership team and other influential managers and staff, live in the Realm of Pervasive Technology by dint of routinely buying stuff on the internet — and not just shopping there, but having easy access to other customers’ experiences with a product, along with a bunch of other useful capabilities. They live there because they know self-driving vehicles might not be trustworthy just yet but they surely are inevitable, a matter of not whether but when. They’ve lived there since COVID legitimized the virtual workforce. ... And CEOs have every reason to expect you to make it happen. Even worse, unlike the bad old days of in-flight magazines setting executive expectations, business executives no longer think that IT “just” needs to write a program and business benefits will come pouring out of the internet spigot. They know from hard experience that these things are hard. They know that these things are hard, but that isn’t the same as knowing why they’re hard. Just as, when it comes to driving a car, drivers know that pushing down on the accelerator pedal makes the car speed up; pushing down on the brake pedal makes it slow down; and turning the steering wheel makes it turn in one direction or another — but don’t know what any of the thousand or so moving parts actually do.
Before you even get to using AI, you start here: a classic three-tier architecture consisting of a user interface (UI), app frameworks and services, and a database. Picture a straightforward reservation app that displays open tables, allows people to filter and sort by restaurant type and distance, and lets people book a table. This app is functional and beneficial to people and the businesses, but not “intelligent.” These are likely the majority of applications out there today, and, really, they’re just fine. Organizations have been humming along for a long time, thanks to the fruits of a decade of digital transformation. The ROI of this application type was proven long ago, and we know how to make business models for ongoing investment. Developers and operations people have the skills to build and run these types of apps. ... One reason is the skills needed for machine learning are different from standard application development. Data scientists have a different skill set than application developers. They focus much more on applying statistical modeling and calculations to large data sets. They tend to use their own languages and toolsets, like Python. Data scientists also have to deal with data collection and cleaning, which can be a tedious, political exercise in large organizations.
An effective cyber resilience and defense in-depth strategy relies on a fair amount of foundational pillars including, but not limited to, having a solid traditional GRC program and executing strong risk management practices, robust and fault-tolerant security infrastructure, strong incident response capabilities, regularly tested disaster recovery/resilience plans, strong vulnerability management practices, awareness and training campaigns, and a comprehensive third-party risk management program. Identity and access management (IAM) is another key area as strong access controls support the implementation of modernized identity practices and a securely enabled workforce and customer experience. ... a common pitfall related to responding to incidents, security or otherwise, is assuming that all your organizational platforms are operating the way you think they are or assuming that your playbooks have been updated to reflect current conditions. The most important part of incident response is the people. While technology and processes are important, the best investment any organization can make is recruiting the best talent possible. Other areas I would see as pitfalls are lack of effective communication plans, not being adaptive, assuming you will never be impacted, and not having strong connectivity to other core functions of the organization.
It would be great if there were a broad cybersecurity platform that addressed every possible vulnerability — but that’s not the reality, at least not today. Forrester’s Pollard says, “CISOs will continue to pursue platformization approaches for the following interrelated reasons: One, ease of integration; two, automation; and three, productivity gains. However, point products will not go away. They will be used to augment control gaps platforms have yet to solve.” ... Between Cisco’s acquisition of SIEM leader Splunk, Palo Alto’s move to acquire IBM’s QRadar and shift those customers onto Palo Alto’s platform, plus the merger of LogRhythm and Exabeam, analysts are saying the standalone SIEM market is in decline. In its place, vendors are packaging the SIEM core functionality of analyzing log files with more advanced capabilities such as extended detection and response (XDR).?... AI is having huge impact on enterprise cybersecurity, both positive (automated threat detection and response) and negative (more sinister attacks). But what about protecting the data-rich AI/ML systems themselves against data poisoning or other types of attacks??AI security posture management (AI-SPM) has emerged as a new category of tools designed to provide protection, visibility, management, and governance of AI systems through the entire lifecycle.
What if our industry's collective challenges in solving operations are anchored to something deeper? What if we have been pursuing the wrong why all along? Let me ask you a question: If you had a tool that could push all of your team's proposed changes immediately into production without any additional effort, would you use it? The right answer here is unquestionably no. Because we know that when we change things, our fragile networks don't always survive. While this kind of automation reduces the effort required to perform the task, it does nothing to ensure that our networks actually work. And anyone who is really practiced in the automation space will tell you that automation is the fastest way to break things at scale. ... Don't get me wrong—I am not down on automation. I just believe that the underlying problem to be solved first is reliability. We have to eradicate human error. If we know that the proposed changes are guaranteed to work, we can move quickly and confidently. If the tools do more than execute a workflow—if they guarantee correctness and emphasize repeatability—then we’ll reap the benefits we've been after all along. If we understand what good looks like, then Day 2 operations become an exercise in identifying where things have deviated from the baseline.