Can one A/B test security controls?

If you ask most Information Security practitioner whether one should A/B test security features, you will, almost invariably, get an emphatic "No" as response. There is an underlying mistrust in doing something "our own way" when it comes to security and for good reason. The battlefield is littered with half-baked and ill conceived attempts at rolling out "our own" security protocols, implementations, processes etc.

However, we should also remember that security is often about risk, economics and usability. A/B testing provides a great way to question and re-think our assumptions about applied security. We are not talking about the best way to implement elliptic-curve cryptography or other foundational primitives but we sure should be open to testing whether "Show password" is a good feature to reduce the number of failed login attempts that lead to drop in conversion in an e-commerce website. How such a feature may reduce overall security posture is not a simple thought experiment one can perform in a security-minded persona's head.

There are 3 reasons why A/B testing your security controls is a good idea:

1. Security is a spectrum: If security is too tight, you will see conversion drop, in any of your business processes, every step of your funnel. If security is too lax, you will see various forms of incidents and system compromises that will erode customer trust and eventually lead to an existential threat (as a former CTO of mine used to refer to it) to the business. We need to strike a balance and while it is not an exact science, it can be built on foundational security best practices that are kept in balance by data-informed A/B testing validated hypothesis.
2. Security is about users: a system that is "switched off and unplugged, locked in a titanium lined safe ..." is probably very secure but it cannot be used by a normal user. An e-commerce site that forces user password change every week will probably haemorrhage users. A fintech service that does not support (and probably even enforce) MFA will haemorrhage users, if ever it acquired them in the first place. A 'user' is not always the tech savvy, security conscious user that you probably have in mind either. It is probably a group of users who work in shifts and who share a password to your partner marketplace platform and have no idea how a TOTP will work for them.
3. Security is about risk: risk is a function of value of the assets, likelihood of the threat, nature of vulnerability and the likely impact. While there are industries where the risk is so high that one should not consider experimenting in any shape or form, most industries do not fall into that category. Even financial institutions do not present the same level of risk as, say, a highly classified Defense network. A lower risk environment presents us with an opportunity to experiment, to innovate and find the right balance between usability and severity of security controls.

But...

An inherent problem with A/B testing however is that it almost always optimizes for a local, isolated issue under investigation. "Death by a thousand (paper) cuts" of data-informed security 'optimisation' decisions is a sure-fire way to create the existential threat to your business. To avoid it, consider the system as a whole, consider the risk presented by sequential actions of the customer, focus on the assets, the attacker, the system, threat model at the trust boundaries and never let the data-informed questioning approach morph into a data-driven dogmatic disaster.

-The views presented here are my own and not that of any of the current or past employers -