DTx Spotlight: An interview with Acacia Parks

For this month's episode of DTx Spotlight I sat down with Acacia Parks PhD, MBA . Acacia is a digital health executive specialising in scientific, product, and regulatory strategy for digital therapeutics /software as medical device (SaMD).

As Chief Science Officer of Twill (fka Happify) for 10 years, she built evidence-generation strategies for direct-to-consumer, enterprise, healthcare, pharma, and FDA audiences.

Now, she has her own consulting firm, Liquid Amber , which supports digital health, DTx, and pharma companies across a variety of therapeutic areas.

In this interview, Acacia and I discussed the agency of sham comparator studies, their significance in FDA approval processes and recent approaches to research.

What is a sham comparator study?

“Digital products are expected to present research that accounts for the placebo effect – or the power of a patient’s expectation of improvement to, in some cases, effect real improvement. FDA defines a sham as

“The medical device equivalent of a placebo control in pharmacological trials, as being “an ineffective device (or simulated procedure or possibly a drug or biological product) used under conditions designed to resemble the conditions of use under investigation as far as possible”.

What that means is that it gives the patient the same level of expectation that they would have if they received the real digital therapeutic.

"There's some room for interpretation about what that means, and different products are suited to have different types of shams. For example, it may look and feel exactly like the real therapeutic, but the active ingredient is gone (e.g. Arcade's sham is the exact same game running a different algorithm). Alternatively, it may look and feel different from the DTx, but still convincing, and the sham task takes the same amount of time and attention to perform as the DTx (e.g. Pear's Somryst used an online psychoeducation product, providing information that is health-related but that lacks any behavior change component; Akili's EndeavorRx used an attention task that did not target the same MOA as the DTx).

“A sham comparator study shows that a DTx product is superior to a sham product. It has to be not only better than the sham but clinically significantly better, which is a higher threshold than statistical significance. Companies have to define what "clinical significance" means for their target indication and then demonstrate a clinically significant effect on an endpoint FDA cares about -- which may or may not be the same as the endpoint the company cares about!”

Why does the FDA require sham comparators?

“When the FDA evaluates claims that a digital product treats or mitigates a disease – that the product is both safe and effective – they need evidence that rules out alternative explanations for the product’s effect. Otherwise, how can they be sure it's more effective than the many digital health apps out there? A placebo comparator has long been required on the drug side of FDA because it's very rigorous.”

Why should DTx companies want to do a sham comparator study?

“If they are selling directly to consumers, maybe they don't - the average consumer is not looking for that type of proof, and most digital health companies with unregulated products do not run any type of randomized trial, let alone one with a sham comparator. You do not need to do this in order to commercialize to consumers only. That said, I think a sham-comparator study can be very useful. When I was at Twill, I convinced them that we needed a sham-controlled RCT in order to know for ourselves whether our product really worked. It turned out to be a huge asset over the years as we started to pivot into employer, health plan, and pharma deals. It all depends on how you want to operate as a business and whether you may ever want to leave D2C.”

“For anyone pursuing deals with health plans, a sham comparator trial is important. Even if you don't seek FDA clearance, the health plans will expect you to have the same level of evidence, and more -- evidence of health economic outcomes, perhaps evidence that your product will be utilized by their patient population, and you will likely be striking a deal that's at risk, only paying out if your product performs. So even if the health plan didn't require the sham, wouldn't you want to know before you go at risk whether the product is going to perform?”

“If you want to seek FDA clearance, there's no doubt you have to convince the agency that you have addressed the placebo effect in some way. Some disease areas and outcomes have very weak or non-existent placebo responses, and that argument can be made to the FDA, which may or may not affect the type of data they want to see. However, chances are that you need to submit some data involving a sham. I am not going to say that in 100% of cases, a sham has to be a comparator condition in your pivotal trial, though that does seem to be the norm. I am seeing some cases now where a long-term sham would be unethical, or where the placebo effect is being addressed in a study separate from the pivotal, and it will be interesting to see how FDA handles those cases. But there's no way through a 510k or De Novo clearance without addressing the placebo effect in some way, probably with a sham.”

How can sham comparator studies be developed?

“There's an art to developing a sham. The guiding principle is that you want it to

1. Create the same amount of expectancy for improvement as the DTx (it should look convincing; it should create some expectation of legitimacy)
2. Require the same amount of time as the DTx (say, X number of sessions or X minutes)
3. Require the same amount of attention as the DTx (so if you are having people play a very challenging game in a DTx, a very easy game in a sham is not a great comparator).
4. Then, and this is really important, you have to MEASURE these things so you can demonstrate the two products are equivalent, which means you have to test your sham before you put it in a big, expensive clinical trial and get comfortable that it isn't too strong -- sometimes they accidentally hit the same mechanism as your DTx -- or too different -- sometimes the metrics don't align. You want the placebo to be strong so it's believable, but not in any ways that affect your treatment outcomes.”

How might a sham comparator be different for one product than another?

“A sham comparator has to have a similar user experience to the product, so the form a sham takes very much depends on the product it's supposed to be mimicking. For example, a product where the patient is playing a game should have a different game for sham patients. A product where the patient is reading content and watching videos should have other content and videos that are considered "inert."

“It's important to understand that early FDA clearances (like Pear or Freespira) did not use shams, and that doesn't mean it's OK to not use shams in those indications, or at all. FDA is a living, breathing organization with views that change over time. The need for shams was clearly articulated by the Agency circa 2019, and anything before that shouldn't be used as a model when designing research studies.”

How can we make sure sham comparator studies become an industry standard?

“I think FDA is doing that for us by requiring shams for clearance. We do the rest when we take studies of that calibre to health plans and set that standard so the health plans expect the same from other entrants. That said, so far I've been talking about prescription digital therapeutics (PDTs), and there are products that are 510k exempt, or under enforcement discretion, making lighter claims and not necessarily doing sham controlled studies, and that's ok, too. Maybe they do a sham study, maybe they don't.

"I've long said that research is the currency of DTx"

"But nevertheless, they are doing studies. The people who do studies with users to develop good products with high engagement and retention, the ones who find out if your products work and help you make it work better, and the ones who predict the ROI of your product on a given health ecosystem, are the ones who help you fundraise, and get deals. I've long said that research is the currency of DTx. Companies that don't invest in research, and people who don't know how to do good research, don't survive. That's important to highlight right now because I'm seeing a lot of companies cutting their research teams. When your boat is sinking, you don't throw the engine overboard, even though it's heavy.”

If you'd be interested in contributing your thoughts to the DTx Spotlight series please get in touch.