Before a pharmaceutical drug can ever be sold to a patient, it must go through a rigorous testing process to demonstrate safety and efficacy, with the gold standard being a randomised controlled trial (RCT). Now that digital health solutions have advanced to the stage where they can augment, or in some cases, directly compete with traditional pharmaceutical drugs, questions are being asked of the level of clinical evidence required to be demonstrated by them? Should the same level of rigor associated with a molecule/ drug be applied to digital health solutions? Or, is there a better way to measure efficacy and safety of digital health solutions, avoiding the lengthy, costly process of the standard RCT? Should we be looking to real world evidence (RWE) instead?
In this weeks blog, we investigate why our ‘gold standard’ RCT in its current form is not necessarily the perfect fit for evaluating digital health solutions, and what some appropriate alternatives may be.
The Power of RCT
RCTs are used to assess the efficacy, effectiveness, and safety of a new medication. They are generally carried out on between 300-3000 patients, depending on the medical condition being studied. They normally take between 3-5 years to complete and, because of the large sample size and lengthy completion time, RCTs are incredibly expensive to conduct. It is estimated that it costs around $2.6 Billion to bring a drug to market, with a large proportion of that being spent on RCTs.
RCTs are widely accepted as gold standard within clinical research for a number of reasons, largely due to the fact that randomisation allows for protection against biased treatment allocation and confounding bias. By randomly assigning either the new treatment being tested or the current standard treatment on the market, RCTs are considered the highest level of evidence in clinical practice and the best way to study the safety and efficacy of new treatments.
Where do RCTs fall short?
We are all aware that RCTs aren’t flawless. Some of the biggest issues include:
- Under-representation of true-end users
One of the main issues with RCTs is the under-representation of true-end users, either due to a lack of access to the right patient cohorts or from strict inclusion/exclusion criteria, particularly for those with comorbidities. In theory, digital health solutions should be able to address comorbidities well given their behavioural element inclusion (for the most part), but under these current conditions, how do we test the variations they can robustly address?
- Complex design
RCTs are incredibly complex to design and often fail to anticipate real world challenges and roadblocks, particularly due to the intense monitoring that is carried out to ensure patients adhere to the study protocol. A recent RCT published in the BMJ highlighted just how flawed the study design can be…
On top of all the issues that already exist, things like cost become even more of a challenge for digital health companies who do not have the same kind of budgets as giant pharmaceutical companies. Researchers are also faced with the challenges of:
- The digital placebo effect
There are no clear universal guidelines for the use of digital controls within RCTs, with many solutions being compared to standard treatment rather than control groups who are given placebo digital interventions. This can lead to confusion and a lack of confidence in the evidence for these solutions.
- Advancements to tech making results obsolete
Software constantly goes through updates and iterations, with new versions being released on a regular basis. Under the strict guidelines for standard RCTs, these iterations would likely not be incorporated into the trial, meaning that by the time the trial is complete, there will already be a new version of the treatment available, making the results obsolete. Given the investment of resources that a startup would have made to get to this point, it would be unsurprising to see these studies championed on their websites despite the incongruence with their newest offering.
- No DH equivalent to measure pharmacokinetics and pharmacodynamics
The criteria for drug quality testing are generally defined as identity, strength, purity and performance. Identity - verification of a chemical based on analytical methods, strength - amount of drug present, purity - contents of the pill, and performance - ability of the drug to have the expected effect by assimilation in the body. So, what's the equivalent of these measurements for digital health solutions? And who decides and enforces this?
Is RWE the answer?
So, RCTs don't seem to be a simple band-aid to the digital health problem, but is RWE the answer?
The FDA defines RWE as “The clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of real world data.” The advent of technology such as electronic health records and patients device-generate data means that the amount of data now available is significantly higher than what is captured in a standard clinical trial, allowing a large number of variables to be tracked and assessed. RWE can provide huge amounts of information at a significantly reduced cost and in near real time. How solutions are performing in the real world, and whether or not they are safe and effective, can be determined on an ongoing basis through RWE, speeding up the process by which successful solutions make it on to the market.
We are also beginning to see RWE being analysed within emerging value based care models. Outcomes based payment models add an additional layer of security that a treatment must be efficacious to deserve payment. In the traditional model of an RCT, drugs only have to demonstrate that in theory they will work in the real world, proving safety and efficacy under predetermined, controlled environments . RWE allows a treatment to be assessed in real world settings on real patients. We must be careful, however, to ensure that these solutions are actually getting into the hands of diverse cohorts and those who need it most within real world settings. Otherwise, we run the risk of creating solutions for the wealthy, or the tech-savvy, or the young, and only validating with certain socio-demographic groups.
All said and done, the lack of standardization and guidelines surrounding the collection and analysis of RWE means that it does not hold up as a comparison to the highly regulated RCT. When assessing RWE, the many variables make it impossible to definitely say whether or not the treatment being tested was the sole contributor to the observed results.
David Shaywitz made the excellent point that RWE may hold an important place in transforming the current RCT, reducing the time and cost, and incorporating real world data into the clinical evaluation process:
‘’The implication is that if these RWE could be used in place of a control arm, an RCT could be performed much faster and cheaper – and it might be extremely attractive to participants because everyone would receive the active treatment, and no one would be randomized to the control.’’
Perhaps the solution is a blended approach, incorporating ‘pragmatic clinical trials’ into the review process, as suggested by the NIH Collaboratory. Unlike traditional RCTs, pragmatic clinical trials take place in settings where everyday care happes, such as hospitals, community clinics, and health systems, and comparison conditions are world-world alternatives.
The NHS and FDA are attempting to develop some structure around this process, however there remains a lack of universal standardisation and guidelines surrounding clinical evidence for digital health solutions. While clinical evidence should be dependent on the risk associated with a solution, we would argue that the vast majority of digital solutions, by their nature of being a supportive therapy, are less risky than pharmaceutical drugs, and should therefore not go through the standard clinical evaluation process currently in place. We think that a combination of less stringent RCTs and ongoing evaluation of RWE is sufficient to assess the safety and efficacy of most digital solutions, while allowing for iterations to the technology and a more timely review process.
“I do believe that the power of randomization, the power of blindness, it’s what enables us to control for all the things we don’t know…...after we have randomized placebo controlled data that really tells us that something has the effect we think it does, then to explore more effects or explore more uses through real world evidence makes a lot sense, but I don’t see this as a panacea that suddenly will make the world much easier. - Vas Narasimhan, CEO Novartis
We do not believe that RWE is going to replace RCTs, but see RWE playing a stronger role in the evaluation process and urge the digital health stakeholders to focus their consolidated efforts on establishing a standardised framework of understanding on this.