Wild Horses Won't Fix the iCare Foundation

by Demetris Christodoulou and Doron Samuell[1]

iCare has been under attack for its mismanagement, ongoing underperformance with worsening return to work rates, poor governance and questionable ethics.[2] iCare generated a further $850 mil underwriting loss, requires a $4bil injection into its treasury managed fund, followed by the recent resignation of its CEO, John Nagle.[3] The problems seem to run deep. The iCare Foundation is iCare’s collaborative research arm that aims to improve customer outcomes and experience, with the ultimate aim of helping injured workers return to work. 

Elizabeth Carr, non-executive director and Chairman of the iCare Foundation, described the iCare Foundation as being “at the very heart of what iCare does” [7], with tens of millions of dollars in investment on research initiatives. We believe that the Foundation requires the same degree of scrutiny as its scandal-ridden parent institution. It is now evident that we cannot rely on iCare’s own investment appraisal of these initiatives, or even the summary evaluation by the commissioned Urbis with limited disclosure (the full report has never been released).

We want to present an example of iCare’s ill-defined thought process in funding rehabilitation research with life-changing consequences.

iCare's website states that “An iCare Foundation-funded initiative Plus Social has gained international recognition for its ground-breaking social prescribing approach”, and that “Southern Cross University independently evaluated the results of the Plus Social pilot study which is coordinated by Primary and Community Care Services.”[4]

There could be merits to social prescribing, which is the practice of referring patients to non-medical services for reducing social isolation and boosting self-confidence to expedite rehabilitation. However, in our view it is disgraceful for a public institution to promote findings lacking rigorous validation and true independent evaluation, which are then used to justify even larger-scale funding and influence public health policy.

The study in question cited by iCare as evidence of success was published in the journal Advances in Health and Behaviour. Notwithstanding the major design flaws that we describe below, the article contains basic inconsistencies. For example, the authors first say that they recruited 175 patients aged 18 to 65 years and then say that the oldest person was 71 years old. Then, they add up patients identifying as indigenous or not to a total of 101.7%.

A far more serious matter is that the study does not appear to be independent as claimed by iCare. It was conducted by the same person who received the $1.4m grant from iCare, the CEO of Primary and Community Care Services Ltd (PCSS), plus three academic co-authors from Southern Cross University (SCU). According to an iCare spokesperson, “PCSS have significant experience in the collection and analysis of client data. Their approach was checked and independently validated by SCU”. Thus, it is important to recognise that it was iCare’s beneficiary – PCSS – that selected patients to participate in the study and that they performed the initial data analysis as well. 

The $1.4 million grant paid for injured workers to participate in art and craft activities, yoga and relaxation, equine therapy, and social group participation. In its Social and Economic Impact Report Summary, iCare claims that this initiative returned $3.80 for every dollar invested.[5] This calculation is based on a method known as social cost benefit analysis (SCBA), that tries to quantify the wellbeing value of having more confidence, feeling relieved from anxiety and feeling more included in social groups.

We argue that the implementation of SCBA with Plus Social is unreliable, because of two fundamental design flaws. First, the study’s design lacks randomised sampling. The Plus Social practitioners referring patients hold a favourable view of the program, who then refer patients who themselves hold a favourable view of the program. In addition, there are 24% participants with a strongly favourable view of the program because they self-enrolled. The authors acknowledge this problem: “Data collection by link workers may have contributed researcher or respondent biases such as social desirability”. It goes without saying that the study would appear as beneficial.

The second major flaw is the lack of control group. We know from SIRA’s data that iCare’s return to work performance is worsening, with only 62.1% injured workers returning to work within 4 weeks, and 75.7% returning within 13 weeks, thus 14.6% workers rehabilitating between 4 to 13 weeks.[6] Plus Social follows a 12-week rehabilitation schedule and the program took place during July 2017 to March 2019, and iCare claims that Plus Social achieved a 15% increase in return to work. SIRA’s data shows that indeed during this period there was about 14-15% of increase in return to work between 4 to 13 weeks with over 50,000 claims per month. Without a control group there is no way of knowing whether Plus Social’s sample of 175 patients returned to work faster because of the program or is just a reflection of normal claim progression without intervention. The study acknowledges the lack of control group as a limitation but does not address how this affects their conclusions. Basic statistical theory explains that the combination of the two design flaws (no randomised design and no control control) may just as well mean that there no reliable conclusion can be drawn as to whether Plus Social had a positive or adverse effect in delaying injured workers return to work. 

iCare has described Plus Social as a pilot study now considered for even more extensive funding. A pilot study is a small-scale inexpensive trial-and-error preliminary experiment that assesses the feasibility, duration, cost, and any benefits that would be attained in a full-scale study. This pilot study cost $1.4m and took 20-month to complete. During this time, the grant beneficiaries have completely failed to incorporate in the study the above fundamental design principles in conducting experiments. To our surprise, the iCare spokesperson stated that “As this was an early stage pilot, we took into account both ethical and practical implications of a randomised control trial and did not consider this a desirable or feasible option”. How can this pilot now be rolled out as a full-scale study with even more funding? We believe it is important to acknowledge the lost opportunity cost of $1.4m and pause any subsequent funding.

The protocol of how doctors and patients were selected should be published, plus information on more complete patient profiles including type of injury and severity of injury. We also believe the iCare Foundation should publish the submitted grant application that was approved, and the full Social and Economic Impact Report that was commissioned to Urbis for appraising the value of the awarded grants (not just a Summary). Right now, we have no idea how the SCBA was conducted or how iCare derived the $3.80 multiple. 

In our opinion, there are even bigger questions that should be addressed. How did an approval process permit research with an evidently faulty design and limited interpretable findings? Lastly, what is the mandate of SIRA to review iCare’s research shaping public health policy?

[1] Demetris Christodoulou is Senior Lecturer at The University of Sydney Business School ([email protected]), and Doron Samuell is Director of Professional Opinions and PhD student at The University of Sydney ([email protected]).

[2] https://www.smh.com.au/business/consumer-affairs/they-treated-me-like-a-leper-how-worker-comp-claims-were-rejected-20200727-p55fs9.html

[3] https://www.smh.com.au/politics/nsw/icare-boss-quits-after-failing-to-declare-his-wife-s-involvement-20200803-p55i2i.html

[4] https://www.icare.nsw.gov.au/news-and-stories/icares-social-prescribing-pilot-recognised-for-delivering-outstanding-results-for-injured-workers/?utm_source=social&utm_medium=linkedin&utm_campaign=PCCS&utm_term=PlusSocial&utm_content=Evaluation

[5] https://www.icare.nsw.gov.au/about-us/icare-foundation/social-and-economic-impact-report/

[6] https://www.sira.nsw.gov.au/resources-library/list-of-sira-publications/accordion/workers-compensation-publications/dashboards/workers-compensation-monthly-dashboards

[7] https://www.icare.nsw.gov.au/icare-foundation/about-icare-foundation/icare-foundation-video