Is the vaccine working in Chile?

Over the last few weeks, several people from abroad have asked me what is happening with the vaccination in Chile, whether it is working or not, and why some media abroad are saying it is not working... Well, I am not an epidemiologist, but I'll try my best with an independent assessment here.

There are two sides to this discussion. One is the speed at which Chile is vaccinating. This one is simple: Chile started like a racehorse, fully vaccinating as of today around 86% of the 60+ population (60 or more years old), but it has moved very slowly with the 60- population, which has reached only 24% full vaccination. As a consequence, we are observing reduced ICU hospitalizations and reduced death rates for the 60+ segment, and growing rates in the 60- segment. Since the 60- represents 83% of the Chilean population, no wonder the overall numbers still look doubtful to some.

The second side of the discussion is whether the vaccine is effective or not ("effectiveness" here represents the reduction of a vaccinated person's probabilities of getting infected, getting hospitalized, and dying of the infection, as compared to the corresponding probabilities for a non-vaccinated person). This one is a bit trickier, but let's give it a try as well.

Chile has been mainly using the Chinese vaccine Sinovac (~ 90% of the shots), followed by Pfizer (~ 10% of the shots). Let's start with the "official" statement on vaccine effectiveness. There is a recent cohort analysis with results published by the Chilean Ministry of Health:

The English version of the press release here:

https://www.dropbox.com/s/585dwpnyzaz3jwp/Press%20Release%20Coronavac%20Vaccine%20Study%20ENGLISH.pdf?dl=0#

Spanish version:

https://www.minsal.cl/la-vacuna-coronavac-demostro-ser-efectiva-en-un-89-para-evitar-hospitalizaciones-uci/

Preliminary PDF report of the study (Spanish only):

https://www.minsal.cl/wp-content/uploads/2021/04/20210416_ESTUDIO-EFECTIVIDAD-CORONAVAC.pdf

Performed by Universidad De Chile, this study concludes that Coronavac has been 89% effective in preventing ICU hospitalization and 80% in preventing death. The Ministry of Health said they would update the analysis every month.

However, for many people, the "official statement" of the Ministry of Health is far from enough to believe that the vaccine is working, so I have performed an independent inference of the vaccination effectiveness by using a different source of data: the detailed data on deaths, hospitalizations, and infections, by age group, made publicly available on a daily basis by the Ministry of Science (https://www.minciencia.gob.cl/covid19/). This data is independent of the data used for the cohort analysis referred to in (1). Still, it has a problem: it is missing the information on how many infections, ICU patients, and deaths correspond to vaccinated patients. If we had that information available, it would be much easier to corroborate the results of the cohort analysis. Moreover, we could have the effectiveness of the vaccines monitored daily.

I took the data from the Ministry of Science and focused the analysis on the 60+ patients. To cope with the missing information, I made some reasonable assumptions (at least they seem reasonable to me :-) ):

(a) The ratios of daily infections and daily "cases" (i.e., positive PCR test results) of the 60+ over the 60- are equal and remain equal over the whole period of analysis, that is,

infections 60+ / infections 60- = cases 60+ / cases 60-

This assumption is important since we do not know the real daily number of infections.

(b) Under no vaccination, the %'s of daily infections, daily ICU patients, and daily deaths of the 60+ over the corresponding numbers of the total population remain stable.

(c) The time window Jan/1 to Feb/1 is a pre-vaccine ("pre") period because penetration of vaccination was still negligible by Feb/1.

(d) The date Mar/31 is a post-vaccine ("post") snapshot for the 60+ because the penetration of the 2nd dose was at 70,9% for that age group, and it is a pre-vaccine snapshot for the 60-, given that less than 10% of the 60- had received the second dose.

(e) Quarantines have little effect on the fraction of infections, ICU hospitalizations, and lethalities of the 60+ over the total population. Note that Chile has taken a local, multi-phase approach to quarantines: each town or comuna enters or leaves a phase (a "phase" represents a given level of intensity of the applicable restrictions) based on the evolution of its infection rates and other triggers, so along the analysis period there are multiple quarantine scenarios.

(f) The probability of death of a vaccinated person that gets infected, and the probability of death of a non-vaccinated person that gets infected, are both invariant.

As we can see on the following graph, assumptions (b) to (e) seem consistent with the observed data.

The key "enlightenment" here is that, given the assumptions above, one can express the fraction of infections (or "cases"), the fraction of ICU patients, and the fraction of daily deaths, of the 60+ over the total population, as functions of the ratio of probabilities of infection (x1), the ratio of probabilities of ICU hospitalization (x2), and the ratio of lethalities (x3), of the 60- over the 60+ population. So, given that we know the former three fractions, we have 3 equations to solve for the latter 3 fractions (i.e., for x1, x2, and x3). And we can do this for the pre-vaccine time window as well as for the post-vaccine snapshot.

The following table shows the result of this analysis. The last two columns include:

(i) The vaccine's impact on the ratios of infections, ICU hospitalizations, and death rates, of 60+ over 60- computed by solving for x1, x2, and x3 in both scenarios (pre and post) and comparing both values of x (post vs. pre);

(ii) The proxy for the vaccine's effectiveness, obtained by linearly scaling the penetration of the vaccination from 70,9% up to 100% on the 60+ population (an additional assumption made here is that the effectiveness of the vaccine is immediately perceived in the numbers, that is, with no delay, which is a conservative assumption, as it underestimates the impact).

The values of the fractions % Cases_60+, % ICU_60+, and % Death_60+, in the pre and post instances, used here to solve for x1, x2, and x3, may be read from the graph above:

Note that these numbers contain variances that are carried into the results, as may be clearly perceived in the graph. So results contain errors associated with these variances and with the assumptions made.

In conclusion, this analysis suggests that the effectiveness of the vaccine in Chile on the 60+ population is around 37% on infection, around 75% on ICU hospitalization, and around 92% on lethality (which means ~95% effectiveness on death), not far from the numbers published by the Ministry of Health for the whole population (note that regarding effectiveness on infection, the Ministry of Health considered only symptomatic patients while I am considering all infected 60+ patients here).

Despite the assumptions made in this analysis and the errors introduced by the variance of the data, these results look as relatively strong evidence that the vaccine is really working, at least in the 60+ population.

However, it is obvious that Chile needs to accelerate its vaccination pace again if it wants to finish as it started: like a racehorse and not like a race donkey.

Late notes: I have not mentioned the effect of people that have already recovered from the disease in the pre-vaccine and post-vaccine samples chosen. Patients that were not vaccinated and recovered during the observation period may introduce an overestimation bias in the effectiveness figures on infections and ICU hospitalizations estimated here. The total number of recovered patients is not available, however, we can look at those that tested positive ("cases") to get a lower bound. Between February 1 of 2021 and the end of March, the cumulative number of cases grew around 180.000, which is less than 1% of the total population. So, even if cases represented just 1/5 of the total infections, the resulting bias would not be substantial on the estimates of effectiveness.