Why Outcomes Studies Are Flawed - The Primary Care Example

There are numerous flaws in health outcomes studies. One major problem is that various populations within the United States have a wide variation of better and worse outcomes. They also tend to have better and worse situations and conditions. Population based studies have problems due to these variations. Regression based research studies also have numerous limitations specific to primary care as helping to shape health care outcomes. If the advocates get what they want, they could get more primary care attention and emphasis - but this might not improve the levels of primary care where most Americans have half enough.

If you can understand why primary care levels are not likely to shape better outcomes, you might also understand that most health outcomes studies are seriously flawed. Read on to look at some of those flaws.

This does not mean that primary care improvement efforts are futile. It does mean that we need better studies. It also means that we need to focus on improvements of populations. We must change how our nation and its people and politicians value most Americans behind in so many designs. We also need them to value basic health care services.

An emphasis on primary care alone is not going to be sufficient to significantly improve health outcomes. Research studies appear to show a linkage between more primary care and better outcomes. These regression studies have serious and basic research method flaws (apples to oranges, multicollinearity, heteroskedasticity, biased across collection, analysis, interpretation, others). Implementation of primary care emphasis is often not specific to increasing primary care levels where most Americans have half enough. Primary care is a smaller portion of Basic Health Access and levels could be increased without increasing basic health access if general specialist shortages were not addressed or if primary care delivery team personnel are impaired by innovative or regulatory designs.

Summary of this Post

1.     Apples to Oranges Flaw - Some studies are flawed because the two populations being compared are quite different – as with rural vs urban or high vs low volume hospital outcomes. Such a flaw should have resulted in outright rejection. Instead these studies are published, even by major journals.

2.     Correlation is not causation – Regression equations do not indicate causation. Only association is implied, and only if the correlations are strong. Social sciences correlations are rarely strong and usually explain little of the variance. They will always be suspect. Multimethod research and randomized double blind interventions come closer to causation and “proof.”

3.     Heteroskedasticity - Regression equations can demonstrate significant differences in the variance explained, but can be invalid. This happens as the result of comparisons of means. Populations are not well-represented by means. And when the two populations being compared have large variations in the values of the variables, it is not valid to compare the means. Such a comparison will tend to increase the variance explained – making it look more significant than it is. As Jha and others point out, means testing works out only for the small 10% of the population that has variable values close to the mean. In other words, if you put too much credit in equations that appear to demonstrate that primary care levels shape outcomes, you would be supporting the same equations that demonstrated that non-white Americans were inferior in intelligence. Regressions were very popular originally, and it took decades to figure out how they were flawed. Apparently this learning has been lost as we accept various health outcomes studies without understanding apples to oranges flaws, correlation is not causation, and variations within populations compared.

4.     The primary care deficit is just one factor that is associated with the populations left behind in the US that have had numerous deficits in many other areas across social determinant and other factors far more likely to influence population outcomes.

5.     The few social determinant-variables used as controls in outcomes studies are insufficient. They do not capture the conditions, situations, relationships, and environments of the patients. Frankly we know that some of these influence outcomes in some people, but we do not understand which population is influenced, or how, or how to translate this into a research design. Health outcomes studies just insert proxy income or other data into the equation even if not representative of most of the population. People are dynamic and so is health care. Flat variables do not capture this.

6.     Access to care is likely more important than traditional office based primary care. Primary care is a smaller part of access to care albeit and important one. The populations behind in primary care are behind across generalists, general specialists, public health, hospital access, pharmacies, health care dollar investments, and more. No studies have tried to parse these out. But primary care researchers have specifically looked for primary care relationships – and have found them. There is bias in the intent, the designs, and the interpretations.

7.     Increases in primary care levels alone, may or may not improve health access. On the plus side is that increases in dollars invested where needed could impact jobs, social determinants, and eventually outcomes. On the negative side is that increases in primary care did not involve increases in funding more specific to social determinants – and outcomes. Primary care needs tens of billions more to level up. Health care outcomes needs hundreds of billions more.

8.     Emphasis on primary care may not result in improvements in the primary care levels relevant to the populations in most need of primary care. Politicians, CMS, and even primary care associations have found ways to emphasize primary care without increasing primary care levels where needed. There are far too many layers between emphasis and more and better primary care delivery team members. The current time period gives us massive examples regarding emphasis that results in dollars not really specific to the emphasis area. Political actions and lobbying by corporations and associations distorts intent. Administration and management costs eat away at dollars invested.

Apples vs Oranges Flaws Are Common – Comparisons of Two Populations with Many Differences Between the Two Population

 Correlation is Not Causation – Flawed Interpretation

 “Correlation tests for a relationship between two variables. However, seeing two variables moving together does not necessarily mean we know whether one variable causes the other to occur. This is why we commonly say “correlation does not imply causation.”

A strong correlation might indicate causality, but there could easily be other explanations:

·        It may be the result of random chance, where the variables appear to be related, but there is no true underlying relationship.

·        There may be a third, lurking variable that that makes the relationship appear stronger (or weaker) than it actually is.”

The above quote from https://www.jmp.com/en_us/statistics-knowledge-portal/what-is-correlation/correlation-vs-causation.html

There are not even strong correlations. And the researchers inject the primary care variable to look for primary care influence. The controls used are few compared to life influences impacting populations.

I think that it is quite obvious that there are dozens if not hundreds of “third lurking variables” that share collinearity with primary care levels and with health outcomes.. The same is true with education levels and education outcomes.

Note that the outcomes studies done are population based – and are not specific to individual patients and their individual social determinant factors. No one can state that there is a direct causal link.

None of these major studies have direct indications of the relationship to primary care. Some studies measure continuity of care – somewhat. So there is a gap in this direct relationship.

When the researchers turn to population based studies, they introduce major challenges. If you say that the US has major differences shaped by their zip code – you can see that the zip code does not shape outcomes. But the life situation forces living in such a zip code. And within the zip code there are vast differences in outcomes within the zip code. In my zip code reviews of various data, there were amazing differences across adjacent zip codes, especially states such as Florida.

Most people do not understand population based studies or the problems when people use geographic locations (county, zip code, state, nation) with income and other proxy variables inserted into the equation. As an editor of Rural and Remote Health, it was very common to have proxy variable studies. It was almost as common for the researchers to ignore or minimize the problems associated with proxy variables. Consider that at the current time, it is not popular to point out that many studies that insert the race or ethnicity variables, are flawed because of proxy variables. This does not mean that the problem does not exist, and is major.

Regression Equations Also Have Other Problems – Particularly in Population Based Studies

5 | Problems and Issues of Linear Regression  https://methods.sagepub.com/book/understanding-regression-analysis/n5.xml

• Specification.
• Proxy Variables and Measurement Error.
• Selection Bias.
• Multicollinearity.
• Autocorrelation.
• Heteroskedasticity.  https://en.wikipedia.org/wiki/Heteroscedasticity
• Simultaneous Equations.
• Limited Dependent Variables.

Understanding heteroskedasticity is important to understanding the misleading Race Vs Intelligence studies over 100 years ago. These studies often have major flaws with the variations within the populations compared are greater than the differences demonstrated in the study. In other words in a nation such as the US that has major variations within a population, the small change findings associated with primary care and outcomes are more likely to be spurious when comparing two populations. Proxy variables make this more likely.

Note that there are many lurking variables as Starfield reviews the evidence on primary care and outcomes

Milbank Q. 2005 Sep; 83(3): 457–502. doi: 10.1111/j.1468-0009.2005.00409.x

PMCID: PMC2690145 PMID: 16202000

Contribution of Primary Care to Health Systems and Health

Barbara Starfield, Leiyu Shi, and James Macinko

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2690145/

From the abstract

“On the basis of the studies reviewed in this article, we believe that health of the U.S. population will improve if this maldistribution is corrected. Specifically, a greater emphasis on primary care can be expected to lower the costs of care, improve health through access to more appropriate services, and reduce the inequities in the population's health.”

My issue is with her wording.

1.    A greater emphasis on primary care alone will not change costs and inequities in population health, although it would likely improve access if and only if the primary care emphasis resulted in half of the US having ready access to enough primary care. This would be a change from half of the US population with half enough. The United States is also unique in creating and maintaining populations that have sufficient primary care in close proximity to people that lack health access. They simultaneously have barriers of income, transportation, and health care insurance design.

2.    Note her words “improve health through access to more appropriate services.” There is some connection between ready access to health care and better health. Once again these are correlations. Note that better access is not specific to primary care. Note that generalists and general specialists along with basic hospital services are deficient locally for half of the population. The problem is far more than primary care across basic, cognitive, office, mental health, women’s health, and basic surgical. Our nation does not value any of these. Primary care is only part of this. An emphasis on primary care alone will not address the other deficits.

3.    More emphasis on primary care or on primary care where needed is not likely in our nation that does not value primary care and that also does not value most Americans clearly left behind. We cannot even get one of the few remaining primary care associations to focus on specific primary care funding. We have entirely new primary care associations that have been created and get grant funding and donations – but they have done little other than to promote primary care medical home and micromanagement – but they clearly have a name and claim to primary care emphasis

4.    Note how emphasis on primary care at AAFP involves value based and working closely with CMS. This decades long bandwagon with enormous resources devoted to influencing CMS has continued without improvements in primary care levels. The AAFP would say that it has an emphasis on rural health. We know that this emphasis has not played out in staffing, resources allocated, and more. Where is there an emphasis at AAFP on the urban underserved counties – about 32% of the urban population with half enough primary care.

Commonwealth is a foundation that indicates a mission for access to care - but supports insurance plan expansions of insurance plans that are often meaningless for access to care. It also supports metrics, measurements, micromanagements, and value based care - which distract primary care delivery team members, disable primary care finances, and discriminate against providers that serve populations most behind in access and numerous other social determinant and outcome areas - inherently.

5.    There are many problems with generic promotions of "more primary care emphasis" – which is commonly associated with so many other areas. The families of family medicine often fail for family medicine specific solutions. Academic, social, and other interests divert attention from financial solution emphasis.

6.    Associations and Foundations Fail to Focus on Primary Care Finances - Note that the various state primary care reports were delayed in emphasis and in development and in publication for decades – and so far they have failed to result in insurance paying more to primary care except perhaps in a few states (2 or 3) where this may be happening.

• Do you really think that the 30+ states most behind will enforce laws forcing higher payments for primary care by insurance, given that insurance companies have highest value to legislatures and government?
• Do you see how the designs hurt the smaller, independent providers doing the basic services where needed?

7.    Note how more emphasis on economic recovery has been distorted to result in more concentrations of wealth. It is possible that a generic emphasis on primary care could also result in more primary care where primary care is concentrated – the theme of internal medicine.

• Yes, internal medicine primary care is among the more concentrated specialties in the US while also being least likely to be found in primary care. This has never stopped claims about being primary care or contributing to places with deficits of primary care. Do you see how the primary care distortions are endless.
• Each time NHSC or HRSA or AHRQ or CHC comes up for funding, there is primary care emphasis. But none of these have resulted in actual improvements in primary care levels in the populations most behind. Each of these programs have primary care emphasis, but have other areas of emphasis.
• Note how emphasis on primary care has been distorted to non-primary care specialties. NP and PA and DO and MD have promised primary care solutions, but graduates decline to lower and lower proportions going to and staying in primary care.
• Workforce deficit programs such as Teaching CHCs are diverted to fund programs least likely to result in improvements in workforce where needed – less than 20% were correct in all 3 key areas associated with distribution where needed– family medicine as the specialty, training location in a state in need of workforce, training location in a county in need of workforce.
• Political deals are made to get primary care emphasis programs passed – without actually doing much for primary care. And as we have discussed, training fails to do much when the financial design shapes fewer and lesser delivery team members where most Americans most need care
• Federally designated shortage areas have primary care emphasis – but clearly have failed to address primary care deficits and the designations also include locations where primary care levels are not in deficit. Family medicine departments and programs have used loopholes such as the Medicaid without ready access loophole, to help get FQHC designations. Note that the failure of primary care and family medicine training emphasis is the reason. Note that the Medicaid design with poor payment is what shapes the deficit. Note that these FQHCs are located in counties that have higher to highest levels of primary care. We do not even have primary care emphasis addressing shortage area reform – to be sure that areas in most need of primary care levels can get them.

From the Starfield evidence review

There is a body of evidence reviewed stating the numerous assumptions discussed, but in my opinion she buried the lead. The main body involves the international comparisons. Her text is quoted then we can discuss her findings and associations with regard to international comparisons.

International Comparisons

"International comparisons extended our examination of the impact of primary care according to the achievement of its characteristics. Studies of the characteristics of different health systems were particularly useful because they enabled us to assess the impact of various policy characteristics on the practice and outcomes of primary care. Three studies, one using data from the mid-1980s and two from a decade later, demonstrated not only that countries with stronger primary care generally had a healthier population but also that certain aspects of policy were important to establishing strong primary care practice.

The first study examined the association of primary care with health outcomes through an international comparison conducted in 11 industrialized countries (Starfield 1991, 1994). Each country's primary care was rated according to the four main characteristics of primary care practice: first-contact care, person-focused care over time, comprehensive care, and coordinated care, as well as family orientation and community orientation. Policy characteristics were the attempts to distribute health services resources equitably (according to the extent of health needs in different areas of the country); universal or near-universal financial coverage guaranteed by a publicly accountable body (government or government-regulated insurance carriers); low or no copayments for health services; percentage of physicians who were not primary care physicians; and professional earnings of primary care physicians relative to those of other specialists. (Operational definitions of these indicators and the method of scoring them are described in Starfield 1998.) The first important finding is that the score for the practice characteristics was highly correlated with the score for the policy characteristics. That is, the adequate delivery of primary care services was associated with supportive governmental policies. The second finding is that those countries with low primary care scores as a group had poorer health outcomes, most notably for indicators in early childhood, particularly low birth weight and postneonatal mortality.

A more recent comparison, with 13 countries and an expanded set of indicators of both primary care policy characteristics and health outcomes, also showed better health outcomes for the primary care–oriented countries even after controlling for income inequality and smoking rates, most significantly for postneonatal mortality (r = .74, p < .001) and rates of low birth weight (r = .38, p < .001). Countries with weak primary care also performed less well on most major aspects of health, including mental health, such as years of potential life lost because of suicide (Starfield and Shi 2002). The positive impact of primary care orientation on low birth-weight rates may reflect a beneficial effect of primary care on mothers’ health before pregnancy (Davey Smith and Lynch 2004; Starfield and Shi 2002). The characteristics of primary care practice present in countries with high primary care scores and absent in countries with low primary care scores were the degree of comprehensiveness of primary care (i.e., the extent to which primary care practitioners provided a broader range of services rather than making referrals to specialists for those services) and a family orientation (the degree to which services were provided to all family members by the same practitioner). The most consistent policy characteristics were the government's attempts to distribute resources equitably, universal financial coverage that was either under the aegis of the government or regulated by the government, and low or no patient cost sharing for primary care services (Starfield and Shi 2002). The latter two were studied and confirmed by Or (2001).

The positive contributions of primary care to health also were found in a much more extensive time-series analysis of 18 industrialized countries, including the United States (Macinko, Starfield, and Shi 2003). The stronger the country's primary care orientation (as measured by the same scoring system as in the earlier international comparison) was, the lower the rates were of all-cause mortality, all-cause premature mortality, and cause-specific premature mortality from asthma and bronchitis, emphysema and pneumonia, cardiovascular disease, and heart disease. This relationship held even after controlling for various system characteristics (GDP per capita, total physicians per 1,000 population, percentage of elderly people) and population characteristics, including the average number of ambulatory care visits, per capita income, alcohol consumption, and tobacco consumption. The analyses estimated that increasing a country's primary care score by five points (on a 20-point scale) would be expected to reduce premature deaths from asthma and bronchitis by as much as 6.5 percent and that the reduction in premature mortality for heart disease could be as high as 15 percent."

My Comments on the Starfield Review

Even a brief review indicates that there are major differences between the various nations with regard to policy, system differences, and healthier populations in countries with stronger primary care. The US remains the major outlier in health insurance coverage. The US and the UK are worst among developed nations for the child wellbeing ratings. The US invests one-third to one fourth as much of GDP on age 0 to 6 spending (UNICEF studies). The US actually punishes the poor, especially poor families from the very start of the family unit all the way to the few that retire and the many that die too soon.

To me this indicates that nations and states that value people and basic services and designs for better health are the ones that focus on shaping population improvements. They also tend to invest more in primary care, education, social support, child development, and other funding that distributes dollars and income. As a contrast the US leads in income inequality, destruction of the middle class, gini indices worsening, income quintile distributions worsening, and worsening outcomes for most Americans. It is difficult to find a health outcome marker that has not been worsening in the places and populations that have been falling further behind in many areas. Education, economic, and other policies also make matters worse.

• Will emphasis on primary care address these inequities and disparities that shape worsening outcomes?

It is difficult to break this down and say that primary care is the reason for improved outcomes. The other factors shape the life influences for every minute awake dating back to birth and before.

Why is it so easy to believe that a few minutes exposure to primary care a year shapes outcomes - out of 350,000 minutes of life influences while awake during a year, much less during a previous lifetime?

Situations, conditions, environments, and other life influences have not been measured. These third party factors likely have influence and are clearly related to levels of primary care and outcomes.

The US Designs Concentrates Those Doing Well in Higher Concentration of Health Care Workforce, Best Economics, and Best Outcomes.

But the US concentrates the poor, elderly, disabled, Native, Veteran, mentally ill, and other populations most behind in the 2621 counties lowest in health care workforce – especially primary care.

• The more that you understand about regression equations, the less credit you are likely to give to primary care for health outcomes improvements.
• The more that you study disparities involving so many health, education, economic, and other indicators – the less you will see primary care as the solution – especially with regard to population based outcomes.
• You might just see that cuts in Social Security, Food Stamps, Disability, Veterans Benefits, child development, basic education – will worsen outcomes.
• You could also see how the primary care emphasis on primary care medical home and value based designs actually will likely worsen outcomes because of the billions less being sent to the counties most behind.

And if you understand how the cost of living and housing drives people away from places with sufficient to higher concentrations of workforce, you can see how they are being forced to move to counties lowest in health care workforce, social supports, and outcomes. And as Americans grow older, and poorer and have more debts for medical and other reasons – they will have no choice in the matter.

Bob Bowman

Basic Health Access