Calculating Cut Points for CAHPS measures

Now that we've reviewed the calculations for how star cut points are determined for the non-survey measures (if you haven't read that article yet, you can find it here), let's move on to the the survey measures.

The methodology used to calculate the cut points for the CAHPS measures is called the relative distribution and significance testing method. This one starts off relatively straight forward and then gets a curve ball thrown in. The relative distribution part is easy. If you plot everyone’s scores, you’ll get a familiar distribution curve. That curve can be divided up into percentiles, where a specific score will find a specified percentage of the groups at or above it and the remaining percentage of the groups below it. The 50th percentile score would have 50% of the groups with scores at or above that number while the other 50% of the groups would be below. The 2-star cut off is at the 15th percentile mark, meaning that, 15% of groups have a score lower than that score (resulting in a score of 1-star) and 85% of groups have that score or better. The cut off points for the remaining stars are 30th percentile for 3-stars, 60th percentile for 4-stars and the 80th percentile for 5-stars. So far, so good.

Now come the complicated parts. First of all, they don’t just use the raw scores. They adjust the scores based on the contract’s case-mix. Remember, here we’re talking about the CAHPS survey data, so there can be differences in the characteristics of enrollees across contracts that could impact their responses. The variables they adjust for include age, dual eligibility status, presence of the low-income subsidy (LIS) indicator, use of the Asian language survey, and self-reported education, general health status, mental health status, and proxy usage status. Each one of those variables has an associated coefficient representing how much higher or lower people with that characteristic tend to respond to each question compared to a reference group. To me, this is similar to how RAF scores are calculated where the presence of a condition is given a risk value compared to baseline and someone with multiple factors will have their RAF adjusted by the sum of those values. The difference is that in this case, some of these adjustments are positive (meaning that group tends to score the question higher, so you need to adjust the final score down to compensate) while some of the characteristics are associated with a lower (negative) score (meaning you need to adjust the final score up to compensate). Here’s an example of those coefficients for the C19 (Getting Needed Care) measure with the baseline reference group being patients between the ages of 70-74 who are high school graduates, rate their general and?mental health as good, didn’t need help filling out the survey and weren’t a dual, receiving the LIS or completing the Asian language survey. Populations with those characteristics will be shown as having a 0.0000 adjustment.?

You may be saying that this isn’t so complicated after all, but wait, this is CMS we’re talking about. Once again they pull out their SAS software and calculate the “reliability” of each score. From what I can tell, the reliability score represents how variable the responses are for a given survey question between physicians under that contract. If you’re into this kind of thing, CMS refers to this tutorial by Rand to explain. If the reliability rating is “very low” (<0.60), the contract doesn’t receive a star rating. The reliability is considered to be “low” if there are at least 11 respondents and the reliability score is between 0.60 and 0.75. They also designate the 12% of contracts with the lowest reliability score as having “low reliability” regardless of the absolute score. One last step is to calculate, for each measure score, if it is statistically significantly different than the national average CAHPS measure score. So, in order for a contract to get a specific star rating in a given measure, not only do they consider which percentile it falls into, they also asses if the score has low reliability and if it is statistically significantly different than the national average. Just so you can see how crazy this is, here is a table reflecting the assignment rules:

To summarize, the relative distribution and significance testing methodology used for CAHPS measures involves the following key steps:

1. Case-mix adjustment
2. Defining percentile cut-points (the current year’s distribution of case-mix adjusted contract mean scores is used to set percentile cut-points that are rounded to the nearest integer between 1-100).
3. Contracts are then grouped into “base groups” relative to their cut-points.
4. Stars are then assigned based on which group the contract falls into, if their score is statistically significantly different than the national mean (and the direction of that difference), the statistical reliability (based on between-contract variability) and the standard error of the contract’s mean score.?