GPT-Python Pulse: Mastering Cohen's Kappa with Python

AI is revolutionizing how we work, learn, and innovate—but understanding its practical applications doesn’t have to be overwhelming.

In today’s issue, we’re diving into Cohen's Kappa, a statistical measure for inter-rater agreement, using Python. Let’s break it down step-by-step, demonstrating its power in the context of radiologist evaluations.

What is Cohen’s Kappa?

Cohen's Kappa is a metric that helps measure the agreement between two raters or evaluators, like radiologists, on a classification task. Unlike simple percentages of agreement, Cohen’s Kappa takes into account the chance agreement, making it a more reliable measure. This is crucial when working in fields like medical imaging, where evaluating images consistently and accurately is paramount.

I recently asked ChatGPT to help me calculate Cohen's Kappa for a following task, with Python code to display the output at each step.

Here's how we can solve it step by step:

Step-by-Step Guide to Cohen’s Kappa Calculation

Step 1: Create a Contingency Table

The first step in calculating Cohen’s Kappa is constructing a contingency table. This 2x2 matrix holds the classifications made by two radiologists:

• A: Both agree on "Yes" (1,1).
• B: Radiologist 1 says "Yes" but Radiologist 2 says "No" (1,0).
• C: Radiologist 1 says "No" but Radiologist 2 says "Yes" (0,1).
• D: Both agree on "No" (0,0).

This table is essential for calculating the observed agreement and expected agreement.

Step 2: Calculate Observed Agreement

Observed Agreement is simply the proportion of times both raters agree, whether on "Yes" or "No". The formula for this is:

Step 3: Calculate Expected Agreement

Expected Agreement is based on the probability that the two raters would agree by chance. This is computed by multiplying the individual probabilities of choosing "Yes" or "No" for each radiologist:

Step 4: Cohen’s Kappa Formula

Finally, Cohen’s Kappa is calculated as:

This formula gives us a value that tells us how much better the two radiologists are agreeing compared to random chance.

Python Code to Calculate Cohen’s Kappa

Here’s the Python code to calculate Cohen’s Kappa based on the evaluations from two radiologists:

import numpy as np

# Given radiologist evaluations (Yes=1, No=0)
radiologist_1 = [1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0]
radiologist_2 = [0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0]

# Step 1: Create a contingency table
A = sum((r1 == 1) and (r2 == 1) for r1, r2 in zip(radiologist_1, radiologist_2))  # Both agree on "Yes"
B = sum((r1 == 1) and (r2 == 0) for r1, r2 in zip(radiologist_1, radiologist_2))  # R1 says "Yes", R2 says "No"
C = sum((r1 == 0) and (r2 == 1) for r1, r2 in zip(radiologist_1, radiologist_2))  # R1 says "No", R2 says "Yes"
D = sum((r1 == 0) and (r2 == 0) for r1, r2 in zip(radiologist_1, radiologist_2))  # Both agree on "No"

# Display the contingency table
print(f"Contingency Table:")
print(f"A (both agree on Yes): {A}")
print(f"B (R1 says Yes, R2 says No): {B}")
print(f"C (R1 says No, R2 says Yes): {C}")
print(f"D (both agree on No): {D}\n")

# Step 2: Calculate Observed Agreement (P_o)
P_o = (A + D) / len(radiologist_1)
print(f"Observed Agreement (P_o): {P_o:.3f}")

# Step 3: Calculate Expected Agreement (P_e)
p1_yes = (sum(radiologist_1) / len(radiologist_1))  # Proportion of Yes for radiologist 1
p2_yes = (sum(radiologist_2) / len(radiologist_2))  # Proportion of Yes for radiologist 2
p1_no = 1 - p1_yes  # Proportion of No for radiologist 1
p2_no = 1 - p2_yes  # Proportion of No for radiologist 2

P_e = (p1_yes * p2_yes) + (p1_no * p2_no)
print(f"Expected Agreement (P_e): {P_e:.3f}")

# Step 4: Calculate Cohen's Kappa
kappa = (P_o - P_e) / (1 - P_e)
print(f"Cohen's Kappa: {kappa:.3f}")
        

Output:

Another way of calculating Cohen's Kappa using SKLEARN:

from sklearn.metrics import cohen_kappa_score

# Given radiologist evaluations (Yes=1, No=0)
radiologist_1 = [1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0]
radiologist_2 = [0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0]

# Compute Cohen's Kappa
kappa = cohen_kappa_score(radiologist_1, radiologist_2)

# Print the result
print(f"Cohen's Kappa for Binary Classification: {kappa:.3f}")
        

Output:

Cohen's Kappa for Binary Classification: -0.200

Conclusion

By following these steps, you can calculate Cohen’s Kappa for any pair of evaluators and understand how well they agree, beyond simple chance. This is especially useful in fields like medical imaging, where accuracy and reliability are crucial. Cohen's Kappa gives you a solid, statistically sound measure of that agreement.

Stay tuned for more hands-on coding tutorials and AI-powered strategies in future editions of GPT-Python Pulse!