How Should We Interpret Test-Retest Variability in Conflict Mode Assessments?

I’m publishing this newsletter series to discuss the nuances of the four timeless topics for people and organizations: conflict, change, transformation, and consciousness. You can subscribe to never miss an article.

There are test-retest guidelines when validating any assessment tool: Given the nature of the theory behind the construct, do you expect the construct to be rather stable over time (such as an enduring personality trait) or do you expect it to change in the short run (such as a behavioral choice in a conflict situation)?

If an assessment’s test-retest correlations change in the short run when the theory suggests that the construct is supposed to be an enduring trait, then the validity of the assessment is questionable. In a similar way, if the theory suggests that the results of the assessment should change when test conditions change (for example, from low stress to high stress situations) and yet the test-retest correlations remain very stable, then the validity of the assessment is also questionable. Thus, an assessment can be considered valid (or, better yet, fail to be considered invalid), only when the test-retest results are totally consistent with the theory behind the construct.

In terms of the TKI , since this assessment claims to measure behavioral choices (whether conscious or not), test-retest correlations are not expected to be stable in the long run, especially when people are subjected to conflict awareness training and other educational experiences that expose them to alternative choices (and, even more so, give them an opportunity to PRACTICE making different choices). Since the “first” administration of the TKI provides awareness training (especially when the scores are interpreted and discussed in a workshop or coaching setting), there can already be changes when the TKI is administered just a few hours or a few days later. Indeed, the heuristic power of the TKI is that the results and the interpretive materials CATCH the person, wake up the person, to alternatives that hadn’t been consciously realized before. There can be test-retest changes in the TKI without any other intervention than giving a person the assessment!

As a case in point, in the 1970s, the TKI was often used as a before-and-after comparison for weekend assertiveness training programs. Not surprisingly, the test-retest comparisons typically showed that people’s scores on competing, collaborating, and compromising (the three modes that involve assertiveness) became significantly higher — as a result of a one-day or weekend workshop.

When Ken Thomas and I collected data in the early 1970s to examine the test-retest properties of the TKI, we chose a period of two weeks. We had no reason to expect that our convenience sample of graduate students would be taking an awareness or assertiveness training programs during that time frame AND we did not score, interpret, or discuss the student’s TKI results until AFTER both test-and-retest administrations were complete. For that two-week time period, the test-retest correlations were moderate to high…consistent with the theory of conflict modes with no intervention. But if those same test-retest results had occurred if the students had received some conflict mode training…the validity of the TKI would have been questionable.

In sum, it is meaningful to collect test-retest data with the TKI whenever you have conducted an educational program or coaching session to provide greater awareness about choosing conflict modes, let alone giving people practice in changing their use of conflict modes. Since the TKI has been validated in many diverse settings over several decades, if the TKI scores do NOT change in the test-retest period (in predictable directions, such as more assertiveness, more cooperativeness, more balance of conflict modes…whatever the program was attempting to bring about), then I might question the effectiveness of the workshop — only because the measurement has already been shown to be fairly reliable and valid.

Kilmann Diagnostics offers a series of eleven recorded online courses and nine assessment tools on the four timeless topics: conflict management, change management, consciousness, and transformation. By taking these courses and passing the Final Exams, you can earn your Certification in Conflict and Change Management with the Thomas-Kilmann Instrument (TKI). For the most up-to-date and comprehensive discussion of Dr. Kilmann’s theories and methods, see his 2021 Legacy Book: Creating a Quantum Organization: The Whys & Hows of Implementing Eight Tracks for Long-term success.

Photo by The Climate Reality Project on Unsplash