How can you ensure that an oral communication skills assessment is both reliable and valid?

1 Define the purpose and scope

The first step to ensure reliability and validity is to define the purpose and scope of your oral communication skills assessment. What are you trying to measure and why? What are the learning outcomes or competencies that you expect from the candidates? What are the contexts and situations that require oral communication skills? How will you use the results of the assessment? By answering these questions, you can establish the rationale and objectives of your assessment, as well as the criteria and indicators that you will use to evaluate the candidates' performance.

2 Choose the appropriate format and method

The next step is to choose the appropriate format and method for your oral communication skills assessment. Depending on your purpose and scope, you may opt for different types of oral communication tasks, such as presentations, interviews, discussions, debates, role-plays, or simulations. You may also use different methods of delivery, such as face-to-face, online, synchronous, or asynchronous. You should consider the advantages and disadvantages of each format and method, as well as the feasibility and practicality of implementing them. You should also ensure that the format and method match the level and complexity of the oral communication skills that you want to assess.

4 Train and calibrate the raters

The fourth step is to train and calibrate the raters who will conduct and score your oral communication skills assessment. Raters are the people who observe and evaluate the candidates' oral communication performance, such as teachers, peers, or external experts. To ensure reliability and validity, you should train and calibrate the raters to ensure that they have a common understanding of the purpose and scope of your assessment, the format and method that you have chosen, and the rubrics that you have designed. You should also provide them with opportunities to practice and discuss their ratings, as well as to monitor and adjust their ratings if needed.

5 Pilot and review the assessment

The fifth step is to pilot and review your oral communication skills assessment before you implement it on a large scale. Piloting means testing your assessment on a small sample of candidates who are similar to your target population. Reviewing means collecting and analyzing feedback and data from the pilot, such as the candidates' performance, the raters' ratings, the candidates' and raters' perceptions, and the technical issues or challenges. By piloting and reviewing your assessment, you can identify and address any problems or gaps that may affect the reliability and validity of your assessment, as well as to improve the quality and usability of your assessment.

6 Evaluate and update the assessment

The final step is to evaluate and update your oral communication skills assessment after you implement it on a large scale. Evaluating means measuring and reporting the reliability and validity of your assessment, as well as the impact and outcomes of your assessment. Updating means making changes or revisions to your assessment based on the evaluation results, as well as the changing needs and expectations of your candidates, raters, and stakeholders. By evaluating and updating your assessment, you can ensure that your assessment remains reliable and valid over time, as well as to enhance the effectiveness and relevance of your assessment.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?