Why is it necessary to limit the time allocated to in-app questions?

Most Tosa Certification exams are a mix of multiple-choice questions (MCQ) and hands-on exercises, called in-application questions. The beauty of MCQs is that they allow you to ask a lot of questions in a limited amount of time, while in-app questions allow you to engage candidates by offering them a more interactive experience, while assessing specific skills that would be difficult to validate with an MCQ.

We regularly receive feedback from candidates who state, in the comment section at the end of their test, that the time allocated for each in-app question is too short, or that there shouldn’t be a time limit for the in-app questions at all.

I would like to explain why a time limit is needed for in-app questions and how we determine how much time should be allowed to complete an in-app question.

First, during Tosa’s in-app questions, when the candidate submits their answer, they immediately find out if they have answered the question correctly or not. If it is not correct, the candidate can edit their answer and submit again. This will be important in a moment.

Second, all Tosa certifications are adaptive. It means that the difficulty level of the test adapts to the candidate’s answers. The difficulty increases progressively as the candidate answers correctly and vice versa.

Therefore, at the beginning of the test the algorithm does not know what questions it is going to ask and so it is not possible to provide the candidate with the list of questions they will have to answer, which would have allowed them to manage their time based on what they know and don’t know. If there was no time limit for in-app questions, the risk that the candidate persists on one of the first in-app questions and runs out of time to complete the test would be high. We can say that it is very risky to let candidates manage their time if they don’t know ahead of time what questions they will be asked.

The third reason is even more important and refers to the methodology. If we gave an unlimited amount of time to a candidate to perform a task, while also telling them whether their answer is correct or not after each attempt, then, by trial and error, the candidate will always end up solving the exercise. On the other hand, if we gave a very short amount of time and allowed just one attempt to complete the task, then even skilled candidates run the risk of making a careless mistake and answering the question incorrectly. We can therefore see that the difficulty of a practical exercise depends almost entirely on the time allowed to complete the task. The same question can be very easy, or very difficult, depending on the time allotted.

The way we determine the right length of time to give a candidate to answer an in-app question is very important. It is far too important to be set without a well-defined methodology behind it. For instance, let’s take a situation where a candidate must sort data in Microsoft Excel – and suppose that we’ve allocated two minutes to answer the question. If the candidate completes the question in two minutes and one second, as opposed to one minute, 59 seconds, does that really mean that the candidate is not competent in a particular skill? Of course not!

That is why it is essential to use a scoring model. It allows us to set a reasonable amount of time, and let the calibration determine the statistical difficulty of the question. If the question appears to be statistically difficult, the candidates who succeed will likely have a high proficiency level, while failing this question will have a low impact on the final score. In short, we set a time limit and we let the model determine the difficulty level of the question and its impact on the candidate’s final score.

But how do we determine how much time should be allotted for a given in-app question? Our method is to set the time limit as one and half times the average length of time it took candidates to successfully answer the question. The IRT scoring model then determines the difficulty of the question once it has been exposed a sufficient number of times. We can then adjust the time limit based on the calibration result. If the question appears to be too difficult we can increase the time limit.

We see that it is necessary to set a time limit for in-app questions, first so the candidate can properly manage the total time allotted for the certification, and second, above all, to set a reasonable difficulty level to the in-app questions.

We also see that setting a time limit without using a mathematical scoring model is a perilous, or even arbitrary, exercise, since it can affect the candidate’s score for reasons that have nothing to do with their individual skill level.

Without such a model, we would not be able to set a time limit to practical exercises that is reliable enough to offer a certification with a real scientific validity.