Other Forms of Assessment

A widely misunderstood part of the OQ Rule can be found in Definitions 192.803(e) and 195.505(e) regarding how operators should use “other forms of assessment.” This is sometimes construed to allow "talking through" or pretending to perform part of or even all of performance evaluations (PE).

While you can’t blame people for trying to find a way to conduct evaluations more quickly, this is clearly not an acceptable use of “other forms of assessment.”

I’ll explain why, in my opinion, and we’ll consider PHMSA’s view as found in their Operator Qualification Frequently Asked Questions.

In my experience, any opening in this area will be initially used to skip a step or two and will eventually progress to talking through the entire evaluation for employees the evaluator knows well, or someone who has decades of experience, or when the correct equipment isn’t readily available, it’s raining/snowing, etc. It’s one of those doors that, when opened a little, tends to get thrown completely open.

You may have heard the horror story I tell about a contractor individual who was evaluated to perform hot taps 2” or less with TDW equipment by an evaluator who brought the individual into a hotel conference room and asked the individual to imagine a piece of 2” steel pipe on the table. He told the individual to imagine a T-101 tapping machine there as well. Then, the evaluator asked the individual to describe and pretend to perform the steps he would follow to perform the imaginary tap on the imaginary pipe. Eventually, the individual was told he had passed the PE and was now qualified to perform hot taps for the operator. I doubt any operator would be comfortable with an individual assessed this way performing hot taps for them.

“Other forms of assessment” should be limited to company-approved and OQ Plan-defined assessment methods. Not other defined assessment methods watered down. Despite the challenges of implementing this method, I understand why PHMSA allows it, which is because the regulatory process moves so slowly. It is intended to encourage innovation by providing room for operators to create new assessment methods without the need for a regulation change before these new methods can be used. Virtual Reality is one such method. Operators can create, validate, and implement new assessment methods without requesting a code waiver.

Those are my opinions, which are simply that – one man’s opinions. More persuasive guidance can be found in the PHMSA OQ FAQs.

OQ FAQ #14 asks: “What are acceptable evaluation methods, and what is observation of an individual?”

• In its answer, PHMSA says, “The evaluation methods selected must be appropriate for the covered task.”

Meaning if the operator has determined a performance evaluation (PE) is appropriate for the covered task, then that is the method that should be used. Talking through any steps of the PE rather than performing the steps becomes assessing for knowledge of those steps, not assessing the performance of the steps. In effect, the operator is not following their OQ Plan when they allow talking through.

• PHMSA further states in the same answer: “Operators must be able to describe the evaluation method(s) associated with each covered task in their written OQ programs used to determine whether an individual is qualified to perform that task.

Meaning any evaluation methods used must be included in the operator’s written program. Simply listing the Code language: “other forms of assessment” in the written OQ program, without defining what those other forms are and how/when they are permitted, doesn’t meet this guidance.

• Again, in the same FAQ answer: “In order to determine an individual's ability to perform a covered task, observation must include methods of assessing the individual's knowledge of the task, as well as the individual's ability to perform it.”

Meaning that evaluations must include both methods of assessing knowledge and their ability to perform the covered task. Talking through any steps of the PE defeats using a PE to assess ability.

PHMSA’s OQ FAQ #15 asks: “What capabilities should be evaluated to qualify an individual to perform covered tasks?”

• In its answer PHMSA states “The qualification process, whether for an initial qualification or a requalification, should address the following factors:

(1) the individual's knowledge of the task (e.g., knowledge gained through self-study, classroom training, or computer-based training);

(2) the individual's skill in performance of the task (e.g., craftsmanship in performing the steps of the task);

(3) the individual's ability (e.g., proficiency/physical capability, to include vision, strength, or agility; or mental comprehension and understanding) to perform the covered task; and

(4) the individual's ability to recognize and react to an abnormal operating condition.”

We see from PHMSA”s response that the regulatory expectation is that, regardless of the methods used, both initial qualification and requalification should include all of the items listed in (1) through (4). Again, talking through any steps of the PE removes the operator’s ability to assess (2) and (3).

As you can see if an operator intends to use “other forms of assessment,” they need to define what these other forms are, as well as how and when they may be used while ensuring knowledge, skill, ability, and the individual’s ability to recognize and react to an AOC will be adequately assessed using these other methods. This information must also be described in the operator’s written OQ Plan.

Just listing “other forms of assessment” in an operator’s OQ Plan without this descriptive information will lead to many pointed questions during regulatory inspections and investigations.