ICAI MUST REINVENT THE EXAMINATION PROCESS

V Pattabhi Ram

The Institute of Chartered Accountants of India (ICAI) is both the certifying body and the regulator for the audit profession. Its role as a regulator has often drawn flak. Its responsibility as a certifier is now under pressure.  The system needs adjustments, lest it should suffer a serious setback.

Before I continue, here are my credentials. I am a proud member of the Institute and have spent a large part of my adult life in teaching besides being engaged in CA work. I have therefore no ax to grind, and yet I don't want the seriousness of this article to be lost in the noise of misguided jingoism. 

What I write is what I would like to see in a model situation in a professional course. If some of these are already followed by ICAI, that is wonderful.

SETTING PAPERS

Exam questions, both at the Intermediate and at the Final levels, should be original. Each time an examiner cannot pick questions from study materials, past exams or identifiable sources. If a problem has appeared four times in the last eight years with numbers modestly altered, it only means we have got our priorities wrong. And I believe that going by history, in the last ten years we have our priorities misplaced.  If an odd item is to be so sourced and then tweaked, there should be 100% certainty that there are zero errors. For example, we should not have a question on inflation with inflation rate missing or one on portfolios with the risk-free rate carrying a beta of 0.99 w.r.t Sensex.

The syllabus is already extensive and to expect a 20-year-old to know everything under the sun is unfair. The concepts tested should have been discussed in the study material. If those who wrote the study material did not find an idea significant enough to be addressed, why should we make it a part of testing? In short, we should not test the candidate on what we have not taught him under the wrong thinking that learning has to be unlimited. The aim should be to keep the questions smart, not predictable. The objective should be to find out what the candidate knows, and not what the candidate does not know. 

The inclusion or exclusion of topics in the Law and Tax papers, based on statutory changes made effective before a threshold of six-months, is excellent. But upgradation of the syllabus (and study material) on similar lines for those and other papers has to be carried out very early. It’s essential that the views of the members and students are heard, considered, and disposed of suitably. The portal that provides for feedback needs to be managed efficiently, lest people take to the social media to ventilate views when it sees the ICAI turning a deaf ear. 

QUESTIONS BANK:  

Let us create a pool of questions and guideline answers round the year. These can be outsourced. Each question in each paper should get vetted (including language editing) by at least two persons before it goes to a data pool. Let these men and women be either academically strong or have rich, relevant industry background. Once the items go to the data pool, they reach the stage of finality.  There should be two paper-setters for every exam who will agree on the sub-topic to be tested.  After that allow the computer to select the relevant questions from the database. This is a process that will take time to implement, but we must begin now. Ensure that we pay everyone at above market rate and hold them accountable. 

Next, let's remember that how we test a student has a bearing on how he learns. If we repeat questions and if the issues become predictable, rote learning will become standard practice.  If rote reading happens, we are doing a disservice to a generation that will work in an era where the premium is on rapid learning and innovation

Typically, the exam paper at Intermediate, be it Quantitative or Theory should have: (a) Multiple choice questions that test concepts (30 marks); (b) Short problems (30 marks); (c) Case lets (20 marks); Theory questions (20 marks).  Having 30% of the paper on the MCQ mode is welcome. On MCQ, it would be perfectly all right for each student to get a different paper. Modifying the sequence of questions and the series of answers can actually help convert 15 raw questions into about 930! 

At the Final Level, the intensity of the questions should be stronger than what it is at Intermediate. However, question papers should be such that a good student is able to complete the paper in given time. Today, in certain subjects, it is virtually impossible to do so.  The Case-lets should test practical knowledge imbibed in the Internship. Would it not be interesting to offer the candidates writing an audit paper a few documents and ask him to identify whether there is a fraud?  

The paper must cover the entire syllabus. If the question is found to have an error, conceptual or otherwise, if an answer has a mistake, theoretical or otherwise, we must hold the person who set it accountable. An odd mistake once in ten years is understandable.  

We should ensure that the questions are of international standards and the answers are not obvious. Examinations should be run every quarter with a student having the option of writing only the alternate exam. This will reduce the workload involved in each exam. The ultimate goal would, of course, be to let a candidate take an exam anytime when he is ready but that - I think - is far into the future. 

I would even like to have the system to allow students to progressively move from one group to another only after they finish a particular group. Like, a student to take Group 2 of Intermediate should have completed Group 1 and similarly to take Group 2 of Final should have completed Group 1 of Final. In short, he takes only one Group at a time. This will ensure he gets a finer grip on the subject.  

VALUING PAPERS

We must get into a central valuation of exam papers. We have surely moved to an era of work from home, or work from anywhere. But, valuation of exam papers requires a conducive environment and 100 percent dedication to the job.  This can happen only when the valuer drops everything, comes to a central place ` and carries out the task. Yes, this would cost, but it's a cost worth taking for the cause. Only those who are in touch with the subject must value papers. Let's not get into the unrealistic assumption that a CA knows everything and can do anything.  If for example, I specialize in valuation in my profession and do nothing with taxes, it would be utterly unfair for me to value a direct tax paper merely because I had once passed the CA exam and now have a guideline answer to guide me through the evaluation.  Valuers should be subject-experts without any compromise.

Today, marks are scaled up for all candidates based on exigencies. There may be a variety of reasons for this, into which we do not have to now get into. But let's not forget that this is the era of transparency.  We need to be up to the mark.  Courts are unlikely to take tardiness in their stride. Raw scores are now visible to the judges, as also to the students as a photocopy of answer scripts are made available to them. Further, the institute should take ownership of the guideline answers shared with valuers or students.

Suffice to say, to avoid the anomalous situation of scoring not being uniform across exams and across subjects, the system of relative grading (a k a Grade Point Average) should be adopted. All business schools and all engineering colleges have transitioned to this with success. While life is not about the standard distribution curve, GPA will help iron out the troublesome case of marking. Next up, consider using Artificial Intelligence to grade essay questions. The Graduate Record Examination (GRE) uses AI to grade essays, and thus removes the human bias and tedium inherent in the lengthy drawn evaluation.  

AUTONOMY 

There may today be a Chinese wall between the Board and the Examination department, but so long as the two report to the same head, there will be a source of conflict.  Most importantly, the examination body should be spun out of the office of the President and of the ICAI’s Standing Committee. An independent CEO, who could be elected or selected exclusively for this purpose and holds a 5-year term, must head it.  This should be a highly paid job handed out to a person of proven competence and integrity. No other elected member of the council should be a part of the body.  This CEO will keep the President posted only on a need-to-know basis. 

Results should be announced within three weeks. One way of doing it is running quarterly exams. A second would be to add evaluators. Before valuation, the evaluators should be walked through the answers in a workshop.  This should be done after a few papers have been test-valued to understand how some answers and assumptions are different from the guideline answers. It would also be an idea to have at least two persons evaluate each answer paper, rather than a supervisor carrying out a random review of papers being assessed. The first examiner will value the first three questions and the second examiner will value the next three, across all papers that they value. This will bring about a sense of greater efficiency to the process. This is how internationally valuation in the CFA exam works.

I have not spoken of actual cases of commissions and omissions, lest it is sensationalized. The objective is that we move with changing times and uphold the integrity, sanctity, and thoroughness of our exam process, which increasingly has turned opaque.  We should strive to make the examination and evaluation process enviably robust to ensure it is world-class.

First published in KSCAA

Pattabhi Ram is a member of the ICAI. His debut novel Ticking Times is set in the backdrop of the audit profession.