Everything you want to know about Arbitrator Intelligence!

Today, most arbitration practitioners have heard about Arbitrator Intelligence. They have seen it referenced in this blog space, heard it mentioned at a conference, or noticed that it has been identified as an important new innovation. But as often as people say they have heard of Arbitrator Intelligence, they also still have basic questions about what AI is, what it aims to accomplish, and why it needs support from you.

This post answers those and other basic questions about Arbitrator Intelligence, or AI.

What is Arbitrator Intelligence?

Arbitrator Intelligence is a non-profit, academically affiliated entity. The mission of Arbitrator Intelligence is to increase transparency, accountability, and diversity in arbitrator selection by making key information about arbitrators’ past decisionmaking more generally available through publication of AI Reports.

Why is AI a Non-Profit?

Being a non-profit comes with many challenges, most notably limited financial resources. Despite these challenges, AI’s status as a non-profit permits it to commit to the larger goals in its Mission Statement (transparency, accountability, and increased diversity in arbitrator selection), rather than focus primarily on earning profits. As an academically affiliated entity, AI is able to benefit from the expertise of leading scholars (see below description of the AI Board of Directors that will develop AI Reports).

What is the AIQ?

The AIQ is an online survey administered on a secure website maintained by Penn State University. The AIQ facilitates systematic collection of feedback at the end of each arbitration regarding information about arbitrators’ case management and decisionmaking that has historically been only available through individualized person-to-person inquiries, usually over the telephone.

In developing the AIQ, AI employed state-of-the-art survey design (in coordination with the Penn State Survey Research Center), as well as extensive public and expert input. The ultimate goals of the AIQ are multiple and ambitious: to ensure quality feedback, to avoid questions that even implicitly preference certain cultures or legal traditions, to ensure fairness to arbitrators, and to promote systematic responses.

Why are there Two Phases to the AIQ?

The AIQ is divided into two phases to ensure that responders can provide feedback quickly, typically in less than 10 minutes per phase. Phase I concentrates on objective background information about the case, and can be completed by anyone who has access to the award or case file. At the end of Phase I, the responder will be asked to nominate one or more persons to take Phase II of the AIQ.

Phase II contains questions that are evaluative and, in some questions, request professional assessments. Phase II should be completed by an attorney or party who actively participated in the case. Certain background information from Phase I questions is used to prefill the questions in Phase II to make it even easier to complete Phase II.

How does the AIQ protect the parties’ confidentiality?

To protect parties’ confidentiality, the AIQ does not ask for information that would readily identify the case or the parties. Instead of these details, the AIQ asks for key data that will facilitate assessment and analysis of arbitrator decisionmaking and case outcomes in the absence of identifying information. For example, the AIQ asks for the date of filing, the industry in which the dispute arose, the date of the award, and arbitrator names. The AIQ does not ask for parties’ names or the names of law firms or lawyers. For security and quality assurance purposes, responders will be required to register in order to complete an AIQ, but their identities will be kept confidential and will not be published or otherwise connected with any AIQ responses.

How does the AIQ ensure that information provided will be relevant, accurate, and reliable?

The quality of information generated by the AIQ is key. For this reason, the AIQ was specially designed in consultation with the Penn State Survey Research Center, with other survey design experts, and after extensive public consultation with arbitration experts.

At a more substantive level, the AIQ includes questions that will facilitate qualitative assessment of responses and filtering of data from AIQ responses. For example, responders must identify their perspective (in-house counsel, outside counsel, etc.), their level of experience, and whether the award was more or less favorable than expected, etc. Moreover, questions that require subjective evaluation, such as questions addressing the extent of document production ordered by the tribunal, are asked only after responders provide responses to several objective questions. Responses can be assessed against the background information about the case, such as the amount in dispute, the governing law, the seat, or the industry in which the dispute arose. Responses can also be assessed for their consistency with the objective information derived from related responses on a particular topic.

Who can complete the AIQ?

AI aims to have a range of participants—including parties, their in-house counsel, and their outside or external counsel, and third-party funders—routinely fill out AIQs. Multiple responses from different participants in a single case are welcome as they will facilitate increased quality control and more nuanced data assessment. Arbitrators, arbitral institutions, and arbitral secretaries, however, are not invited to participate to avoid disclosure of any confidential information.

How will information from the AIQs be made available?

Once sufficient data are collected from responses to the AIQ, those data will be made available (usually for a fee) to parties, counsel, institutions, and arbitrators through “AI Reports.” AI Reports will be published by Wolters Kluwer. The specific features and content of AI Reports are still being developed.

When complete, AI Reports will provide analytics on a number issues on which information is collected through the AIQ. These issues include arbitrators’ historical practices on ordering interim measures, document production, and case management, the nature and quality of questions asked during hearings, interpretative methodologies in the reasoning of awards, and timeliness in issuance of awards.

Who is developing the AI Reports?

AI’s Board of Directors is overseeing development of the AI Reports and the software needed to generate them. The Board is composed primarily of university professors who collectively possess the essential range of expertise in relevant fields. Professor Chris Drahozal (University of Kansas Law School) is a leading scholar of empirical research on international arbitration. Professor Chris Zorn (Penn State Political Science) has experience in cutting edge data analytics in the legal profession, including as co-founder of Lawyer Metrics. Professor Scott Gartner (Penn State School of International Affairs) is expert in mass data collection (notably in his role in building the award-winning Historical Statistics of the United States) and strategic decision-making and conflict management. Professor Lee Giles (Penn State Professor of Information Sciences and Technology, Computer Science and Engineering, and Director of the Intelligent Systems Research Laboratory) is internationally recognized in search engine design and artificial intelligence. The newest addition to the Board is Johannes Fedderke, a South African economist who specializes in econometrics, is globally recognized for data analytics of complex phenomena in a range of fields.

How will AI ensure that AI Reports are fair to arbitrators?

The AIQ has a number of features designed to ensure that the information it provides is not only accurate, but also fair in its assessments of arbitrators. The primary assurance of fairness is that the overwhelming majority of questions asked in the AIQ seek objective information, not subjective assessments. Questions that do not seek purely objective responses instead as for “professional judgment” (not purely subjective assessments) about arbitrator specifically identified conduct and qualities.

Meanwhile, to promote accuracy, AIQ responses will be treated as anonymous, but responders will be required to register and verify their identities before completing the AIQ. Before any AI Report about an arbitrator is published, that arbitrator will be asked to consent to publication of the Report. Arbitrators will also have an option of withdrawing consent, in which case AI will no longer offer Reports on that arbitrator.

Meanwhile, AI is working to establish editorial policies to ensure that only AIQ responses that meet minimum indicia of reliability will be incorporated into AI Reports. Open-text feedback provided through AIQs will similarly be vetted through established editorial policies and generally published in summary form, similar to existing practices used by the International Mediation Institute and the Almanac of the Federal Judiciary for mediators and U.S. judges. AI is also exploring other means of ensuring that feedback is fair to arbitrators.

How can AI Reports be used?

Most obviously, AI Reports will be useful to parties and counsel in selecting party-appointed arbitrators. AI Reports will also be useful, however, in quickly assessing the opposing party’s nominated arbitrator, as well as any proposed or appointed chairpersons. Other data collected through the AIQ will provide useful information for case strategy planning and potential future reforms in international arbitration more generally.

Who supports AI?

Arbitrator Intelligence enjoys broad interest and support from around the world by arbitration practitioners who share AI’s belief that more information about arbitrators will improve international arbitration for all. In particular, young and diverse arbitrators see the value of breaking the information bottleneck that prevents newer arbitrators from developing reputations that will increase the likelihood of future appointments.

AI is also supported by its publishing partner, Wolters Kluwer, and by its Board of Advisors, which represents diverse leading perspectives from among in-house and external counsel, leading arbitrators, institutional representatives, and academics specializing in international arbitration. The Board’s members include: Chiann Bao, Gary B. Born, Stavros Brekoulakis, Charles N. Brower, Petra Butler, Yemi Candide-Johnson, Alan Crain, Stephen Denyer, Elisabeth Eljuri, Huascar Ezcurra, Babatunde Fagbohunlu, Juan Fernández-Armesto, Isabelle Hautot, Gabrielle Kaufmann-Kohler, Jeffrey Kovar, Deborah Masucci, Michael McIlwrath, Sundaresh Menon, Luke Nottage, Maria Irene Perruccio, Mirèze Philippe, Sundra Rajoo, Peter Rees, Cecilia Flores Rueda, Patricia Shaughnessy, Delissa Ridgeway, Frederico José Straube, Albert Jan van den Berg, Mohamed S. Abdel Wahab, David B. Wilkins, and Mathias Wolkewitz.

Why should you complete AIQs?

The biggest challenge AI currently faces is collecting AIQ responses, meaning collecting data needed to develop AI Reports. Without more responses to the AIQ, AI cannot generate AI Reports. In addition to collecting AIQ responses at the conclusion of each arbitration, AI is also seeking “retrospective” AIQs for arbitrations completed in the past 2 years (in exchange for free access to AI Reports for 2 years from when AI Reports become available).

To date, Gary Born and Wilmer, Cutler, Pickering, Hale & Door have committed to provide such retrospective AIQ responses. Several other firms are currently in discussions with AI to follow suit.

Completing AIQs, and particularly retrospective AIQs, involves a relative minor time-commitment compared with other pro bono activities. But by completing AIQs, however, you can contribute to the collectivized data needed to generate AI Reports, which will in turn ensure more and more equally accessible information about arbitrators, which will in turn promote transparency, accountability, and increased diversity in the arbitrator selection process.

* * *

Arbitrator Intelligence is an ambitious project that seeks to harness the collective intelligence of the international arbitration community. It can only be truly successful with your help, by completing AIQs. Please, join your professional colleagues today by taking a few minutes at the end of each arbitration to complete an AIQ or, better yet, committing to provide retroactive AIQs for those arbitrations you have participated in over the past few years. Your contribution will hasten development of the AI Reports and contribute to increase transparency, accountability and diversity in arbitrator selection.