Can we trust the accuracy of FIFA-accredited position detection in football?

Undoubtedly, FIFA initiated something very important when they kicked off their Resource Hub and the EPTS Certification 7 years ago, in 2017. At the beginning “only” security related manners were evaluated, soon being followed by the performance test in 2019. Over the past years, FIFA continued to add more segments, such as Optical Tracking Systems, which are based on stationary cameras, Broadcast Camera Tracking Systems, and lower limb tracking more recently in 2023.

Many important decisions are based on positional and multiple services utilizeing positional data to enable add on services. Besides event data, positional data became the second major source of information in Performance Analysis.

Consequently, it makes total sense to assess the quality of systems producing this kind of data. Furthermore, it seems to be only natural to ask for reliable data, especially for sufficient accuracy of position tracking. This can reasonably only be achieved by validation studies conducted by a third party with an accepted methodology.?This article tries to evaluate whether the “FIFA stamp” is a valid indicator for the quality of tracking systems or not.

Was there any quality assessment before EPTS Certifications?

In the first decade of position tracking in football, there were only a few validation studies available, all of them with very poor methodological standards. ?Examples are the validation studies of ProZone [1] and Amisco [2].

A couple of years later we’ve seen new test designs being applied and setting new standards in the assessment of the validity of tracking systems [3]. Based on these studies the current gold standard was applied – using marker-based 3D position tracking systems working with infrared cameras like VICON or Qualisys, including a new study design [4].

Based on these and some other studies, FIFA designed the EPTS certification procedure.?

How does FIFA accreditation work?

The full accreditation is a multi-step process, which contains more than only the assessment of the reliability of the product and service. This article, however, only focuses on the so-called Performance Test. A description of the full process can be found here.

In general companies must apply and pay a fee to attend the Performance Test. Once attendance is confirmed, companies are informed 6 weeks prior to the test.

After the test is conducted and the company wants to use the FIFA certification they have to pay an extra fee. If they do not want to publish the results they do not have to.?

How does the test look like?

The test is conducted in a stadium following the official FIFA regulations. Furthermore, participants of the test (simulating players) are recommended to be at least 16 years old. Finally, the test is conducted on a 30 x 30m area in the centre of the pitch. Last but not least, systems using cameras are guaranteed with a sufficient height and a minimum 20° viewing angle.

The test itself consists of different runs and small-sided games. The handbook of the Test Manual can be found here.?

Why the test is far from ideal

The biggest restriction is the test space. To be fair, the limitation of 30 x 30 m test volume stems from the limitation of the gold standard used. It is inevitable, because the infrared light emitted by the cameras must be reflected by the markers and collected again by the cameras which is possible only over a limited range. Nevertheless, the consequences of this limitation do have a strong impact on the actual test meaningfulness as, for example, you never reach a top speed similar to a match in a small-sided game. Furthermore, difficult or at least challenging situations are avoided in FIFA’s test protocol, e.g. top-intensity sprints and accelerations including changes of direction, players leaving the pitch and re-entering it, or non-ideal pitch locations of the testing area. For computer vision-based systems, their performance under critical weather conditions (fog, rain, snow, dirty shirts) is not scrutinized.

In general, it seems that FIFA’s philosophy is rather to demonstrate the general capability for position detection of the accredited systems than to make them subject to strict testing under all conditions that may occur in practice.

Of course there is also a test efficiency criterion, but we are still talking about the top-professional soccer and FIFA, so there is still some room left, right???

Potential improvements

Set up multiple 30 x 30m areas to cover all pitch zones - including the "difficult ones". By this FIFA could validate, how the different locations on the pitch impact the quality of the data. Additionally, transitions from shadowy areas into sunny areas and reverse could be assessed, as well as the fact that 22 participants could simulate players, instead of a maximum of 10 during the 5 v 5. This would also evaluate the system quality in the most challenging situations like corner kicks.

Furthermore, referees, coaches and an audience could be positioned around the pitch to challenge (especially computer vision-based) systems regarding their robustness.

Simulating challenging weather conditions could be done via water sprinklers or artificial fog and snow or the test can take place in the evening when the sun is very low.

Is it really a problem?

Yes! Because providers use the FIFA certification to improve their chances of selling. And this is not acceptable as the test itself is not a valid instrument to assess the quality of a system in a real-world environment.

So, what now?

We need a valid testing procedure! Test procedures must be designed in a way that ensures that positions may be tracked reliably in sports practice (difficult locations, weather conditions, positions on pitch, difficult test exercises including high speed, high accelerations and sharp changes of direction).

Furthermore, we need a full-pitch gold standard! The best testing exercise is a full pitch match. Only here we have the kinematics of the “worst case scenarios” we find in real matches. This would allow a "ecological validity", finally.

Until the available methods are limiting this, the test protocol needs to be adjusted; some ideas have been shared above.

Summary

Accurate position tracking data is indispensable for many important tasks in performance analysis in football.

FIFA’s testing and accreditation procedure has at present deficits, and cannot guarantee valid position tracking for the needs of sports practice.

What is needed is, first, an appropriate testing procedure with a “worst-case scenario” testing philosophy and second, a full-pitch gold standard to allow for valid accuracy checks for practical purposes.

Only if these demands will be achieved, we will have reliable support for practical decisions. If not, we will have to accept error-prone positional data in the most interesting high-intensity situations and lacking comparability for example between tracking results from competition and training.

[1] Di Salvo, V., Adam, C., Barry, M., & Marco, C. (2006). Validation of Prozone ?: A new video-based performance analysis system.?International Journal of Performance Analysis in Sport,?6(1), 108–119. https://doi.org/10.1080/24748668.2006.11868359

[2] Zubillaga, A. (2006). Work-rate of elite soccer players – an analysis of variability. Doctoral thesis, University of Malaga (unpublished)

[3] Siegle, M., Stevens, T., & Lames, M.?(2013).?Design of an accuracy study for position detection in football.?Journal of Sports Sciences,?31(2), 166-172.?https://doi.org/10.1080/02640414.2012.723131

[4] Linke D, Link D, Lames M (2018) Validation of electronic performance and tracking systems EPTS under field conditions. PLoS ONE 13(7): e0199519. https://doi.org/10.1371/journal.pone.0199519