What you need to know about FRR and FAR

In a?previous blog post, we mentioned the terms FRR (False Rejection Rate) and FAR (False Acceptance Rate) in connection with the accuracy of biometric authentication systems. Now we will talk about these values in more detail.

What are FRR and FAR?

To measure the effectiveness of an authentication system, we can calculate the False Rejection Rate (FRR) and the False Acceptance Rate (FAR), which are the two types of errors that a system can make.

FRR?(also called Type I error)?is the likelihood that a legitimate user will be rejected by the system; for example, a 10% FRR means that the system will reject 1 out of 10 genuine login attempts on average.

FAR?(also called Type II error)?is the likelihood that an impostor will be accepted by the system; for example, a 10% FAR means that the system will mistake 1 out of 10 impostors for a genuine user on average.

This means that?the lower the FRR and FAR values are, the better. An ideal biometric authentication system would have?0% FRR and 0% FAR, however, this?is impossible to achieve.

In reality, if we want to achieve a?lower FAR, we will have to settle for a?higher FRR, and vice versa. The point where the?FAR and FRR values are equal is called equal error rate?(EER) or crossover error rate (CER).

Balancing security and usability

When developing an authentication system, we have to consider what is important to us.?Do we want to minimize the number of unauthorized users that the system lets in,?or do we want to make sure that it is as easy for genuine users to get authenticated as possible?

In general, we should try to?find a good balance between security and usability, but sometimes one is much more important than the other.

For example,?if our system is used to secure high-level classified information, we need to aim for a very low FAR?since giving access to unauthorized people would come with serious consequences. Authorized users might need multiple attempts to access the system in this case, but that seems irrelevant compared to the cost of sensitive information getting into the wrong hands.

On the other hand,?if we are talking about something like unlocking a smartphone, a low FRR might be more important?since we do not want to frustrate users by rejecting them too often when they are trying to unlock their phones. A positive user experience is worth more in this case than trying to avoid unauthorized access as much as possible, especially since an intruder attempt on someone’s smartphone is a rare event in general.

A biometric authentication system works by assigning a score between 0 and 1 to authentication attempts. This score represents the likelihood that the attempt is genuine, meaning that a score of 0 should definitely be rejected, while a score of 1 should definitely be accepted.

Based on the above considerations,?we need to decide on a threshold value between 0 and 1. For example, if we set the threshold to 0.8, then attempts with a score between 0.8 and 1 will be admitted. The lower this threshold is, the more people will get admitted – so?if we want to achieve a lower FRR, we need to lower the threshold, and?if we would rather have a lower FAR, we need to raise the threshold.

If we are not sure?what the best approach would be when developing our system, we might want to?optimize for the CER to be as low as possible?(so that the FRR and the FAR are equal).?

What are “good” FAR and FRR values?

Ideally, a biometric authentication system would always be sure (and correct) about whether an authentication attempt is genuine or not, so it would assign either 0 or 1 to all attempts. In this case, changing the above-mentioned threshold value wouldn’t matter, the FRR and FAR values would both be 0% regardless. But like we said, this is impossible to achieve. What?we need to aim for when developing our system is that the scores of genuine and impostor attempts are as well separated as possible.

What is considered?a “good” FAR or FRR value highly depends on the particular application?that we are talking about. Going back to our previous example of a system that secures high-level classified information, even a 1% FAR does not sound acceptable since such a system might be heavily targeted by intruders, and one out of a hundred intruders getting in could cause great damage. In a lot of cases, though, 5-10% FRR and FAR values might be considered good enough.

You might come across reports of very low?FRR or FAR values, such as 0.0001% or even lower; be wary of such claims?since even just to be able to measure such small error rates would require an unrealistically huge sample size, not to mention that the methods used to calculate these extremely low error rates are often scientifically incorrect.

For example, in the absence of hundreds of thousands of real samples,?data augmentation might be used to generate more samples from the existing ones, but error rates measured on augmented samples cannot be considered trustworthy.

It is also important to mention that?for measuring FAR and FRR, we need samples that we have not used before during training and calibration; and in the case of FAR, we even need to make sure that these samples are from completely new users that our model has not seen yet.

Results achieved with too small sample sizes can be misleading, too; just because the system works well on a few people’s carefully selected samples does not mean it will work well in the real world, where a large number of people will use it.

An example calculation

Let’s say that we have a?database of electronic signatures from 100 users,?and for each user separately, we built and calibrated authentication models using a part of this database. We have exactly?5 signatures left from each user to test our model’s false rejection rate, which means that we can get 100 x 5 =?500 genuine scores in total. If we find that (for example)?20 of these scores are below the?(user-specific)?thresholds, then that?means?that?the FRR?of our system?is?20 / 500 =?4%.

To calculate the FAR, we need users that our model has not seen yet (during training and calibration). Let’s say that for every model,?we have 20 such users with 20 signatures each. This means that we can generate 20 x 20 x 100 =?40000 impostor scores. If we find that (for example)?1000 of these scores are above the?(user-specific)?thresholds, then that?means?that?the FAR?of our system?is?1000 / 40000 =?2.5%.

As it is apparent from this example, the number of samples we have will affect how accurately we can measure these values.?The more users and the more signatures per user we have, the better, both for training the authentication models and for measuring the FAR and FRR values more accurately.