FiftyOne Computer Vision Aggregation Tips and Tricks — Nov 25, 2022
Authored by Jacob Marks , ML Engineer/DevRel at Voxel51
Welcome to our weekly FiftyOne tips and tricks blog where we give practical pointers for using FiftyOne on topics inspired by discussions in the open source community. In this Thanksgiving week installment, in the spirit of togetherness, we’ll cover?aggregations .
Wait, What’s FiftyOne?
FiftyOne ?is an open source machine learning toolset that enables data science teams to improve the performance of their computer vision models by helping them curate high quality datasets, evaluate models, find mistakes, visualize embeddings, and get to production faster.
Ok, let’s dive into this week’s tips and tricks!
An Aggregations Primer
Datasets ?are the core data structure in FiftyOne, allowing you to represent your raw data, labels, and associated metadata. When you query and manipulate a?Dataset?object using?dataset views , a DatasetView object is returned, which represents a filtered view into a subset of the underlying dataset’s contents.
Complementary to this data model, one is often interested in computing aggregate statistics about datasets, such as label counts, distributions, and ranges, where each Sample is reduced to a single quantity in the aggregate results.
The?fiftyone.core.aggregations ?module offers a declarative and highly-efficient approach to computing summary statistics about your datasets and views. Continue reading for some tips and tricks to help you do just that.
One Aggregation; Multiple Datasets
If you want to compute a single aggregation on a single dataset, you can call the aggregation method directly on the dataset. For instance, to compute the?Bounds ?aggregation on the?uniqueness?field, you could write:
dataset.bounds("uniqueness")
However, if you want to use the same aggregation method (on the same field) on multiple datasets or views, you can define the aggregation on its own, and then compute that aggregation using the?aggregate()?method:
mean_agg = fo.Mean("predictions.detections.confidence"
mean_value1 = dataset1.aggregate(mean)
mean_value2 = dataset2.aggregate(mean))
Learn more about the?aggregate ()?method in the FiftyOne Docs.
Multiple Aggregations
Conversely, if you want to compute multiple aggregations on the same dataset, not necessarily on the same field, you can do so efficiently by batching them in the?aggregate?method:
# will count the number of samples in a datase
sample_count = fo.Count()
# will retrieve the distinct labels in the `ground_truth` field
distinct_labels = fo.Distinct("ground_truth.detections.label")
# will compute a histogram of the `uniqueness` field
histogram_values = fo.HistogramValues("uniqueness")
# efficiently compute all three results
aggs = [sample_count, distinct_labels, histogram_values]
count, labels, hist = dataset.aggregate(aggs)t
Learn more about?batching aggregations ?in the FiftyOne Docs.
Unwinding Lists of Lists
If we want to compute aggregations that are not built into FiftyOne, such as the median, we can do so by first extracting the values for the relevant field and then applying our aggregation to this result. Due to the unstructured nature of computer vision data, where different samples may contain different numbers of detected objects, this result may be a list of lists of differing sizes.
For instance, if we get the prediction confidence values,
pred_conf_field = "predictions.detections.confidence
pred_confs_jagged = dataset.values(pred_conf_field)"
we can see that the first ten sublists all have different lengths by running:
print([len(p) for p in pred_confs_jagged[:10]])
If we wanted to compute the median from this jagged list of values, we would first need to flatten the list. However, FiftyOne does this for us when we pass the argument?unwind = True?into the?values()?method:
pred_conf_field = "predictions.detections.confidence
pred_confs = dataset.values(pred_conf_field, unwind=True)"
From there, we can pass the resulting flat array straight to numpy:
median_conf = np.median(pred_confs)
Learn more about unwinding and?values ()?aggregation in the FiftyOne Docs.
Aggregations on Transformed Field Values
When we use the FiftyOne?Aggregations?class in conjunction with?ViewField, we can easily perform aggregations over transformed field values. For instance, to compute the mean of squared prediction confidence values, we can write either of the following:
## Option
aggregation = fo.Mean(F("predictions.detections.confidence") ** 2)
squared_conf_mean = dataset.aggregate(aggregation)
## Option 2
squared_conf_mean = dataset.mean(F("predictions.detections.confidence") ** 2))1
Learn more about?expressions ?and?ViewField ?in the FiftyOne Docs.
Going Beyond Aggregations
While aggregations over an entire?Dataset?or?DatasetView?are powerful, sometimes they are not sufficient to fully understand your model or your data. In fact, it is precisely this need for better transparency during data-model co-design that led to the creation of the FiftyOne App and FiftyOne Teams. Beyond dataset and view level aggregations, FiftyOne provides class-specific reports for classification and multi-class object detection techniques via the?print_report()?method.
For multi-class detection tasks, this looks like:
results = dataset.evaluate_detections
"predictions",
gt_field="ground_truth",
eval_key="eval",
compute_mAP=True,
)
# Get the 10 most common classes in the dataset
counts = dataset.count_values("ground_truth.detections.label")
classes_top10 = sorted(counts, key=counts.get, reverse=True)[:10]
# Print a classification report for the top-10 classes
results.print_report(classes=classes_top10)(
For a binary classification task, this looks like:
results = dataset.evaluate_classifications
"predictions",
gt_field="ground_truth",
eval_key="eval",
method="binary",
classes=["classA", "classB"],
)
results.print_report()(
Learn more about?print_report ()and?evaluating detection ?and?classification ?tasks in the FiftyOne Docs.
What’s next?