Can we consider Kallisto Shield as an adversarial patch?

As we physically place in a scene a patch (our Kallisto Shield decoy) that is extremely salient to any neural network or Transformer-based algorithms, we can fool the computer vision algorithms into predicting a class encoded in the adversarial (physical) patch and/or completely deceive the algorithm increasing the Error Rate for the different classes predicted by the Classifier as seen on this video.

Creating an adversarial patch that can hide a vehicle across multiple sensing modalities, including visual, infrared (IR), thermal, synthetic aperture radar (SAR), and hyperspectral bands, involves several steps:

1. Understanding the Sensing Modalities

• Visual: Standard cameras capturing visible light.
• Infrared (IR): Cameras capturing thermal radiation.
• Thermal: Focuses on the mid-wave infrared (MWIR) and long-wave infrared (LWIR) bands, typically ranging from 3 to 5 micrometers (MWIR) and 8 to 14 micrometers (LWIR). These bands are used to detect heat emitted by objects, making them useful for night vision and detecting temperature variations.
• SAR: Radar imaging that can penetrate through obstacles like foliage and weather.
• Hyperspectral: Captures a wide spectrum of light across many narrow bands, providing detailed spectral information for each pixel.

2. Designing the Patch

• Visual: Use patterns and colors that blend with the background or create optical illusions.
• IR/Thermal: Use materials that alter the thermal signature, such as thermal insulation materials.
• SAR: Use materials that absorb or scatter radar waves, like radar-absorbing materials (RAM).
• Hyperspectral: Use materials that can alter the spectral signature across multiple bands, potentially using advanced coatings or materials that mimic the spectral properties of the background.

3. Optimization Algorithms

• Use machine learning algorithms to optimize the patch design. Techniques like gradient-based optimization can be adapted to learn the best shapes and locations for the patches.

4. Physical Implementation

• Materials: Select materials that can achieve the desired effects in each modality.
• Attachment: Ensure the patch can be securely attached to the vehicle without affecting its performance.

5. Testing and Iteration

• Test the patch in various conditions and angles to ensure it effectively hides the vehicle.
• Iterate on the design based on the test results to improve effectiveness.

Creating effective adversarial patches without access to the identification algorithm or the training data is challenging for several reasons:

1. Lack of Model Understanding: Without the identification algorithm, it’s difficult to understand how the model processes inputs and makes decisions. This understanding is crucial for crafting perturbations that can effectively mislead the model.
2. Absence of Training Data: Training data provides insights into the patterns and features the model has learned. Without this data, it’s hard to predict how the model will respond to specific perturbations, making it challenging to create patches that consistently fool the model.
3. Black-box Nature: In a black-box setting, where the internal workings of the model are unknown, attackers must rely on trial and error, which is less efficient and less likely to succeed compared to having full access to the model and its training data.
4. Generalization Issues: Adversarial patches need to generalize well across different inputs. Without training data, it’s difficult to ensure that the patches will work on a wide range of images, reducing their effectiveness.

So, the short answer is NO, Kallisto Shield is not an adversarial patch (although its top layer panels could be used to support interchangeable adversarial stickers). However we can argue that the introduction of independent Kallisto Shield decoys in the battlespace could impact error rates in vehicle detection systems due to:

1. Distraction and Confusion: Kallisto Shield decoys could introduce additional elements that vehicle detection algorithms need to process, potentially leading to higher error rates due to distraction or confusion.
2. Algorithm Sensitivity: Different algorithms have varying sensitivities to such decoys. For instance, traditional convolutional neural networks (CNNs) might be affected differently compared to more advanced architectures like YOLO10 or Transformers, which can capture more complex patterns and dependencies.
3. Feature Capture: Understanding what features are captured by these algorithms during training is crucial. Transformers might be better at distinguishing between real vehicles and decoys due to their ability to model long-range dependencies and contextual information.
4. Algorithm Adaptation: Exploring how algorithms can be adapted or trained to better handle such decoys could also be a valuable area of research. This might involve developing new training techniques or incorporating additional data that includes decoys.

Further research is indeed needed to quantify the impact of these decoys on error rates. This would involve controlled experiments with different types of decoys and various algorithm architectures to measure performance changes. Some of our simulations using YOLO8 and YOLO10 algorithms show a huge impact on the Error Rates for the decoys detection depending on the camera viewing angle:

1. For small angles (close to nadir) the error rate could reach a 25% mainly due to False Positives (vehicles covered with Kallisto Shield are identified as decoys by the algorithms).
2. For higher angles (60o) the error rates can reach 75% due to False Negatives (decoys are not detected by the algorithms).

We are focusing our research on how the introduction of Kallisto Shield decoys affects the error rates in vehicle detection systems. By collaborating with our European partners and utilizing synthetic datasets generated by our partner, the Ukrainian company QuData , we aim to:

1. Measure Error Rate Changes: Quantify how the presence of Kallisto Shield decoys influences the accuracy of vehicle detection algorithms.
2. Algorithm Comparison: Compare the performance of different identification algorithms, including traditional CNNs and advanced architectures like Transformers, in the presence of these decoys.
3. Feature Analysis: Investigate what features are captured by these algorithms during training and how they contribute to changes in error rates.
4. Robustness Testing: Assess the robustness of vehicle detection systems against these decoys and identify potential areas for improvement.

This research will provide valuable insights into the effectiveness of Kallisto Shield and help us develop more resilient detection systems.