Structure Updates: Introducing the Structure Sensor 3

It has been almost two years since Structure spun off of Occipital. I have been mostly quiet about the many events that took place since then, but I assure you we have been busy. Structure's main accomplishments since the spin off include the following:

1. Build the team: from origins to value creation through innovation
2. Launch Structure SDK 3.0, including 3D reconstruction using 3rd-party sensors
3. Launch automated anatomic landmark detection and cloud processing (Beta)
4. Launch the Structure Sensor 3

In the next lines I discuss each one of these topics in a little more detail, but not too much: for brevity, I'll keep detailed discussions as topics for future updates.

Build the team: from origins to value creation through innovation

Those of you familiar with Occipital know that we were responsible for the original launch of the Structure Sensor, the Structure Sensor Mark II (and Pro versions), and the Structure SDK, thus bringing high-quality and easy-to-use 3D reconstruction within reach of iOS developers. Since my joining of Occipital, back in 2018, I was pleasantly surprised to verify how our technology was readily adopted by medical developers, especially in the orthotics and prosthetics (O&P) industry. Having worked in the medical equipment and athletic monitoring industries, I knew how difficult - and how valuable - it is to have a foothold in those segments, and how they can be readily disrupted by new digital technologies capable of providing patients with the unique benefits of personalized healthcare.

Seeing the potential value that could be unleashed by focusing on medical applications, Ravi Shah (CEO) and I (CTO) convinced Occipital to let us find investors interested in growing the hardware business with a medical focus. Thus, in August 2022 Structure was founded, initially counting with a small but dedicated team of 12 team members brought from Occipital [1]. In August 2023 the team doubled in size and since then we have grown to be 30-strong, with main offices in Boulder, Colorado, with offices across the US (Michigan, California, Nevada) and abroad (Europe and Asia).

Building a quality team takes time. Building a vertically-integrated team capable of developing value across all aspects of 3D reconstruction takes an incredible amount of time and dedication. Today Structure is one of the few companies in the World capable of designing 3D sensors that take into account the needs of Computer Vision (CV) Engineers while also counting with engineers who understand how the data used for 3D reconstruction and mesh processing was captured in the first place; who are capable of continuously improving the calibration and accuracy of those sensors, and of automatically processing the resulting mesh to provide clinicians with relevant information.

Launch Structure SDK 3.0, including 3D reconstruction using 3rd-party sensors

Structure started offering support to Apple's TrueDepth 3D sensor, in Beta, as soon as September 2022. Back then few examples of sample applications were available, and the SDK was only made available to a few key customers. It was in December 2023, with the launch of Structure SDK 3.0 [2], that the SDK was widely available to the public, including expanded support for Apple's LiDAR sensor and access to Structure's unique Scan Quality Indicator (SQI) - a feature especially developed with the O&P industry in mind, and a testament of the unique value of innovation via vertical integration.

Structure understands that different 3D scanners excel at different scanning modalities. Benchmarking with other solutions also demonstrated that Structure's 3D reconstruction algorithms continued to provide its customers with best-in-class tracking and accuracy, especially after the latest improvements designed to support 3rd-party sensors. Thus, it makes sense to embrace other sensors and provide value to our customers. Our vision continues to be to provide our customers with the best 3D scanning experience possible, regardless of the scanner they have available. Hence, we will continue to be attentive to other sensors that our customers find valuable, as we continue to strive to provide them with the best possible scanning experience for medical applications.

Launch automated anatomic landmark detection and cloud processing (Beta)

Structure always understood that vertical integration of its technology development was critical to provide value to its customers. For example, application developers do not simply want the accurate, calibrated 3D depth maps provided by their Structure sensors: they also want to be able to easily and robustly perform accurate 3D reconstruction using those depth maps. Taking that concept to the next level, Structure then understood that clinicians do not simply want 3D meshes that accurately represent their patient anatomy: they want actionable information that can be extracted from their patient's unique anatomy, thus enabling them to diagnose or treat medical conditions - a concept known as personalized medicine.

Example of extracting information from 3D meshes includes the use of 3D scanning to diagnose lymphedema [3] (a diagnostic application) and the development of custom-made compression garments to treat this medical condition (a therapeutic application). Another example includes the use of augmented reality (AR) to scan the patient anatomy and then register 3D data (MRI or CAT scans) over the patient anatomy, thus enabling pre-surgical planning of brain tumor removal [4], or intra-operative navigation during soft tissue tumor removal [5].

It was at the American Orthotics and Prostethics Association (AOPA) annual meeting, in Sept. 2023, that Structure announced the ability to perform automated landmark detection. Given the O&P focus of that meeting - and of Structure's customer base - the initial focus has been in the detection of anatomic landmarks in the patient's feet. These landmarks include not only the metatarsals (1st and 5th) and heel point (i.e., contact points especially important to the fabrication of orthotics), but also the distance from the foot arch to the best-fit ground plane. That last landmark has the particular ability to be detected using load-bearing and non-load-bearing feet, being thus capable of estimating the vertical translation of the navicular bone under those two conditions, being thus especially useful in the diagnostics of pes planus [6] (i.e., flat feet).

Anatomic landmark detection required processing beyond that capable in most portable tablets. Understanding this requirement, back in December 2023, Structure started making available to its customers (in Beta) the ability to upload patient meshes to the cloud, not only for post-processing, but also for hosting, thus facilitating the storage of patient data in a secure and HIPAA compliant cloud platform, and the sharing of this data with central fabricators.

Launch the Structure Sensor 3

As Structure strived to support 3rd-party sensors, it was clear that many medically relevant applications remained best served by Structure's line of sensors. These include applications in which it was important to know the accuracy and calibration of the sensor, and also applications in which the sensor architecture had to be known - and maintained constant - over generations. Also, in applications that required the high-resolution scanning of body parts larger than a face, such as whole head scanning, the scanning of torsos, limbs or whole bodies. That is to say, the demand for Structure's line of sensors not only remained solid over the years, but also presented the opportunity to grow by listening to the unique requirements of medical customers.

Unlike the Structure Sensor Mark II and Pro, developed with many applications in mind, the Structure Sensor 3 [7] focused on the special needs of medical customers, such as increased robustness and longer battery life. Freed from the requirement to serve drones and other light-weight applications, the Structure Sensor 3 benefits from the robustness provided by an all-metal chassis that can better withstand drops and the demanding physical requirements of a busy clinic, with minimum effect on its calibration and accuracy. That larger chassis also supports a significantly larger battery, enabling the Structure Sensor 3 to operate significantly longer before requiring a recharge.

The processing power of the dedicated graphic processor has also been significantly increased: not only has its clock rate almost doubled, the amount of memory available has also increased almost ten-fold. In addition, the new processor includes the ability to efficiently execute convolutional neural network (CNN) inference, thus unlocking the potential development of applications that benefit from the sensor's unique embedded AI abilities [8].

During the development of the Structure Sensor 3 we also took the opportunity to change the operating wavelength, from 825nm to 940nm, thus making the sensor more robust and less sensitive to ambient light. We did it while ensuring user safety by maintaining Class 1 laser classification - common to the entire line of Structure sensors.

New features will continue to be unlocked after launch, while some are already available. For example, confidence thresholding. That is to say, not all pixels in the depth map are equally valuable. So, this feature allows developers to select the level of filtering: by picking a value ranging from zero (all passing) to fifteen (all blocking), developers can trade off the number of pixels available in a given depth map for the number of pixels available. High dynamic range (HDR) is another feature that should be available as soon as at launch, or shortly thereafter. This feature is especially valuable when scanning objects with highly contrasting albedos. For example, a light object in a dark background, or vice-versa. Again, it makes the sensor more robust by enabling it to provide good scans over a wide range of applications while requiring fewer adjustments to presets.

Some features remained the same by design: the Structure Sensor 3 has nearly the same field of view (FOV) as its predecessor, thus ensuring the similarity of the scanning experience and, thus, no retraining. Another very important feature: the Structure Sensor 3 will be launched along with Structure SDK 3.1. That version of the SDK ensures forward-compatibility of existing apps with the new sensor. That is, developers can simply re-build their existing apps using the latest SDK, thus enabling backward compatibility with previously supported sensors, as well as future compatibility with the new sensor.

Conclusion

Structure has been busy building the team while continuing to deliver value to its customers. This value has been derived by focusing on medical applications while also expanding into vertical innovation. By developing excellence on all levels of 3D reconstruction, from building hardware to extracting information from the 3D meshes, Structure has been able to provide its customers with unique solutions that significantly unlock the potential of personalized medicine.

Stay tuned: we are just getting started.

References

[1] Occipital has since focused on home and room scanning applications, changing its name to Canvas: The easiest way to get as-builts | Canvas. (n.d.). Retrieved August 3, 2024, from https://canvas.io/

[2] Structure SDK 3.0 - Structure. (n.d.). Retrieved August 3, 2024, from https://structure.io/structure-sdk/

[3] Yahathugoda, C., Weiler, M. J., Rao, R., de Silva, L., Dixon, J. B., Weerasooriya, M. v, Weil, G. J., & Budge, P. J. (2017). Use of a Novel Portable Three-Dimensional Imaging System to Measure Limb Volume and Circumference in Patients with Filarial Lymphedema. The American Journal of Tropical Medicine and Hygiene, 97(6), 1836–1842. https://doi.org/10.4269/ajtmh.17-0504

[4] KIMS doctors in Hyderabad use Skia’s augmented reality, AI technology in 16 brain surgeries. (n.d.). Retrieved August 3, 2024, from https://www.newstap.in/city-news/kims-doctors-in-hyderabad-use-skias-augmented-reality-ai-technology-in-16-brain-surgeries-1530459

[5] Lee, M., Woo, J., Se, ·, Peak, H., Kim, H. G., Woo, ·, Lim, S., Chung, J., Jee, ·, Lee, E., Jeoung, ·, Kim, H., Park, S., Kim, J. M., Jun, ·, & Lee, W. (2024). An exploratory clinical trial of preoperative non-invasive localization before breast-conserving surgery using augmented reality technology. 206, 31–44. https://doi.org/10.1007/s10549-024-07272-3

[6] Sp?rndly-Nees, S., D?sberg, B., Nielsen, R. O., Boesen, M. I., & Langberg, H. (2011). THE NAVICULAR POSITION TEST – A RELIABLE MEASURE OF THE NAVICULAR BONE POSITION DURING REST AND LOADING. International Journal of Sports Physical Therapy, 6(3), 199. /pmc/articles/PMC3163999/

[7] Structure Sensor 3 - Structure. (n.d.). Retrieved August 3, 2024, from https://structure.io/structure-sensor-3/

[8] What are the differences between Structure Sensor Pro (ST02B) and Structure Sensor 3 (ST03)? - Structure FAQs. (n.d.). Retrieved August 3, 2024, from https://support.structure.io/article/453-what-are-the-differences-between-st02-and-st03