"FingerGesture" making progress

My leisure time project "FingerGesture" is making progress.

What is it?

It is a finger ring with optical sensors in order to realize my finger positions. It should be used to realize finger and hand gestures.

A finger ring with optical sensors (here as ToF sensors, 3x, or it could be also implemented with small cameras), should "see" all my other finger movements and positions. In addition with an IMU sensor, which should measure the "arm rotation" (the rotation of the hand, wrist, with the fingers, twisted), I can orient my fingers in a 3D sphere.

This is used to realize finger and hand gestures. Use it in order to control and navigate in software applications (instead of a mouse, with full 3D awareness and without external means, like a stationary camera watching my hand and fingers from above).

How does it work?

The optical Time of Flight sensors (ToF) will watch all the neighboring fingers. I use (for a test/demo) the VL53L5CX ToF sensors. They should "see" with the 8x8 optical sensor, how all my other fingers are positioned (in relation to the finger ring).

The sensors send the data via WiFi (using a Portenta C33) to a host. This one acts as the "hub", mounted on my arm (or used like a wrist watch). The IMU sensor will realize any twisting of my arm (and therefore of my hand and fingers). The "hub" will forward the pre-processed data (e.g. converted into a 3D sphere vector format) to a final host (evaluating and using the finger and hand gestures).

Is it working?

In general - yes, it works. Just the minimal distance of the sensors is pretty large. I wish to have a better resolution on very small distances (I do not need 4000mm distance).

See a first test here: https://youtu.be/PPOBycXjrQ0

I have to look for sensors which have a better resolution on smaller distances. I am thinking to try BROADCOM APDS-9500 (with a 60X60 view) or use real cameras (like OV6948). The idea is around optical sensing, used on my finger(s) itself, without external, stationary means.

Objectives

For my leisure time project I want to get more familiar with ML, AI, doing math in a 3D sphere (just to learn for myself). The HW should be a "piece of cake". My interest is more in learning how to use sensors, with ML, AI (and doing all the math).

I want to "see" my fingers in relation to my "reference finger" (where the "RingGesture" is mounted) and realize as many different finger and hand gestures as possible.

This can be used in a VR application (like my flight simulator) in order to realize what I want to do (e.g. turning a knob, click on a button, lift a handle (flaps)...)

This project is "just for fun" (for myself), not intended to be launched as a product at the end.