Weaknesses of Health and Fitness Devices
Patrick E. Sewell MD
Gene and Cell Therapy Specialist, Advanced Interventional Oncologist
I have tried a number of activity monitors like OURA, Apple watch, Fitbit, and have always been disappointed with the accuracy. I decided to analyse the hardware and software, then wrote a short report, as well as a summary like Cliff Notes. See both below. My impression in real life and from the research - i have enough variables and abnormalities in my life that the results seem to be garbage about 80% of the time or on 80% of the parameters i look at. The analysis gives me plenty of reasons why my impression of inaccuracy is accurate.
Weaknesses of Health and Fitness Devices
Key Sensors Commonly Found in Modern Fitness Trackers and Smartwatches
1. Core Motion Sensor
2. Physiological Sensor
3. Environmental Sensor
4. Galvanic Skin Response (GSR) Sensor
5. Proximity Sensor
6. Gesture Sensor
Core Motion Sensors
- Accelerometer: Measures acceleration, speed, and movement in three dimensions to track steps, activity levels, and orientation.
- Gyroscope: Detects angular velocity, rotation, and precise orientation changes.
- Magnetometer/Compass: Works with GPS to determine direction and precise location.
Known Core Motion Sensor Limitations
Technical Limitations
- Step count accuracy typically varies with a 5-10% discrepancy compared to pedometers.
- Heart rate monitoring becomes less reliable during high-intensity exercises.
- Quick movements, especially of lower body parts, may not be properly captured.
- Real-time performance may suffer when running on mobile device CPUs.
Conclusion: Compounded inaccuracies in the Motion Sensor are evident!
Algorithm Issues
- Difficulty differentiating between different types of movements.
- Challenges in accurately identifying niche or unusual exercises.
- Limited accuracy in detecting complex movement patterns.
Conclusion: Algorithms are very narrow at this time, with poor extrapolation.
Environmental Factors
- Skin tone can affect heart rate sensor accuracy due to light penetration issues.
- Temperature variations can impact sensor performance.- Poor GPS reception in certain locations affects tracking accuracy.
Conclusion: Skin pigmentation and external environmental temperature negatively impact optical sensor
User-Related Issues
- Incorrect device placement or loose wearing significantly impacts data accuracy.
- Vigorous arm movements can lead to overestimated step counts.
- Sweat levels can interfere with sensor readings.
- Activities like cycling or pushing strollers may not be accurately captured because of arm position.
Conclusion: Device placement, unusual or unfamiliar physical activity and arm position significantly negatively impacts data collection, assignment, and interpretation.
Physiological Sensors
- Heart Rate Monitor: Uses optical sensors (PPG) or ECG to measure heart rate and pulse.
- ECG Sensor: Measures electrical signals from heart activity for detailed cardiac monitoring.
- SpO2 (Pulse Oximeter): Measures blood oxygen saturation levels.
- Skin Temperature Sensor: Monitors body temperature changes.
- Bioimpedance Sensor: Measures body composition, hydration levels, and respiratory rate.
- Electrodermal Activity (EDA) Sensor: Measures skin conductance for stress monitoring.
Physiological Core Sensor Limitations
Accuracy Issues
Heart Rate Monitoring
- Error rates of ±3% even under optimal conditions.
- Decreased accuracy during high-intensity exercises.
- Optical sensors struggle with rapid heart rate changes.
- Performance varies based on skin tone and light penetration.
- Sweat levels can interfere with sensor readings.Conclusion: The act of exercise decreases the sensor accuracy in a near linear fashion it seems. Its great at telling you how well you are exercising when you are at rest, but if you are exercising, the accuracy of exercise assessment drops as your exercise increases.
Sleep Tracking
- Overestimates total sleep time and sleep efficiency by more than 10%.
- Underestimates sleep onset latency.
- Error ranges from 12% to 180% compared to clinical sleep studies.
Conclusion: Completely inaccurate sleep tracker.
Activity Tracking
- Step counts typically underestimated by about 9%.
- Energy expenditure calculations show significant variability.
- Difficulty accurately capturing non-standard activities like cycling or pushing strollers.
Conclusion: Estimated steps are off by 10% and energy/calories used varies greatly in all activities, people, situations, and activity choice.
Technical Limitations
Data Access
- Limited access to raw sensor data for research and development.
- Proprietary algorithms restrict data interpretation.
- Lack of standardization across devices.
Conclusion: Future algorithms are of questionable accuracy. Formulas built on mistaken data are worthless.
Device Placement
- Incorrect wearing position significantly impacts accuracy.
- Loose fitting affects data quality.
- Dominant hand placement can lead to overestimation of activity.
Conclusion: Sensor position impacts accuracy and data collection.
Environmental Factors
External Interference
- Temperature variations affect sensor performance.- Poor GPS reception impacts location-based tracking.
- Ambient light can affect optical sensor readings.
Conclusion: Measurements outside of a statice environment are subject to questions of accuracy. In the Lab, you are golden. In real life, your data accuracy sucks.
Clinical Considerations
Reliability Issues
- Rapid device obsolescence hampers validation studies.
- Less than 5% of consumer wearables have been clinically validated.
- Lack of standardized testing methodologies across research.
- Variable performance across different user demographics.
Conclusion: The device market and the data collected is composed of apples and oranges on both ends.
Environmental Sensors
- GPS: Tracks location, distance, speed, and elevation.
- Altimeter/Barometer: Measures altitude and atmospheric pressure.
- Ambient Light Sensor: Adjusts screen brightness and monitors light exposure.
- UV Sensor: Measures UV radiation exposure.
Environmental Sensors Overview
Primary Environmental Sensors
- Ambient light sensor adjusts display brightness and monitors light exposure.
- Altimeter detects elevation changes and climbing activities.
- UV sensor measures harmful sunlight exposure.
- Magnetometer/compass works with GPS for location accuracy.
- Temperature sensors monitor both body and environmental conditions.
Accuracy Challenges
Environmental Interference
- Temperature variations affect sensor performance.
- Poor GPS reception impacts location tracking.
- Ambient light can interfere with optical sensor readings.Conclusion: Sensor accuracy is definitely affected by environment.
Testing Complexities
- Multiple environmental factors must be tested simultaneously.
- Difficulty in simulating real-world environmental conditions.
- Complex testing requirements for multi-parameter measurements.
- Temperature variations of ±5°C can affect calorie calculations.
Conclusion: Like other sensors and situations mentioned already, this senser has an increasing inaccuracy as the activity and environment varies from the generic and preprogramed.
Galvanic Skin Response (GSR): Measures skin conductivity for stress and arousal levels.
Technical Issues and Limitations
Accuracy Limitations
Emotional Detection
- Only measures intensity of emotional arousal, not the specific type of emotion.
- Cannot differentiate between positive and negative emotions (e.g., excitement vs. fear).
- Requires complementary sensors for emotional context validation.
Conclusion: Only measures emotion without differentiation between positive and negative.
Data Collection Issues
- Systematic noise generation during signal detection.
- Continuous fluctuations at the minute scale from stress induction to recovery.
- Dehydration of electrodes increases noise in high-impedance sites.
Conclusion: Environmental and physiological noise is a big issue!
Technical Interference
Environmental Factors
- Vulnerability to motion artifacts.- Position changes (sitting, standing, walking) affect signal detection.
- Laboratory measurements may not translate to real-world settings due to environmental conditions.
Conclusion: Sensor output varies with change in position equaling unreliability.
Physical Constraints
领英推荐
- Electrode placement must be precise for accurate readings.
- Requires proper skin contact and conductivity.
- Limited reception range (approximately 5 meters) for wireless devices.
Conclusions: Accuracy is user dependent.
Implementation Challenges
Signal Quality
- Bluetooth signals cannot pass through water, human tissue, or concrete.
- Connection drops when subjects move to different rooms.
- Reduced electrode adherence affects measurement quality.
Conclusion: Huge issue with signal transmission in different real-life environments. Algorithm compensates mathematically by assumption.
Measurement Reliability
- Requires multiple sensor types for comprehensive emotional assessment.
- Lower accuracy and reliability in naturalistic settings compared to controlled environments.
- Need for constant calibration and validation of measurements.
Conclusion: Mood assessment is garbage.
Proximity Sensor: Detects when device is being worn.
Technical Problems and Limitations
Sensor Design
- Virtual proximity sensors rely on gyroscope and accelerometer combinations instead of dedicated hardware
- Limited reception range (approximately 5 meters) for wireless connectivity- Hardware sensors require precise positioning for accurate readings
Performance Problems
- Screen activation issues during calls due to inconsistent detection
- Frequent mistouch events when using virtual sensors
- Interference from movement and position changes
Accuracy Limitations
Detection Issues
- Inconsistent performance in detecting when device is raised to ear
- False positives in pocket detection
- Screen may activate unexpectedly during calls
- Difficulty distinguishing between intentional and unintentional proximity
Conclusion: significant positional false positives and inaccuracies.
Environmental Interference
External Factors
- Performance affected by ambient conditions
- Signal degradation through human tissue and solid materials
- Bluetooth connectivity issues through water or concrete
- Temperature variations can affect sensor performance
Conclusion: Multiple real life conditions interfere with sensor function.
Software Limitations
Algorithm Challenges
- Virtual sensors rely heavily on software interpretation
- Multiple sensor data must be combined for accurate readings
- Calibration requirements affect real-time performance
- Battery drain from continuous sensor operationConclusion: Algorithms are written to compensate for sensor limitations including environmental interference, signal loss, signal dampening, and hardware malfunction based on population probability. A recipe that equals a somewhat educated guess at best.
Gesture Sensor: Recognizes specific hand movements and gestures.
Hardware Constraints
- Resource limitations on smartwatch platforms affect performance.
- Limited processing power impacts real-time recognition.
- Battery consumption issues with continuous monitoring.
- Sensor accuracy varies with device positioning and strap tightness.
Conclusion: Hardware issues remain.
Recognition Accuracy
- Accuracy drops from 95% to 88% with new users not in training data.
- Performance decreases from 95% to 75% when watch strap changes from tight to loose.
- Complex gestures are harder to recognize reliably.
Conclusion: User variability significantly affects data accuracy.
Environmental Challenges
External Factors
- Ambient lighting conditions affect visual-based sensors.
- Background clutter can interfere with tracking.
- Shadows and reflections cause misinterpretation.
- Environmental variations impact signal quality.
Conclusion: Environmental issues also affect this sensor data collection and accuracy.
User Variability
- Different cultural contexts affect gesture interpretation.
- Variations in hand shapes and movement speeds.- Individual user differences impact recognition accuracy.
- Walking while gesturing creates additional complexity.
Conclusion: Differences in culture, human habits, size, environment etc. all affect data collection, recognition, and registration.
Weaknesses of Health and Fitness Devices Summation
Key Sensors Commonly Found in Modern Fitness Trackers and Smartwatches
Core Motion Sensor
Composition and/or duties:
1. Accelerometer: Measures acceleration, speed, and movement in three dimensions to track steps, activity levels, and orientation.
2. Gyrocope: Detects angular velocity, rotation, and precise orientation changes.
3. Magnetometer/Compass: Works with GPS to determine direction and precise location.
Limitations:
1. Compounded inaccuracies in the Motion Sensor are evident!
2. Algorithms are very narrow at this time, with poor extrapolation.
3. Skin pigmentation and external environmental temperature negatively impact optical sensor
Physiological Sensor
Components and/or duties:
1. Heart Rate Monitor: Uses optical sensors (PPG) or ECG to measure heart rate and pulse.
2. ECG Sensor: Measures electrical signals from heart activity for detailed cardiac monitoring.
3. SpO2 (Pulse Oximeter): Measures blood oxygen saturation levels.
4. Skin Temperature Sensor: Monitors body temperature changes.
5. Bioimpedance Sensor: Measures body composition, hydration levels, and respiratory rate.
6. Electrodermal Activity (EDA) Sensor: Measures skin conductance for stress monitoring.
Limitations:
Environmental Sensors
Component and/or duties:
1. GPS: Tracks location, distance, speed, and elevation.
2. Atimeter/Barometer: Measures altitude and atmospheric pressure.
3. Ambient Light Sensor: Adjusts screen brightness and monitors light exposure.
4. UV Sensor: Measures UV radiation exposure.
Limitations:
1. Sensor accuracy is definitely affected by environment.
2. Like other sensors and situations mentioned already, this senser has an increasing inaccuracy as the activity and environment varies from the generic and preprogramed.
Galvanic Skin Response (GSR) Sensor
Component and/or duties: Measures skin conductivity for stress and arousal levels1.
Limitations:
Proximity Sensors
Component and/or duties: Detects when device is being worn.
Limitations:
Gesture Sensor
Component and/or duties: Recognizes specific hand movements and gestures.
Limitations:
1. User variability significantly affects data accuracy.
2. Environmental issues also affect this sensor data collection and accuracy.
3. Differences in culture, human habits, size, environment etc. all affect data collection, recognition, and registration.
Patrick E Sewell MD