POC: Augmented Reality - Signal Strength Visualization for WiFi, LTE & 5G

Proof of Concept: Augmented Reality - Signal Strength Visualization for WiFi, LTE & 5G

RE-POST: This is a re-post from https://dreamtolearn.com/ryan/data_analytics_viz/130/en Dream to Learn, and also touched on here https://www.ibm.com/blogs/think/2019/09/how-drones-help-visualize-the-wireless-grid-to-serve-stronger-5g/

"MAKING VISIBLE THAT WHICH IS INVISIBLE" (this extends beyond radio frequencies, can include anything that is detectable and/or calculate-able - like smell (gas leaks, odors), NBC threats (atomic particles, chemical contaminants), risks (soft spots in fence perimeter), or even human states or emotions (e.g. a data driven aura)

Drone Surveys WIFI Signal Strength and AR Visualize

Overview

We did a side project to equip DRONES with RF signal strength detectors, and then fly them around to SURVEY a 3D area - creating a signal strength map data set. Then, we imported that data set, using IBM and Open Source tools, compatible with UNITY. Then - we exported to an Augmented Reality compatible platform (could be Hololens, but we used iPAds) to let the user "See" the signal strength.

This is the raw data , images and thoughts that occurred during the side project.

12 minute playback video here (informal) and the 'journey' below...

Where we started - POC Visualization (April)

Friend of IBM Jake Madden pushed some synthetic data into his Magic leap AR kit to add a bit of color. The data in the video is just test data (no survey was done of the rooms)

Here's a video:  <LINK> Magic Leap POC

Survey Copter Build (June)

Jay Griffin did some pretty cool work configuring this "Survey-Copter"

Survey Copter Test Flight (June)

Medium term vision - Why we want this data

as we wander from WIFI base station we will illustrate how the signal fades. Similarly , at events with 5G "hot zones' smartphone and AR headset users can 'see' where 5G low latency zones exist...

Survey of IBM Watson West in San Francisco (LTE Data)

Working with Jenna Goldberg - we surveyed some RSSI (Received Signal Strength Indication) data from IBM's Watson West location (Howard at 1st street in San Francisco) - more on that below - but the data you see here is actual data. (We pulled LTE and WIFI)

Red 'orbs' signify a lower intensity signal, Green is higher. This was RAW data - not calibrated on exact location, and no scaling / smoothing applied yet. Top right corner near concrete column (across the street on Blackrock building) is the cell phone antenna tower for LTE source.

Top left - this TV monitor is important later - as it's got a big steel plate (aka RF shield) - we will see that there is a shadow cast from the LTE source, behind this TV screen and also concrete pillars. (We hypothesize 5G milimeter wave to be even more pronounced with single source emittors)

 Demonstration: Value in Seeing Data "IRL"

•  Show that AR Tech can be applied to visualize and remedy weak points in evolving 5G networks (better antenna placements)
•  Especially valuable given the gnarly physics of 5G - e.g.
• https://www.forbes.com/sites/annatobin/2018/05/25/could-5g-have-trouble-penetrating-buildings/#242c229f9f97

Visualization:  

Point cloud , anchored to physical location

Data Sources (Real world)

• WIFI Signal Strength Level
• LTE Signal Strengh Level
• Ping Test
• X Y Z coordinates (For above)

"RF Shadows" Cast By Concrete Posts and metal plates

This is not the WIFI Node you Seek

(i.e. I was surprised to learn I was using another base station further away, vs overhead box)

Attenuation in Weather? (explore & visualize) - fog, rain and drizzl - Future project?

source: https://www.sciencedirect.com/topics/engineering/atmospheric-attenuation

Mapping Options:

• Manual (MAC)
• Manual (Raspberry PI with a button - and some 1m resolution locator - GPS or other localized data)
• Drone - fly drone 8 X 8 X 3 (think about three chess boards) - taking measurements

Actionable Insights: 

• Adjust mesh antennas (rotate, relocate, or add more) to ensure sufficient coverage across physical space.
• Modify Policies to ensure choke points are prioritizing network or compute that is priority

    Post It Credit :) - Jenna G.

Telco Trade Show WIFI Visualized (June 2019)

Started with Dummy Data... then

https://github.com/rustyoldrake/Catchers_Mitt/blob/master/Signal%20Strength%20Basic%20Data%20Generator.csv

https://github.com/rustyoldrake/Catchers_Mitt/blob/master/Signal%20Strength%20Basic%20Data%20Generator.xlsx

... Real Data - Mapping Signal Strength for Real

mac terminal >

while x=1; do /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I | grep CtlRSSI; sleep 0.5; done

wifi SSI - you can script, or just walk around your room and holler to your friend the coordinates and 'level'

 Dial and call "*3001#12345#*" to enter iOS Field Test mode.

Credit Duy Nguyen

Test Flight #2 (Jay Griffin) - Drone Programmed in Node Red to Automate Survey

We hope you enjoyed this raw re-post!

See also: https://www.ibm.com/blogs/think/2019/09/how-drones-help-visualize-the-wireless-grid-to-serve-stronger-5g/

----- APPENDIX -----

BIll of Materials

• 1x Tello-EDU drone $99 https://store.dji.com/product/tello-edu?vid=47091* Be sure it is the "EDU" version of the drone. It allows for easier wireless connectivity for accessing the flight SDK
• spare propellers $4 https://store.dji.com/product/tello-propellers
• spare battery $19 https://store.dji.com/product/tello-battery?from=compatibility
• battery charging hub $25 https://store.dji.com/product/tello-battery-charging-hub
• 1x(3x) Raspberry PI Zero W $10 https://www.adafruit.com/product/3400 for $10 ea. spares are good
• I got this kit for a bit more which covered most accessories $60 https://www.adafruit.com/product/3411
• 1x GPS breakout card $40 https://www.adafruit.com/product/746
• wire spools (4 colors needed, 6 in the pack) $16 https://www.adafruit.com/product/1311 / Solder $8 https://www.adafruit.com/product/145 / iron $22 https://www.adafruit.com/product/180 / solder wick (recommended if soldering novice, mandatory if expert.... or so I'm told) $3 https://www.adafruit.com/product/149 /Really short usb cable: $6 (important for weight and mounting balance) / https://www.amazon.com/CableCreation-Compatible-Kindle-Android-Tablet/dp/B01M5GZ3N0/ref=sr_1_1_sspa?keywords=CC0573-2&qid=1559593854&s=electronics&sr=1-1-spons&psc=1 / total ~$300 depending on replacement part insurance.

What Next - What might we poke at in 2020?

• Weather, Humidity, Pollutants, Altitude impact on propagation and model https://www.etsi.org/images/files/ETSIWhitePapers/etsi_wp25_mwt_and_5g_FINAL.pdf
• Mini-Project 1 - MANUAL MAP - Do some REAL WORLD signal strength mapping (hand hold a Mac, or phone) and manually create a (A) WIFI Strength ; (B) LTE / 4G Signal Strength
• Mini-Project 2 - DRONE Survey - program a drone to SURVEY an area - 1 meter grid coordinates, measure, move , measure, move (could also be a ROOMBA)
• Mini-Project 3- Explore 3D mesh of Room / Enviroment (thanks Joe P!) https://www.6d.ai/ APIs for Augmented Reality and the AR Cloud. With just a smartphone camera, 6D.ai's SDK creates a 3D mesh of the world around you. App developers can easily relocalize multiple cross-platform players into the same coordinate system. Objects remain persistent across sessions while respecting occlusion (JP > for "signal strength demo could be that you only see the spheres that are within your view")
• Mini-Project 4 - GHOST Protocol - visualizing an after-image for security or perimiter monitoring
• Mini-Project 5 - overlay multiple frequencies - e.g. 5G and LTE and WIFI on same viz (find juicy network in a stadium or event)

Other Links

• IBM Watson Studio https://www.ibm.com/cloud/watson-studio
• IBM Immersive Data: https://www.youtube.com/watch?time_continue=111&v=6Nni5ywbEqo 
• https://medium.com/ibm-watson/visualize-high-dimensional-data-fast-watson-studio-ebad7e7e1b6a
•  Finding Hidden Cells Phones in Prisons https://medium.com/@gwilymnewton/finding-hidden-cells-phones-in-prisons-16421352e039
• Prior Work (Pre-POC) - Proof of Concept (Magic Leap) thanks Jake Madden! Jake did a POC > he "created a test Magic Leap Unity project that reads in the csv file and creates colored spheres of sizes corresponding to the strength value and similar color scheme to the example table... captured a short video demonstrating on the ML1. It doesn't factor in walls for occlusion presently but could and demonstrates the idea." The video is around 80mb: https://drive.google.com/file/d/1anMDbpGhPwmJgdW0g9LFMvxkqqKu8vUb/view?usp=sharing