UC Berkeley Ubiquitous Swarm Lab
Infrared Localization of Wireless Sensor Networks
Sponsored by the Berkeley Sensor & Actuator Center and INRIA-EVA Paris
Advisor: Professor Kristofer Pister
Collaborators: Dr. Brian Kilberg PhD and Dr. Craig Schindler PhD
Summer 2018 - Spring 2020
TLDR: Infrared Lighthouse Localization of Wireless Sensor Networks
Problem: Autonomous microsystems, such as microrobots, are uniquely resource-constrained with a requirement for processing and communication hardware to be highly miniaturized. Localization with accuracy on the order of a centimeter is paramount to the performance of intelligent tasks in the physical world when operating alone or in a wireless network of complementary autonomous microsystems.
Solution & Contribution: Reverse engineered the HTC Vive infrared lighthouse base station mm-accurate localization system, allowing resource-constrained microsystems (wireless sensor nodes, microrobots, MAVs/UAVs) to localize themselves at up to 60Hz with high-precision. This opened the door to a slew of other research projects I built & worked on spanning cooperative UAV infrared localization using on-board time-synchronized wireless sensor nodes [1] to localizing a monolithic wireless SoC [2] to Lighthouse-IMU sensor fusion for industrial automation control [3]. 4 IEEE publications related to the lighthouse project (1 first author, 2 second author).
Big Picture: Mesh-Networked Autonomous Microsystems
Autonomous microsystems, such as microrobots, are uniquely resource-constrained with a requirement for processing and communication hardware to be highly miniaturized. Localization with accuracy on the order of a centimeter is paramount to the performance of intelligent tasks in the physical world when operating alone or in a wireless network of complementary autonomous microsystems.
ionocraft
crawler
jumper
flocking in a WSN
*All images courtesy of the Berkeley Sensor & Actuator Center (BSAC) unless otherwise specified
MIMSY and SCµM: Localizable Miniature Wireless Sensor Nodes
MIMSY: The Micro-Inertial Measurement System (precursor to SCµM on the right) is a 16mmx16mm open source wireless sensor node equipped with an ARM Cortex-M3 microprocessor, 802.15.4 wireless transceiver, 9-axis IMU, & 5 GPIO pins.
IR Localization: MIMSY can receive & decode IR pulse envelopes via a GPIO pin from a custom IR-sensitive photodiode circuit or dedicated IC (e.g. TS3633 IR light-to-digital converter, blue IC on top right of image).
SCµM: The Single Chip Micro-Mote is a 2mmx3mmx0.3mm 5mg monolithic system-on-chip containing an ARM Cortex-M0 microprocessor and a crystal-free 2.4 GHz radio with IEEE 802.15.4 compatibility and BLE transmit capabilities.
IR Localization: SCµM contains an integrated optical receiver for programming which can also be used as a sensing mechanism for infrared-based localization.
Lighthouse: mm-Precision Localization with Infrared Planar Laser Sweeps
Powered HTC Vive Lighthouse v1 emitting omnidirectional IR LED pulses and sweeping planar infrared beams horizontally and vertically (source)
Uncased HTC Vive Lighthouse v1 showing LED array for omnidirectional sync pulse and horizontal / vertical motors for rotating planar laser sweeps (source)
Nikon’s large-scale metrology industrial lighthouse iGPS system fan beam diagram (source)
My first summer at the Swarm Lab in 2018, I began working on infrared “lighthouse” localization, a sub-millimeter precision optical infrared localization method that uses rotating planar laser sweeps and omnidirectional synchronization pulses that allow an object to determine its relative angle (e.g. azimuth and elevation) to an emitting base station.
Diagram showing how relative angle is computed from a rotating planar infrared sweep and a corresponding wide-angle IR LED sync pulse emitted as the sweep passes over a reference bearing.
I first reverse-engineered HTC Vive’s Lighthouse base station v1, used for 6-DoF pose estimation of HTC Vive’s virtual reality headset and controllers, to localize a IEEE 802.15.4-compatible wireless sensor node (MIMSY) with sub-millimeter-precision.
Wiring MIMSY to a TS3633 infrared-sensitive light to digital converter, I wrote firmware for MIMSY’s TI CC2538 using memory-mapped hardware registers to capture and time rising & falling edges of the IR synchronization pulses and planar sweeps with µs precision (see square wave timing diagram below) enabling sub-mm precision localization without interfering with high-priority radio interrupts used to maintain an active connection in an OpenWSN 6-TiSCH network.
This allowed MIMSY to compute and wirelessly communicate its sub-degree-precision relative azimuth and elevation measurements over an OpenWSN 6-TiSCH network, enabling several follow up projects including Lighthouse-IMU sensor fusion for industrial automation control [3], cm-accuracy localization of the Single-Chip Micro-Mote (SCµM) [2], and replacing the Lighthouse base station with a cooperative quadrotor-based infrared lighthouse localization system [1].
Timing diagram showing rising and falling edges of synchronization pulses and subsequent planar infrared sweep. Axis of sweep is determined by length of the sync pulse.
HTC Lighthouse v1 azimuth distribution from [3] detected by static MIMSY over 1 million samples at 3 meters away, showing standard deviation of 3.5x10^-5 radians, equivalent to 0.1mm at 3 meters away.
Left: (Video) Lighthouse localization of board with array of 4 co-planar MIMSY-TS3633 (infrared photodiode IC) pairs transmitting azimuth and elevation data at 10Hz to a central computer which solves the Perspective-n-Point (PnP) problem (given known relative positions of the co-planar MIMSYs and using Levenberg-Marquadt optimization) to compute and visualize the 6-DoF pose of the board (optimization code, visualization code). Right: Figure from [2] showing how lighthouse azimuth & elevation measurements are projected onto a unit distance image plane to be used as “pixel” coordinates to solve the PnP problem.
Applying Infrared Lighthouse Localization
Millimeter-Precision Localization for Microrobots, UAVs, & Industrial Automation
Lighthouse-IMU Sensor Fusion for WSN Localization and Industrial Automation Control
First-author paper published in 2020 IEEE International Conference on Factory Communication Systems (WFCS)
Using an extended Kalman filter fusing 60Hz Lighthouse position measurements and onboard IMU 4kHz accelerometer readings to continually estimate the position and velocity of a wireless sensor node moving along a conveyor belt, changing the direction of the conveyor belt whenever the wireless sensor node reaches an unsafe zone.
Showed that the EKF improves lighthouse localization-enabled control and allows for reliable control with IMU measurements even with temporary occlusion from the lighthouse base station.
Infrared Lighthouse Localization of a Wireless Monolithic Crystal-Free System-on-Chip (SCµM)
Second-author paper published in IEEE J-MEMS (originally Hilton Head which was cancelled due to COVID)
Using two HTC Vive lighthouse base stations (calibrated to one another using LM-optimization to estimate a solution to the Perspective-n-Point problem with azimuth and elevation samples from SCµM and projected points from the ground truth motion capture system) to enable 3D localization of a fully wireless crystal-free system-on-chip (SCµM) — intended for payload constrained applications like microrobotics — using SCµM’s integrated optical infrared receiver and integrated wireless radio to communicate lighthouse sync and sweep timings over BLE.
Cooperative UAV Lighthouse Localization in a Time-Synchronized Wireless Sensor Network
Second-author paper published in IEEE Sensors Journal (Special Issue UAV Sensor Networks)
Scrapping the static lighthouse base station and using a time-synchronized wireless sensor node and a laser with plane optics onboard a quadrotor as a mobile lighthouse, we demonstrated a system for quadrotor-based infrared localization that can be used to localize all nodes in an time-synchronized wireless sensor network with sub-meter accuracy.
Since each node in the wireless sensor network is time synchronized within 10ms, no synchronization pulse is required and each quadrotor can spin freely to localize all other nodes in the wireless sensor network by simply broadcasting a table of time-synchronized timestamps corresponding to the heading of the quadrotor at each timestep.
List of Publications
1. Brian G. Kilberg, Felipe M. R. Campos, Craig B. Schindler, and Kristofer S. J. Pister. "Quadrotor-Based Lighthouse Localization with Time-Synchronized Wireless Sensor Nodes and Bearing-Only Measurements." Sensors Journal Special Edition: UAV Sensor Networks: IEEE, Jul. 2020. (link)
Some robotic localization methods, such as ultra wideband localization and lighthouse localization, require external localization infrastructure in order to operate. However, there are situations where this localization infrastructure does not exist in the field, such as robotic exploration tasks. Deploying low power wireless sensor networks (WSNs) as localization infrastructure can potentially solve this problem. In this work, we demonstrate the use of an OpenWSN network of miniaturized low power sensor nodes as localization infrastructure. We demonstrate a quadrotor performing laser-based relative bearing measurements of stationary wireless sensor nodes with known locations and using these measurements to localize itself. These laser-based measurements require little computation on the WSN nodes, and are compatible with state-of-the-art 2 mm × 3 mm monolithic wireless system-on-chips (SoCs). These capabilities were demonstrated on a Crazyflie quadcopter using an Extended Kalman Filter and a network of motes running the OpenWSN wireless sensor network stack. The RMS error for X positioning was 0.57 m and the error for Y positioning was 0.39 m. This is the first use of an OpenWSN sensor network to support robotic localization. Furthermore, simulations show that these same measurements could be used for localizing sensor motes with unknown locations in the future.
2. Brian G. Kilberg, Felipe M. R. Campos, Filip Maksimovic, and Kristofer S. J. Pister. "Accurate 3D Lighthouse Localization of a Low-Power Crystal-Free Single Chip Mote." Journal of Microelectromechanical Systems (J-MEMS): IEEE, Jul. 2020. (link)
We present a system for centimeter-precision 3 dimensional localization of a 2 × 3 × 0.3 mm^3, 5 mg, wireless system-on-chip by utilizing a temporally-structured infrared illumination scheme generated by a set of base stations. This 3D localization system builds on previous work by adding a second lighthouse station to enable 3D localization and using the integrated wireless radio, making the localization system fully wireless. We demonstrate 3D tracking with mean absolute errors of 1.54 cm, 1.50 cm, and 5.1 cm for the X, Y, and Z dimensions. This is the first time such a lighthouse localization system has been able to localize a monolithic single-chip wireless system.
3. Felipe M. R. Campos, Craig B. Schindler, Brian G. Kilberg, and K. S. J. Pister. "Lighthouse Localization of Wireless Sensor Networks for Latency-Bounded, High-Reliability Industrial Automation Tasks." 16th IEEE International Conference on Factory Communication Systems (WFCS '20). Porto, Portugal: IEEE, Apr. 2020. (link)
We present the results of a latency-bounded, high-reliability conveyor belt control system for a cart containing a self-localizing wireless sensor node. The node is equipped with an ARM Cortex-M3 microprocessor, 802.15.4 transceiver, 9-axis inertial measurement unit (IMU), and an infrared-sensitive photodiode which allows the wireless node to localize itself using a high-precision localization system for small, resource-constrained, low-cost wireless sensor nodes known as “lighthouse” localization. The cart moves across the conveyor belt, and upon reaching a specified position sends a wireless signal to a set of receiving nodes attached to the conveyor belt’s motor to reverse direction. Using an extended Kalman filter (EKF) running on-board the cart’s wireless sensor node to estimate the position and velocity of the cart, we are able to achieve 3 ms response latency, equivalent to the response latency of industrial photoelectric sensors used in a related implementation. We also show the lighthouse system used in this implementation has no outlier measurements outside the ±1mm error range when stationed 3 meters away from the conveyor belt. This, in addition to use of the EKF, enables high-reliability control with strong occlusion tolerance. We show the wireless sensor node is able to continue estimating its position along the conveyor belt when occluded from the lighthouse base station with a median standard deviation reported by the EKF of 0.875mm after 10 cm of occlusion compared to a median 0.109mm standard deviation of the position estimate when un-occluded.
4. Craig B. Schindler, Daniel S. Drew, Brian G. Kilberg, Felipe M. R. Campos, Soichiro Yanase, and Kristofer S. J. Pister. "MIMSY: The Micro Inertial Measurement System for the Internet of Things." IEEE 5th World Forum on Internet of Things (WF-IoT). Limerick, Ireland: IEEE, Apr. 2019. (link)
The Micro Inertial Measurement System (MIMSY) is an open source wireless sensor node for Internet of Things applications, specifically designed for a small system volume while maintaining functionality and extensibility. MIMSY is a 16mm × 16mm node with an Arm Cortex-M3 microprocessor, 802.15.4 wireless transceiver, and a 9-axis IMU. The system is fully compatible with the OpenWSN wireless sensor networking stack, which enables the straightforward implementation of standards compliant 6TiSCH mesh networks using MIMSY motes. While the application space of MIMSY is quite vast, we present three sample implementations showcasing the opportunities afforded by a small and relatively low-cost mote with mesh networking and inertial measurement capabilities, including: high granularity areal sensing for sleep monitoring with motes embedded in a foam mattress; high reliability, low latency communication for industrial process automation and control; and long lifetime physical event detection and activity monitoring with minimal setup time.
