A Robot in Every Kitchen
Willow Garage tried it 20 years ago, but deep learning for computer vision hadn’t reached the maturity it has now. With the ability to localize dishes and other objects in its environment with millimeter-accuracy, Armstrong’s general-purpose collaborative two-arm robot is ready to scrape and rinse dishes, load and unload your dishwasher, clean countertops, and more: first in commercial kitchens in 2023, and soon enough in your own home.
From January 2022 to January 2023, I worked on building the core 2D and 3D perception pipeline for this new class of robot as part of a team of computer vision, robotics, software, and hardware engineers at Armstrong Robotics - a Lerer-Hippeau-backed seed-stage startup based in San Francisco, CA. I left in January 2023 when I decided I needed to pursue my own visions and creative directions. During my time at Armstrong we piloted with 2 major commercial entities in San Francisco and have been integrating the robotic system with other pilots and customers across Northern and Southern California since.
TLDR: Armstrong Robotics
Problem: In commercial restaurants & cafes, dishwashing employees are increasingly hard to hire and retain, half of them leave after less than a year, frequent no-shows, work is universally disliked and takes time away from attending to customers and making food & drinks, in addition to $3-6k average monthly cost on dishwashing. In U.S. homes, dishwashing is a daily chore averaging 4h per week per household & after-dinner cleanup cuts into priceless downtime even in households with daytime help. For robotic arm applications, a high-volume application that is also not entirely price insensitive has not been deployed (e.g. industrial pick-and-place is low volume, high cost), and dishwashing is an immediately addressable and technically feasible application that has huge commercial and consumer market potential.
Solution & Contribution: Joining the Armstrong team in January 2022, I have been working on bringing up core components of our 2D and 3D computer vision pipeline, focusing on safe and smooth operation around moving personnel with dynamic and static obstruction detection with 3D sensors, as well as mm-accuracy high precision and recall localization of manipulable objects (e.g. dishes) in all environments from dish-rack to sink to countertop. Coming into 2023, we are building to deploy our collaborative arm robots into the kitchens of partner restaurants and coffee shops around SF & the Bay Area (e.g. Souvla, Dandelion Chocolates, and soon more).
Dish Localization with Millimeter Accuracy
From the dirty to the cluttered to the glass clear and occluded, equipped with modern deep learning for semantic & instance segmentation and classical multi-view geometry techniques, our perception system can localize dishes with incredibly high accuracy for environment manipulation and pick & place tasks between dishwasher, countertop, cabinets, and more.
Over the past several months, I have moved us to an instance-segmentation paradigm and leveraged 6-DoF pose optimization to bring our perception system to robust millimeter-accuracy, with nearly 100% precision and recall localization for dishes in countertops, sinks, and dish-racks where before we had to rely heavily on environment priors to accurately localize.
Augmenting 2D Model-based Vision with 3D Sensors
Using solid-state LiDARs and IR-projecting depth cameras, the robot is capable of static and dynamic obstruction detection and avoidance, safely planning around static obstructions using RRT and an updated 3D environment model, and slowing down proportional to dynamic obstruction proximity at a 15Hz update rate.
Video: simple static obstruction detection & avoidance of an open box of sugar (risky stuff I know) and paper towel roll during a plate pick and place.