Check out the project at https://xwalkkeyboards.app
XWalk Keyboards uses video feeds from NYC's network of traffic cameras to transform crosswalks into piano keyboards. As pedestrians step on the white stripes of the crosswalk, the app plays the corresponding notes!
This project won first place at AI Tinkerers' NYC Vision Hack v.2 in Aug 2026. The challenge: leverage AI, computer vision via Roboflow, and deploy via Google Cloud Run to transform raw NYC data into intelligent vision agents.
HOW IT WORKS
The web app uses Roboflow to detect pedestrians in a 511NY traffic cam video in real time. An AI agent running on Cloud Run orchestrates a calibration cycle and uses Roboflow to detect crosswalk stripes on regular intervals. When the web app detects a pedestrian inside a crosswalk stripe boundary, it lights up the stripe and plays the note corresponding to the stripe's position on a piano keyboard.
HACK NIGHT
As mentioned above, I started this project while participating in AI Tinkerers' NYC Vision Hack v.2 on Aug 7, 2026. If you check NYC's weather for that evening, you'll find we had an epic storm. I came up with the idea when browsing 511NY cameras a few days before, when the weather was calm. When I sat down to hack that night, the camera I wanted to use, CAM 5059 was rocking back and forth in the wind.
Ignoring the fact a shaky camera would not present well during demos, it also blew up my implementation strategy and revealed an important requirement for the project, automatic calibration. For the hack, I was going to hand draw the polygons the web app would use to detect if pedestrians stepped into the stripes. Fortunately, the storm calmed down long enough for the camera to settle in one spot and for me to draw a stable set of polygons. But this would not work for a production version of the app.
A VISION AGENT
The production version of the app, the version at https://xwalkkeyboards.app, pulls the stripe polygons from a json file in a Google Cloud Storage bucket. Meanwhile, every 15 minutes an ADK agent running on Google Cloud Run pulls a frame from the live video feed, analyzes the frame to determine if the crosswalk is blocked by obstructions or obscured by weather and, if not, passes a valid frame to a Roboflow tool. The Roboflow tool uses the Roboflow API to call a custom workflow based on Roboflow's SAM segment-anything-model. The tool passes detected polygons back to the agent which writes them to the json file in GCS.
You can view the full system diagram, calibration and playback strategies.
EXPLORE IT YOURSELF
The live app, the design file, and the full source are all public. If you want to see the app in action, check out the designs, or read through the implementation.
These are the places to start!