Work That I am Proud Of
Real-Time Multi-Camera Soccer Tracking System
I helped to develop a real-time, multi-camera soccer ball and player tracking solution. Unlike other competitors in this space, our approach was entirely visual: no GPS trackers, no player- or ball- mounted sensors. The foundation of the system was multi-view stereo computer vision. In production, our full pipeline achieved 60 frames per second and our data stream was used by TV broadcasters to annotate the video feed in real-time.
Many talented consultants and in-house researchers and engineers worked together to make this technology a reality. My contributions were camera calibration, and a 3D triangulation engine to localize players and the ball in 3D space from 2D object detections.
I created a process for calibrating fixed cameras installed in football stadiums using structure from motion. This process required special attention, because unlike typical SfM scenarios, the positions of the cameras were non-negotiable, and the interior of a stadium is highly symmetrical. Tools like COLMAP are prone to misregister images in these situations unless careful guardrails are established. With this process we were able to create highly accurate reconstructions.
I built an engine that triangulated 3D object positions from sets of per-camera 2D object detections using the camera calibration. The design of this engine was based on the RANSAC algorithm, bolstered with epipolar geometry for both accuracy improvement and performance optimization. This engine achieved sub-pixel reprojection error, and operated at more than 120 Hz, triangulating the positions of all 22 players, plus the ball, from 20+ camera views in 4K. The results were so astounding that we recycled them back into our camera calibration procedure to refine camera poses and model the optical distortion of the lenses.
AI Grading of Collectable Trading Cards
I worked with a founder to develop and launch the core technology behind his trading card grading platform. Collectable trading cards are a big market, and the cards are sometimes worth a lot. Collectors trust specialized grading services to assess the condition and value of trading cards during transactions, and our goal was to automate part of this ecosystem.
We released an iOS app, backed by a machine learning pipeline, that lets collectors scan their card using the phone's camera from various angles, and receive a grade and detailed damage report. The kinds of damage we assess are scratches and dents on the surface, squashed or rounded corners, and frayed edges.
To produce a unified map of damage on the card in 3D from 52 separate 2D images, I built a camera calibration system using differentiable rendering. By learning the camera pose of all 52 views, we can collate multiple views of the same damage into a unified instance in 3D space, thereby allowing us to accurately count the number and severity of damage on the card.
After launching the app and machine learning backend, I worked with the founder again to build an auto-scaling system to save thousands of dollars in monthly compute bills.