SPLAT!
Beginnings
This project was built for SDxUCSD's v0 Hackday (sponsored by Vercel), where the judging requirements were very open-ended. We had about 5 hours to build (cut down to 4.5 since we all got there late), and weren't set on an idea until 30 minutes later, as Alex, Shubham, Clarissa, and I brainstormed what we wanted to build, eventually settling on "SPLAT!". The purpose was to build something creative with CV and physics and see where it would take us[1].
Original Idea
Our original vision was for an environment where one could draw all sorts of shapes in a sketch mode, and then instantiate them as physical objects in the playground, allowing them to interact with each other as objects with different physical properties. This would be done fully through computer vision tracking the users' hands (a lightweight alternative to AR[2]). For example: brick, steel, and paper would have their unique properties modelled and collide with each other accordingly.
We wanted to integrate it with multiple people (either through different devices or multiple people on the same one), so that you could pass both sketches and objects between users. The other big feature would be similar to a Gaussian Splat, where a 3D model would be generated from the current physical environment. Users would be able to pan around their splat, and ideally continue to interact with it.
Actual Implementation
Our ideas were slightly ambitious given the tools and time we were working with. We ended up implementing the core functionality of drawing, using a the right hand to draw and left hand to change function between colors, as well as materials including water and fire. Toggling between drawing and physics mode was done by a grand two handed gesture. Multiplayer and the splat feature was never implemented.
Results
Ended up winning Most Creative Award at Vercel v0 Hackday by popular vote.