PicotionAI: Playing Pictionary with mixed reality, camera access, and artificial intelligence

Today I want to tell you about a little but very cute project I’ve developed for Pico headsets. It’s called PicotionAI and it is a game that includes mixed reality, camera access, and artificial intelligence! As I have done in other similar cases, I’m telling you this story hoping that it can inspire you to do cool projects of your own. Let me tell you everything!

Pico(tionAI)

Let’s start from the “Pico” part of the name of the game: as you can guess, it is called this way because it is made to run on Pico headsets, to be exact on the Pico 4 Enterprise and Pico 4 Ultra Enterprise.

The game came to life when last year I was advocating for having camera access on MR headsets, especially the Meta Quest. A Pico representative commented on my LinkedIn post saying that actually, Pico allowed for camera access on its enterprise headsets in specific approved cases. He also invited people interested in using computer vision on MR devices for B2B uses to reach out to them. We started talking and in the end, I managed to get a Pico 4 Enterprise and play a bit with camera access. Even more, in the end, I had the opportunity to make a little demo involving camera access for the launch of the Pico 4 Ultra enterprise…

(Pico)tionAI

Let’s now speak about how the idea of the Pictionary game came out. I was so excited to have camera access that I immediately started playing out with it. I did a lot of experiments… even one that had to be run in the bathroom! Unluckily Pico people commented on this last one by saying “We can not bring people to the toilet to try out our headsets” (Well, actually I agree it would be weird if while you are using the urinal at a VR event venue, someone would approach you and ask “Would you like to have some fun with me and my Pico?”). For one of these demos, I thought that it could have been cool to use passthrough camera access, and possibly AI, to make a game. But I did not want to make something lame like a speaking AI agent or just let you take photos of the surroundings. I wanted to do something more original.

Suddenly a light bulb lit up in my head: I remembered the fun nights with my friends playing Pictionary. The fun was especially because I suck at drawing and they had every time a hard time guessing what I was representing. So I thought: my friends could not guess what I was drawing, but can the AI do that? The game would have been about the user drawing with pen and paper, like in the original game, and the AI friend trying to guess what he/she is drawing. Every round should have been short (around 1-2 minutes), and at its finish, the more words the AI has guessed, the more points the user has scored. The interface would have been in mixed reality, with the system showing the 3D model of the object to draw. In the end, there should have been a classification to trigger gamification mechanics and entice people to play back to be on the first spot.

I started working on the project with the people at VRROOM Studio (that helped me with the 3D models because if I suck at drawing, I suck even more at 3D modeling) and after some days of work (across airports and hotels during my trips), I managed to deliver the tech demo in time to be used in some Pico events.

Do you want to see what it looks like? Then watch me playing a match in the video below! And if you were wondering… unluckily the AI struggles at recognizing what I’m drawing exactly like my friends Read More