Overview
Rythmotion is an experimental rhythm game inspired by titles like Guitar Hero and Pump, but with a completely different control approach. Instead of using a keyboard or controller, players interact with the game using facial expressions and head angle poses captured through the device camera.
The project was built as an Apple technology exploration project during my time at the Apple Developer Academy. The main goal was to explore how machine learning and computer vision could be combined with rhythm-based gameplay to create a novel, hands-free experience on iOS.
Notes move in sync with the music, and player actions are triggered by changes in facial expressions or head movements. This turns the player’s face into the primary input device and creates a unique, unconventional way to play a rhythm game.
Tech Stack
Rythmotion was built using Swift and SpriteKit for the core gameplay and animations. For machine learning, I used MLKit to train and experiment with facial expression models, then integrated the trained model into the iOS app using CoreML for on-device inference. Audio playback and timing were handled using Apple’s audio and video frameworks, while the camera was used for real-time face input processing.
My Role
I was responsible for implementing the system that allows the game to recognize player facial expressions and head poses. This included preparing and training a facial expression recognition model using an open-source dataset, experimenting with different expressions and angle classifications, and integrating the model into the game so predictions could directly control gameplay in real time.
Learnings
This project gave me hands-on experience applying machine learning in an interactive, real-time environment. I learned how sensitive model accuracy is to dataset quality, balance, and size, and how these factors directly affect gameplay responsiveness. While the model worked, its accuracy was still limited, which helped me understand the practical challenges of deploying ML models beyond experimentation. The experience also deepened my understanding of CoreML, MLKit, and how machine learning can be used creatively in game design.
Result & Reflection
Although the facial expression model did not achieve high accuracy, the project successfully demonstrated the feasibility of face-controlled gameplay on iOS. More importantly, it became a valuable learning experience in experimenting with Apple’s ML ecosystem and evaluating real-world limitations of machine learning systems.