Play a piano with your eyes - Gaze estimation

In this example, we will estimate the gaze direction of a person and use it to play a virtual piano when the person looks to certain notes.

This examples uses the Google Mediapipe library to calculate the face mask and obtain the position of the eyes. The calculation of the gaze diration is done with OpenCV.

Requirements

Pipeless: Check the installation guide.
Python OpenCV, NumPy and Mediapipe packages. Install them by running:

pip install opencv-python numpy mediapipe

Python SimpleAudio` package. Install it by running:

sudo apt-get install libasound2-dev # Required by simpleaudio
pip install simpleaudio

Run the example

Create an empty Pipeless project

pipeless init my-project --template empty # Using the empty template we avoid the interactive shell
cd my-project

Feel free to change my-project by any name you want.

Download the stage folder

wget -O - https://github.com/pipeless-ai/pipeless/archive/main.tar.gz | tar -xz --strip=2 "pipeless-main/examples/gaze-piano"

Start Pipeless

The following command leaves Pipeless running on the current terminal

pipeless start --stages-dir .

Provide a stream

Open a new terminal and run the following command to provide a stream from your webcam:

pipeless add stream --input-uri "v4l2" --output-uri "screen" --frame-path "gaze-piano"

ONNX Runtime - Object detection with YOLO Benchmark