• isabellerieken

Final Project: Spherror

Updated: Jan 17

CONCEPT: A playful, unconventional mirror commenting on the peculiarity of our faces. It segments your facial features and isolates them on a sphere, designed to look like a floating orb.


We were inspired by Daniel Rozin’s wooden panel mirror on the department floor. Isabelle and I pass it every day and can’t help but stop to look at our “reflection.” From this, we observed how much people really love looking at themselves. Additional inspiration includes other various projection mapped pieces shown to us in Communications Lab in which a human face was projected on a nonhuman object. We wanted to take this idea to the next level by allowing you, the visitor, to be that face being displayed.


The three main issues with the original version of this project and their solutions were as follows:


Height Problems: The original product unfortunately has a bit of a height discrimination. If you are over about 5'2" - 5'3", your head would block the projection and in turn block the images of your face.

Solution: We created a set up where the visitor has to sit in a chair to interact with the piece. This makes everyone generally the same height.


Scale: The height was only about 48". It had to be set on a table to be at eye-level, which is pretty unappealing.

Solution: We created a base for the dowels to be fixed to . It aligns perfectly with the average sitting height of a person and looks more finished.


Stand-by Image: When no one is in front of the webcam, it picks up the less-than-pretty background of a blank wall and the tripod.

Solution: Using PoseNet, we created an if statement directing the screen to play a video of tv noise when no one is present.


Facial Proportions: In the original model, we only used MadMapper to access the webcam and then sliced the live feed into segments where an approximate eye, nose, or mouth would be. This worked on our faces, but we knew it wouldn’t work on everyone else’s face.

Solution: We used PoseNet to track the eyes, nose, and mouth of the visitor. We then used the get() function to capture the pixels around the point that PoseNet was tracking. This way, our program would extract the facial features from any shape or size face.


MATERIALS:

foam board

Wooden dowels

Polystyrene spheres

Wood

Hot glue

Spray-paint

Webcam

projector


FABRICATION:

In our original design, the piece was very vertical with the spheres only reaching a distance of about two feet in width and three feet in height.




The dowels were implanted to blocks of wood which were then set on a table. In our new design, we wanted to get rid of the table and build a base. We also wanted to expand the area it covered.




To do so, we built a base from foam board with two outreaching arms to allow for more depth in sphere placement and for a more curved arrangement. The base is 26” x 36”.


One feature we added to the base was a strip of LED light to aid in illuminating a visitor’s face since the piece would be displayed in the dark room. We worried that the webcam would not be able to pick up the visitors face, but it turned out that the mere reflection of the light on the foam balls was enough to reveal a face.


To attach the dowels into the base, we put holes in the foam-board base and aligned a 1” wooden cube with a hole through the center with the foam-board hole. The dowels were inserted into the holes, and with the support of the wooden blocks, they wouldn’t topple over or stand crooked.


One of the dowels was replaced by a black acrylic tube to conceal the webcam wire. The wire was fed through pipe and connected to the computer through a hole at the bottom of the base. The camera was secured to the pipel with black tape. We also painted the polystyrene spheres with white acrylic paint to mattify the slightly reflective surface of the foam. Including the pipe as a dowel, we had 11 dowels and 25 spheres included in the final product.




THE CODE

Before working on the code, we knew we needed something that would be able to identify your eyes nose and mouth and keep track of them. We knew we had to somehow keep those specific features in a grid of squares which we would them transfer to MadMapper via Syphon and project them onto the spheres. Here’s how we did it:

First, we downloaded the ml5 library into p5 so we could use PoseNet. In a function we called function gotPoses() we established which keypoints we wanted PoseNet to track:

noseX = poses[0].pose.keypoints[0].position.x

noseY = poses[0].pose.keypoints[0].position.y

lEyeX = poses[0].pose.keypoints[1].position.x

lEyeY = poses[0].pose.keypoints[1].position.y

rEyeX = poses[0].pose.keypoints[2].position.x

rEyeY = poses[0].pose.keypoints[2].position.y

mouthX = poses[0].pose.keypoints[0].position.x

mouthY = poses[0].pose.keypoints[0].position.y


In this segment of code, we are declaring what the X and Y coordinates of each point being tracked and naming the PoseNet’s pre-established keypoints with the name of the feature.

INSERT POSENET CHART


As can be seen in this chart, keypoint 0 is the nose, keypoint 1 is the left eye and so forth. Notice, however, that PoseNet does not detect the mouth as a keypoint. We’ll get back to how we did solved that problem shortly.


Once we established which points we wanted, we had to trap the feature in a square. To do so, we employed the get() function to capture a certain amount of pixels around the point being tracked by PoseNet. We first declared a variable for each feature:


image(webcam,0,0);

c = get(noseX-100,noseY-20, 50, 50);

c1 = get(lEyeX-150,lEyeY,30,30);

c2 = get(rEyeX-20,rEyeY-10,50,50);

c3 = get(mouthX-100, mouthY+30, 50, 50);


The components of get() include the x coordinate of the point being tracked by PoseNet and the y-coordinate followed by the amount of pixels we wanted to display around those points.

Then we applied these variables to the texture of four squares on our canvas:

texture(c);

rect(-250, -250, 250, 250);

texture(c1);

rect(-250, 0, 250, 250);

texture(c2);

rect(0, -250, 250, 250);

texture(c3);

rect(-0, 0, 250, 250);

This gave us a canvas looking like this:



After completing this, we went on to add the code for a standby image. First, we generated a TV noise animation in Adobe AfterEffects and downloaded it as an mp4 so we could put it in our p5 sketch. The goal was for the video to play when PoseNet was not detecting a person in the frame and to disappear when a person entered the frame. Here’s our if statement:


if (poses.length > 0) {

let pose = poses[0].pose;

let keypoint = pose.keypoints[0];

console.log(poses[0].pose.keypoints[0].score);

console.log(keypoint.score);

console.log(keypoint.position.x);

console.log(keypoint.position.y);

noseX = poses[0].pose.keypoints[0].position.x

noseY = poses[0].pose.keypoints[0].position.y

lEyeX = poses[0].pose.keypoints[1].position.x

lEyeY = poses[0].pose.keypoints[1].position.y

rEyeX = poses[0].pose.keypoints[2].position.x

rEyeY = poses[0].pose.keypoints[2].position.y

mouthX = poses[0].pose.keypoints[0].position.x

mouthY = poses[0].pose.keypoints[0].position.y

imageMode(CENTER);

image(webcam,0,0);

c = get(noseX-100,noseY-20, 50, 50);

//image(c,80,80, 100, 100);

c1 = get(lEyeX-150,lEyeY,30,30);

c2 = get(rEyeX-20,rEyeY-10,50,50);

c3 = get(mouthX-100, mouthY+30, 50, 50);

face();

}

else {

blurry();

}

}


The if statement includes everything that happens in the canvas and then says that if anything else is happening in the frame, play the video which is called in the function blurry(). In function blurry(); we have the video play and loop using video.play(); and video.loop();

Test GIF

In this clip, I tested the if statement by blocking the computer webcam with my finger. As can be seen, the TV noise plays when no one is visible in the frame.

INSERT p5 presentation LINK


PROJECTION MAPPING

To connect the code with our sculpture, we used MadMapper and Syphon to projection map. Syphon was used to essentially screen record the Chrome browser p5 was working on and in MadMapper, we took the live feed, separated the grid into the individual squares, spherized them, and adjusted their sizes based on the position of the physical sphere.



FINAL PRODUCT




We were very pleased with the results and found it to be very successful in the ITP/IMA Winter Show.