eyeMovie
Cinematic Composition with Gaze
eyeMovie is a cinematic experiment of gaze-directed film as well as a research into the viability of using eye tracking as the new form of human-computer interaction.
Our goal was to recreate an experience similar to a Choose Your Own Adventure book where the viewers are able to weave their own story within a world created for them by the director of the film. eyeMovie takes the viewer through the movie by following where their eyes are naturally engaging. In Film Form [1], Sergei Eisenstein articulates how a series of images can produce an abstract concept not strictly present in each of the composite images through associations and symbolic resonances. Instead of imposing meaning onto the images, he lets his viewers make the connection. More recently, Peter Greenaway champions the visual sequence as the message and the content of a film [2]. The intent of the eyeMovie is inspired by these avant-garde film-makers: empowering the viewers to re-interpret the film based on their own direction.
Our algorithm evaluates each frame of the film based on color [RGB histogram, dominant colors, distinct colors and hues] and movement [optical flow]. When the viewer's gaze is fixated on an area of the film, a new scene with matching or contrasting visual qualities is presented. We hope to evoke emotions using a montage of contrasting qualities such as in Battleship Potemkin [Eisenstein, 1925] and give new symbolic meanings to the scene via juxtaposition of seemingly unrelated visual elements as in Sweet Movie [Dušan Makavejev, 1974]. The new sequence of the film establishes a different rhythm, tone, and meaning from the original film similar to Eisenstein's diagram of pictorial composition and movement (see below). Each viewing results in a new composition of film.
We chose Birdman or (The Unexpected Virtue of Ignorance) [3] for our first experiment because it is clear when the story diverges from the single shot look of the original film. eyeMovie transitions to another scene when the eyes are fixated on a position. The color palette is reduced to neutralize the visual intention of the filmmaker when the gaze fixation is prolonged. We also added spatial transformations for shorter fixations such as by repeating parts of a scene and by covering or zooming to parts of a scene to amplify emotions and unfold new meanings.
GSD6432 Quantitative Aesthetics | 2015
Harvard Graduate School of Design
Instructor: Panagiotis Michalatos
Programming Languages C#. Used emgu (OpenCV) image processing library
Collaborators Lezhi Li
Eisenstein's diagram of pictorial composition and movement
Eisenstein created storyboards of his films based on the movement and spatial sequences. In eyeMovie, the viewers compose their film based on the color qualities and movements that either match or contrast their current areas of interest.
Testing eye tracking device as the new artistic form of human-computer interaction
We used the Tobii EyeX tracking device, which is often used as a web analytics tool and experimentally as a game controller. Movies are predominantly characterized by their aesthetics which includes color and movement. We wanted to investigate how these visual signals are processed by the viewers and how we can leverage them as inputs to transform the movie watching experience to one that is more personal.
The orange dot in the below picture represents the average gaze between the two eyes.
Image analysis techniques
We divided the screen into five [top, bottom, middle-left, middle-center, and middle-right] to analyze the visual qualities of each frame. The following metrics were used to select the succeeding scene based on the quality of the area the viewer's gaze is focused on. Below are test results using Inception [4].
Color Histogram
Color histogram to extract dominant colors and red, green, and blue tones
Distinct Colors
The bottom circles show the most distinct colors from the rest of the frame from left to right
Movements: Optical Flow
We identified the direction of the movements via optical flow. Yellow signifies the movement to the right and cyan to the right.
Combining Frame Analysis with Gaze
We then tracked the gaze and tested how humans perceive the visual contents of the movie. Below is a short test clip from Inception [4] showing the movement [top left] and distinct color [bottom left] analysis of what the viewers are seeing [right].
Main findings are that people stare at the distinct color if there is no obvious subject in the scene and that they track the opposite direction from the optical flow if they already anticipate what happens next. The results inspired the transformations we applied to the Birdman [3] movie above, which are to connect the temporal shifts via color and spatial shifts via directional qualities.
References
[1] Sergei Eisenstein. "Film Form - Essays in Film Theory". Ed. Jay Leyda. New York: Harcourt, Brace & World, 1949.
[2] UC Berkeley Events. "Peter Greenaway: 'New Possibilities: Cinema is Dead, Long Live Cinema'." Online video clip. YouTube. YouTube, 5 October, 2010. Web. 10 May, 2015.
The movies were solely used for academic purposes.
[3] Birdman or (The Unexpected Virtue of Ignorance). Alejandro G. Iñárritu. Fox Searchlight Pictures. 2014. Film.
[4] Inception. Christopher Nolan. Warner Brothers. 2010. Film.