Cinematic Composition with Gaze
Eisenstein's diagram of pictorial composition and movement
Eisenstein created storyboards of his films based on the movement and spatial sequences. In eyeMovie, the viewers compose their film based on the color qualities and movements that either match or contrast their current areas of interest.
Testing eye tracking device as the new artistic form of human-computer interaction
We used the Tobii EyeX tracking device, which is often used as a web analytics tool and experimentally as a game controller. Movies are predominantly characterized by their aesthetics which includes color and movement. We wanted to investigate how these visual signals are processed by the viewers and how we can leverage them as inputs to transform the movie watching experience to one that is more personal.
The orange dot in the below picture represents the average gaze between the two eyes.
Image analysis techniques
We divided the screen into five [top, bottom, middle-left, middle-center, and middle-right] to analyze the visual qualities of each frame. The following metrics were used to select the succeeding scene based on the quality of the area the viewer's gaze is focused on. Below are test results using Inception .
Color histogram to extract dominant colors and red, green, and blue tones
The bottom circles show the most distinct colors from the rest of the frame from left to right
Movements: Optical Flow
We identified the direction of the movements via optical flow. Yellow signifies the movement to the right and cyan to the right.
Combining Frame Analysis with Gaze
We then tracked the gaze and tested how humans perceive the visual contents of the movie. Below is a short test clip from Inception  showing the movement [top left] and distinct color [bottom left] analysis of what the viewers are seeing [right].
Main findings are that people stare at the distinct color if there is no obvious subject in the scene and that they track the opposite direction from the optical flow if they already anticipate what happens next. The results inspired the transformations we applied to the Birdman  movie above, which are to connect the temporal shifts via color and spatial shifts via directional qualities.
 Sergei Eisenstein. "Film Form - Essays in Film Theory". Ed. Jay Leyda. New York: Harcourt, Brace & World, 1949.
 UC Berkeley Events. "Peter Greenaway: 'New Possibilities: Cinema is Dead, Long Live Cinema'." Online video clip. YouTube. YouTube, 5 October, 2010. Web. 10 May, 2015.
The movies were solely used for academic purposes.
 Birdman or (The Unexpected Virtue of Ignorance). Alejandro G. Iñárritu. Fox Searchlight Pictures. 2014. Film.
 Inception. Christopher Nolan. Warner Brothers. 2010. Film.