Disney Research Method Captures Facial Details at High Fidelity and Real Time
Forehead wrinkles that rapidly deepen and crow's feet that appear suddenly around the eyes might distress the average person, but the ability to quickly and realistically incorporate such details in a facial reconstruction is the key feature of a new performance capture method developed at Disney Research.
The method, which requires only a single video camera such as a webcam, is the first to both operate in real-time and to capture facial features in high resolution, including such details as wrinkles.
"This could open up a variety of new applications, from casual home users who wish to give their own likeness to a game character to commercial applications where, for example, a customer can apply and view virtual makeup," said Derek Bradley, research scientist at Disney Research. "In film production, our method would allow a director to assess an actor's performance transferred to a digital creature even as he is filming."
In addition to Bradley, the research team included Chen Cao and Kun Zhou of Zhejiang University as well as Thabo Beeler of Disney Research.
Capturing facial details has proven critical for conveying emotions or the state of mind of a character and audiences are sensitive to subtle characteristics of the face. Animators thus devote great care to tracking facial performances of actors so that they can create realistic virtual characters. Until now, these facial performance capture methods have been designed to either capture these details in high resolution offline, or to record less detail but at high speed.
The new method represents a substantial step towards the goal of attaining both high quality and high speed.
As with other real-time facial capture methods, the researchers begin by using generic, low-resolution face models as the basis for reconstruction. A face model is virtually fitted to the user by the system as the user changes expressions and contorts his or her face in front of a video camera.
The system then automatically calculates a personalized wrinkle probability map, based on the set of extreme expressions the user assumed to train the program. The wrinkles may not match the user's actual wrinkles exactly, but they exhibit a very similar appearance because of the way they are formed.
"By learning the relationship between local image appearance and local wrinkle formation, we can reconstruct plausible face wrinkles in real-time from a single camera," Beeler noted. Once trained, the online capture approach can be applied to any new user without additional training. The system is generic and can be used with most common types of real-time face trackers.
Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice Presidents Jessica Hodgins and Markus Gross manage Disney Research facilities in Los Angeles, Pittsburgh, Zürich, and Boston and work closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.