INSUBCONTINENT EXCLUSIVE:
Capturing human performances in high-definition 3D is a complicated proposition, and among the many challenges is getting the lighting right
This impressive new project from Google researchers puts the subject in the center of what can only be described as an prismatic LED egg,
but the resulting 3D models are remarkable — and more importantly, relightable.
What called volumetric capture uses multiple cameras in a
360-degree setup to capture what can look like a photorealistic representation of a subject, including all the little details like clothing
deformation, hair movement, and so on
It has two serious weaknesses: First, it more like a 3D movie than a model, since you can&t pose the person or change their attributes or
clothing; The second is an extension of the first, in that you can&t change the way the person is lit — whatever lighting they had when
you captured them, that what you get.
&The Relightables& is an attempt by a team at Google AI to address this second issue, since the first
Their system not only produces a highly detailed 3D model of a person in motion, but allows that model to be lit realistically by virtual
light sources, making it possible to place it in games, movies, and other situations where lighting can change.
Images from the Google AI
paper that show the capture process and resulting 3D model alone and in a lighted virtual environment.
It all thanks to the aforementioned
prismatic egg (and a couple lines of code, of course)
The egg is lined with 331 LED lights that can produce any color, and as the person is being captured, those LEDs shift in a special
structured pattern that produces a lighting-agnostic model.
The resulting models can be placed in any virtual environment and will reflect
not the lighting they were captured in but the lighting of that little world
The examples in the video below aren&t exactly Hollywood-level quality, but you can see the general idea of what they&re going for.
The
limitations of volumetric capture make it unsuitable for many uses in film, but being relightable brings these performances a lot closer to
ordinary 3D models than they were before
Of course, you still have to do all your acting inside a giant egg.
&The Relightables& will be presented by the team at SIGGRAPH Asia.
