RESEARCH

SERVICES

OUR DATA

We offer highest resolution volumetric data worldwide and a team with many years of research experience. Our data provide an ideal basis for the development of next-generation enterprise solutions and as training data for new AI procedures, 5G platforms and computer display technologies.

Showcases

MULTIVIEW

calibrated, high-resolution multi-view video data

GEOMETRY

video-based point clouds, meshes and textures

FACES

dynamic facial expressions

TEMPORALLY COHERENT

meshes and texture topologiess

ENDLESS

3d models with loopable animations

OPTIMIZED

dynamic 3d models for low-end hardware

STREAMABLE

volumetric video data

FEATURES

ADAPTIVE MESH SIMPLIFICATION

Adaptive mesh simplification allows to preserve more triangles and allocate a larger texture budget for important areas such as faces or hands.

TEMPORAL MESH STABILIZATION

Temporally stable meshes allow for better compression and easier post-processing of data. We offer fully automated computing of temporally stable meshes.

INDIVIDUAL FEATURE TRACKING

Tracking of the 3D data allows the placement of new accessories or changes to clothing without having to re-record the volucap. This allows a more flexible use for future applications.

DYNAMIC EYE CONTACT

Eye contact increases immersion as the Volucap always follows the observer. It doesn’t matter where the viewer stands, because he always feels directly addressed.

ARTIFICIAL INTELLIGENCE

Artificial intelligence is used not only for better quality. We can synthesize new poses and movements, e.g. mouth movements for new languages, which were never recorded.

More about our technology

READ LATEST PUBLICATIONS

2021 Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video

We present Non-Rigid Neural Radiance Fields (NR-NeRF), a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. Our approach takes RGB images of a dynamic scene as input (e.g., from a monocular video recording), and creates a high-quality space-time geometry and appearance representation. We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e.g. a `bullet-time’ video effect. NR-NeRF disentangles the dynamic scene into a canonical volume and its deformation. Scene deformation is implemented as ray bending, where straight rays are deformed non-rigidly. We also propose a novel rigidity network to better constrain rigid regions of the scene, leading to more stable results. The ray bending and rigidity network are trained without explicit supervision.

2021-TOWARDS REAL-TIME AI HUMANS WITH NEURALl LUMIGRAPH RENDERING

Novel view synthesis is a challenging and ill-posed in- verse rendering problem. Neural rendering techniques have recently achieved photorealistic image quality for this task. State-of-the-art (SOTA) neural volume rendering ap- proaches, however, are slow to train and require minutes of inference (i.e., rendering) time for high image resolutions. We adopt high-capacity neural scene representations with periodic activations for jointly optimizing an implicit surface and a radiance field of a scene supervised exclusively with posed 2D images. Our neural rendering pipeline accelerates SOTA neural volume rendering by about two orders of magnitude and our implicit surface representation is unique in allowing us to export a mesh with view-dependent texture information. Thus, like other implicit surface representations, ours is compatible with traditional graphics pipelines, enabling real-time rendering rates, while achieving unprecedented image quality compared to other surface methods. We assess the quality of our approach using existing datasets as well as high-quality 3D face data captured with a custom multi-camera rig.

2020-SPLIT RENDERING FOR MIXED REALITY: INTERACTIVE VOLUMETRIC VIDEO IN ACTION

This demo presents a mixed reality (MR) application that enables free-viewpoint rendering of interactive high-quality volumetric video (VV) content on Nreal Light MR glasses, web browsers via WebXR and Android devices via ARCore. The application uses a novel technique for animation of VV content of humans and a split rendering framework for real-time streaming of volumetric content over 5G edge-cloud servers. The presented interactive XR experience showcases photorealistic volumetric representations of two humans. As the user moves in the scene, one of the virtual humans follows the user with his head, conveying the impression of a true conversation.

2020-NEURAL SPARSE VOXEL FIELDS

Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision.

2020-CLOUD RENDERING-BASED VOLUMETRIC VIDEO STREAMING SYSTEM FOR MIXED REALITY

We propose a volumetric video streaming system that offloads the rendering to a powerful cloud/edge server and only sends the rendered 2D view to the client instead of the full volumetric content.

2020-LESSONS LEARNED DURING ONE YEAR OF COMMERCIAL VOLUMETRIC VIDEO PRODUCTION

In June 2018, Fraunhofer Heinrich Hertz Institute (HHI) together with Studio Babelsberg, ARRI, Universum Film AG (UFA), and Interlake founded the joint venture Volucap GmbH and opened a commercial volumetric video studio on the film campus of Potsdam-Babelsberg. After a testing phase, commercial productions went live in November 2018. The core technology for volumetric video production is 3D human body reconstruction (3DHBR), developed by Fraunhofer HHI. This technology captures real persons with our novel volumetric capture system and creates naturally moving dynamic 3D models, which can then be observed from arbitrary viewpoints in a virtual or augmented environment. Thanks to the large number of test productions and new requirements from customers, several lessons have been learned during the first year of commercial activity. The processing workflow for capture and production of volumetric video has been continuously evolved such as process-dependent color grading and a new encoding scheme for volumetric video. Furthermore, specific challenges have been tackled, for instance, fast movements and relighting of actors .

2020-GOING BEYOND FREE VIEWPORT: CREATING ANIMATABLE VOLUMETRIC VIDEO OF HUMAN PERFORMANCES

In this paper, we present an end-to-end pipeline for the creation of high-quality animatable volumetric video content of human performances. Going beyond the application of free-viewpoint volumetric video, we allow re-animation and alteration of an actor’s performance through (i) the enrichment of the captured data with semantics and animation properties and (ii) applying hybrid geometry- and video-based animation methods that allow a direct animation of the high-quality data itself instead of creating an animatable model that resembles the captured data. Semantic enrichment and geometric animation ability are achieved by establishing temporal consistency in the 3D data, followed by an automatic rigging of each frame using a parametric shape-adaptive full human body model.

2019-ADVANCED VOLUMETRIC CAPTURE AND PROCESSING

Fraunhofer Heinrich Hertz Institute (HHI) has developed a novel technology for volumetric video. The 3D Human Body Reconstruction (3DHBR) technology captures real persons with our novel volumetric capture system and creates naturally moving dynamic 3D models, which can then be observed from arbitrary viewpoints in a virtual or augmented reality scene.

2019-REGION DEPENDENT MESH REFINEMENT FOR VOLUMETRIC VIDEO WORKFLOWS

This paper addresses high quality mesh optimization for volumetric video. Real persons are captured with multiple cameras and converted to 3D mesh sequences. These volumetric video assets can be used as dynamic 3D objects in arbitrary 3D rendering engines.

2019-CAPTURE AND 3D VIDEO PROCESSING OF VOLUMETRIC VIDEO

In this paper, a new capture and processing system for volumetric video is presented, called 3D Human Body Reconstruction (3DHBR). The system is based on 16 stereo pairs of high-resolution cameras capturing a moving person in 360 degree.

2018-FRAUNHOFER-CREATING DYNAMIC 3D MODELS OF REAL PERSONS FOR VIRTUAL WORLDS

With the extremely rapid development of the Virtual Reality (VR) and Augmented Reality (AR) markets, creating a convincing 3D representation of persons will bring the field of media production to the next level. New technology developments with volumetric video enable to simultaneously capture real persons with multiple cameras and create naturally moving dynamic 3D models.

TRUSTED BY EXPERTS, USED BY LEADERS

RESEARCH PARTNERS

WANNA FIND OUT MORE?

FX.Center – Studio Babelsberg

Phone: +49 331 88547580

Email: mail@volucap.de

Web: volucap.de