Publications 2017

ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner

arXiv 2017

We introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations.

[paper] [bibtex] [project page]

BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration

Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt

ACM Transactions on Graphics 2017 (TOG)

We introduce a novel, real-time, end-to-end 3D reconstruction framework, with a robust pose optimization strategy based on sparse feature matches and dense geometric and photometric alignment. One main contribution is the ability to update the reconstructed model on-the-fly as new (global) pose optimization results become available.

[paper] [video] [bibtex] [project page]

Publications 2016

FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

arXiv 2016

We introduce FaceVR, a novel method for gaze-aware facial reenactment in the Virtual Reality (VR) context. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-mounted display (HMD), as well as a new data-driven approach for eye tracking from monocular videos.

[paper] [bibtex] [project page]

Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis

Angela Dai, Charles Ruizhongtai Qi, Matthias Nießner

arXiv 2016

We introduce a data-driven approach to complete partial 3D shapes through a combination of volumetric deep neural networks and 3D shape synthesis. From a partially-scanned input shape, our method first infers a low-resolution – but complete – output.

[paper] [bibtex] [project page]

Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging

Zachary DeVito, Michael Mara, Michael Zollhöfer, Gilbert Bernstein, Jonathan Ragan-Kelley, Christian Theobalt, Pat Hanrahan, Matthew Fisher, Matthias Nießner

arXiv 2016

We propose a new language, Opt, in which a user simply writes energy functions over image- or graph-structured unknowns, and a compiler automatically generates state-of-the-art GPU optimization kernels.

[paper] [bibtex] [project page]

3DMatch: Learning the Matching of Local 3D Geometry in Range Scans

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao

arXiv 2016

In this paper, we introduce 3DMatch, a data-driven local feature learner that jointly learns a geometric feature representation and an associated metric function from a large collection of real-world scanning data.

[paper] [bibtex] [project page]

Learning to Navigate the Energy Landscape

Julien Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip H. S. Torr, Shahram Izadi, Cem Keskin


In this paper, we present a novel, general, and efficient architecture for addressing computer vision problems that are approached from an `Analysis by Synthesis' standpoint.

[paper] [video] [bibtex] [project page]

VolumeDeform: Real-time Volumetric Non-rigid Reconstruction

Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger

ECCV 2016

We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method does not require a pre-defined shape template to start with and builds up the scene model from scratch during the scanning process.

[paper] [video] [bibtex] [supplemental] [project page]

Efficient GPU Rendering of Subdivision Surfaces using Adaptive Quadtrees

Wade Brainerd, Tim Foley, Manuel Kraemer, Henry Moreton, Matthias Nießner

ACM Transactions on Graphics 2016 (TOG)

We present a novel method for real-time rendering of subdivision surfaces whose goal is to make subdivision faces as easy to render as triangles, points, or lines. Our approach uses the GPU tessellation hardware and processes each face of a base mesh independently and in a streaming fashion, thus allowing an entire model to be rendered in a single pass.

[paper] [video] [bibtex] [project page]

PiGraphs: Learning Interaction Snapshots from Observations

Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner

ACM Transactions on Graphics 2016 (TOG)

We learn a probabilistic model connecting human poses and arrangements of objects from observations of interactions collected with commodity RGB-D sensors. This model is encoded as a set of Prototypical Interaction Graphs (PiGraphs): a human-centric representation capturing physical contact and attention linkages between geometry and the human body.

[paper] [video] [bibtex] [project page]

ProxImaL: Efficient Image Optimization using Proximal Algorithms

Felix Heide, Steven Diamond, Matthias Nießner, Jonathan Ragan-Kelley, Wolfgang Heidrich, Gordon Wetzstein

ACM Transactions on Graphics 2016 (TOG)

ProxImaL is a domain-specific language and compiler for image optimization problems that makes it easy to experiment with different problem formulations and algorithm choices. The compiler intelligently chooses the best way to translate a problem formulation and choice of optimization algorithm into an efficient solver implementation.

[paper] [bibtex] [supplemental] [project page]

Face2Face: Real-time Face Capture and Reenactment of RGB Videos

Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

CVPR 2016 (Oral)

We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.

[paper] [video] [bibtex] [supplemental] [project page]

Volumetric and Multi-View CNNs for Object Classification on 3D Data

Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, Leonidas Guibas

CVPR 2016 (Spotlight)

In this paper, we improve both Volumetric CNNs and Multi-view CNNs by introducing new distinct network architectures. Overall, we are able to outperform current state-of-the-art methods for both Volumetric CNNs and Multi-view CNNs.

[paper] [code] [bibtex] [supplemental] [project page]

Publications 2015

Real-time Expression Transfer for Facial Reenactment

Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, Christian Theobalt

ACM Transactions on Graphics 2015 (TOG)

We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor.

[paper] [video] [bibtex] [project page]

Activity-centric Scene Synthesis for Functional 3D Scene Modeling

Matthew Fisher, Manolis Savva, Yangyan Li, Pat Hanrahan, Matthias Nießner

ACM Transactions on Graphics 2015 (TOG)

We present a novel method to generate 3D scenes that allow the same activities as real environments captured through noisy and incomplete 3D scans.

[paper] [video] [bibtex] [supplemental] [project page]

SemanticPaint: Interactive 3D Labeling and Learning at your Fingertips

Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, Philip H. S. Torr

ACM Transactions on Graphics 2015 (TOG)

We present a new interactive and online approach to 3D scene understanding. Our system, SemanticPaint, allows users to simultaneously scan their environment, while interactively segmenting the scene simply by reaching out and touching any desired object or surface.

[paper] [video] [bibtex] [project page]

Efficient Ray Tracing of Subdivision Surfaces using Tessellation Caching

Carsten Benthin, Sven Woop, Matthias Nießner, Kai Selgrad, Ingo Wald

High Performance Graphics 2015

In this paper, we propose method to efficiently ray trace subdivision surfaces using a lazy-build caching scheme while exploiting the capabilities of today's many-core architectures. Our approach is part of Intel's Embree.

[paper] [video] [code] [bibtex] [project page]

Real-time Rendering Techniques with Hardware Tessellation

Matthias Nießner, Benjamin Keinert, Matthew Fisher, Marc Stamminger, Charles Loop, Henry Schäfer

Computer Graphics Forum 2015

In this survey, we provide an overview of real-time rendering techniques with hardware tessellation by summarizing, discussing, and comparing state-of-the art approaches.

[paper] [bibtex] [project page]

Shading-based Refinement on Volumetric Signed Distance Functions

Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, Matthias Nießner

ACM Transactions on Graphics 2015 (TOG)

We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices.

[paper] [video] [bibtex] [project page]

Exploiting Uncertainty in Regression Forests for Accurate Camera Relocalization

Julien Valentin, Matthias Nießner, Jamie Shotton, Andrew Fitzgibbon, Shahram Izadi, Philip H. S. Torr

CVPR 2015

In this paper, we train a regression forest to predict mixtures of anisotropic 3D Gaussians and show how the predicted uncertainties can be taken into account for continuous pose optimization. Experiments show that our method is able to relocalize up to 40 percent more frames than the state of the art.

[paper] [video] [bibtex] [project page]

Incremental Dense Semantic Stereo Fusion for Large-Scale Semantic Scene Reconstruction

Vibhav Vineet, Ondrej Miksik, Morten Lidegaard, Matthias Nießner, Stuart Golodetz, Victor A. Prisacariu, Olaf Kähler, David W. Murray, Shahram Izadi, Patrick Pérez, Philip H. S. Torr

ICRA 2015

In this paper, we build on a recent hash-based technique for large-scale fusion and an efficient mean-field inference algorithm for densely-connected CRFs to present what to our knowledge is the first system that can perform dense, large-scale, outdoor semantic reconstruction of a scene in (near) real time.

[paper] [video] [bibtex] [project page]

The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces

Ondrej Miksik, Vibhav Vineet, Morten Lidegaard, Ram Prasaath, Matthias Nießner, Stuart Golodetz, Stephen L. Hicks, Patrick Pérez, Shahram Izadi, Philip H. S. Torr

CHI 2015

We present an augmented reality system for large scale 3D reconstruction and recognition in outdoor scenes. Unlike existing prior work, which tries to reconstruct scenes using active depth cameras, we use a purely passive stereo setup, allowing for outdoor use and extended sensing range.

[paper] [bibtex] [project page]

Database-Assisted Object Retrieval for Real-Time 3D Reconstruction

Yangyan Li, Angela Dai, Leonidas Guibas, Matthias Nießner

Eurographics 2015

We present a novel reconstruction approach based on retrieving objects from a 3D shape database while scanning an environment in real-time. With this approach, we are able to replace scanned RGB-D data with complete, hand-modeled objects from shape databases.

[paper] [video] [bibtex] [project page]

Dynamic Feature-Adaptive Subdivision

Henry Schäfer, Jens Raab, Mark Meyer, Marc Stamminger, Matthias Nießner

I3D 2015

In this paper, we present dynamic feature-adaptive subdivision (DFAS), which improves upon FAS by enabling an independent subdivision depth for every irregularity.

[paper] [video] [bibtex] [project page]

Publications 2014

SceneGrok: Inferring Action Maps in 3D Environments

Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner

ACM Transactions on Graphics 2014 (TOG)

In this paper, we present a method to establish a correlation between the geometry and the functionality of 3D environments.

[paper] [video] [bibtex] [project page]

Real-time Shading-based Refinement for Consumer Depth Cameras

Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, Christian Theobalt

ACM Transactions on Graphics 2014 (TOG)

We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes.

[paper] [video] [code] [bibtex] [project page]

Real-time Non-rigid Reconstruction using an RGB-D Camera

Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rhemann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, Marc Stamminger

ACM Transactions on Graphics 2014 (TOG)

We present a combined hardware and software solution for markerless reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time.

[paper] [video] [bibtex] [project page]

Real-Time Deformation of Subdivision Surfaces from Object Collisions

Henry Schäfer, Benjamin Keinert, Matthias Nießner, Christoph Buchenau, Michael Guthe, Marc Stamminger

High Performance Graphics 2014

We present a novel real-time approach for fine-scale surface deformations resulting from collisions.

[paper] [video] [bibtex] [project page]

Local Painting and Deformation of Meshes on the GPU

Henry Schäfer, Benjamin Keinert, Matthias Nießner, Marc Stamminger

Computer Graphics Forum 2014

We present a novel method to adaptively apply modifications to scene data stored in GPU memory. Such modifications

[paper] [video] [bibtex] [project page]

RetroDepth: 3D Silhouette Sensing for High-Precision Input On and Above Physical Surfaces

David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Tim Large, Steven Bathiche, Matthias Nießner, Alex Butler, Sean Fanello, Vivek Pradeep

CHI 2014

We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces.

[video] [bibtex] [project page]

State of the Art Report on Real-time Rendering with Hardware Tessellation

Henry Schäfer, Matthias Nießner, Benjamin Keinert, Marc Stamminger, Charles Loop

Eurographics 2014

In this state of the art report, we provide an overview of recent work and challenges in this topic by summarizing, discussing and comparing methods for the rendering of smooth and highly detailed surfaces in real-time.

[paper] [bibtex] [project page]

Combining Inertial Navigation and ICP for Real-time 3D Surface Reconstruction

Matthias Nießner, Angela Dai, Matthew Fisher

Eurographics 2014

We present a novel method to improve the robustness of real-time 3D surface reconstruction by incorporating inertial sensor data when determining inter-frame alignment. With commodity inertial sensors, we can significantly reduce the number of iterative closest point (ICP) iterations required per frame.

[paper] [video] [bibtex] [project page]

Publications 2013

Real-time 3D Reconstruction at Scale using Voxel Hashing

Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Marc Stamminger

ACM Transactions on Graphics 2013 (TOG)

Online 3D reconstruction is gaining newfound interest due to the availability of real-time consumer depth cameras. We contribute an online system for large and fine scale volumetric reconstruction based on a memory and speed efficient data structure.

[paper] [video] [code] [bibtex] [supplemental] [project page]

Analytic Displacement Mapping using Hardware Tessellation

Matthias Nießner, Charles Loop

ACM Transactions on Graphics 2013 (TOG)

Displacement mapping is ideal for modern GPUs since it enables high-frequency geometric surface detail on models with low memory I/O. We provide a comprehensive solution to these problems by introducing a smooth analytic displacement function.

[paper] [video] [bibtex] [project page]

Real-time Collision Detection for Dynamic Hardware Tessellated Objects

Matthias Nießner, Christian Siegl, Henry Schäfer, Charles Loop

Eurographics 2013

We present a novel method for real-time collision detection of patch based, displacement mapped objects using hardware tessellation. Our method supports fully animated, dynamically tessellated objects and runs entirely on the GPU.

[paper] [video] [bibtex] [project page]

Rendering Subdivision Surfaces using Hardware Tessellation

Matthias Nießner

PhD Thesis (Published by Dr. Hut)

In this thesis we present techniques that facilitate the use of high-quality movie content in real-time applications that run on commodity desktop computers. We utilize modern graphics hardware and use hardware tessellation to generate surface geometry on-the-fly based on patches.

[paper] [bibtex] [project page]

Real-time Simulation of Human Vision using Temporal Compositing with CUDA on the GPU

Matthias Nießner, Nadine Kuhnert, Kai Selgrad, Marc Stamminger, Georg Michelson

Proceedings 25th Workshop on Parallel Systems and Algorithms 2013

We present a novel approach that simulates human vision including visual defects such as glaucoma by temporal composition of human vision in real-time on the GPU. Therefore, we determine eye focus points every time step and adapt the lens accommodation of our virtual eye model accordingly.

[paper] [bibtex] [project page]

Publications 2012

Feature Adaptive GPU Rendering of Catmull-Clark Subdivision Surfaces

Matthias Nießner, Charles Loop, Mark Meyer, Tony DeRose

ACM Transactions on Graphics 2012 (TOG)

We present a novel method for high-performance GPU based rendering of Catmull-Clark subdivision surfaces. Unlike previous methods, our algorithm computes the true limit surface up to machine precision, and is capable of rendering surfaces that conform to the full RenderMan specification for Catmull-Clark surfaces.

[paper] [video] [code] [bibtex] [project page]

Patch-based Occlusion Culling for Hardware Tessellation

Matthias Nießner, Charles Loop

Computer Graphics International 2012

We present an occlusion culling algorithm that leverages the unique characteristics of patch primitives within the hardware tessellation pipeline.

[paper] [video] [bibtex] [project page]

Efficient Evaluation of Semi-Smooth Creases in Catmull-Clark Subdivision Surfaces

Matthias Nießner, Charles Loop, Günther Greiner

Eurographics 2012

We present a novel method to evaluate semi-smooth creases in Catmull-Clark subdivision surfaces. Our algorithm supports both integer and fractional crease tags corresponding to the RenderMan (Pixar) specification.

[paper] [video] [bibtex] [project page]

Real-time Simulation and Visualization of Human Vision through Eyeglasses on the GPU

Matthias Nießner, Roman Sturm, Günther Greiner


We present a novel approach that allows real-time simulation of human vision through eyeglasses. Our system supports glasses that are composed of a combination of spheric, toric and in particular of free-form surfaces.

[paper] [bibtex] [project page]

Publications 2011

Effective Back-Patch Culling for Hardware Tessellation

Charles Loop, Matthias Nießner, Christian Eisenacher

Proceedings of Vision, Modeling, and Visualization 2011

When rendering objects with hardware tessellation, back-facing patches should be culled as early as possible to avoid unnecessary surface evaluations, and setup costs for the tessellator and rasterizer.

[paper] [video] [bibtex] [project page]

Publications 2010

Fast Indirect Illumination Using Layered Depth Images

Matthias Nießner, Henry Schäfer, Marc Stamminger

Computer Graphics International 2010

We present a novel hybrid rendering method for diffuse and glossy indirect illumination.

[paper] [video] [bibtex] [project page]