Generating Animatable 3D Models from Monocular Visual Data

animatable 3d model generation from 2d monocular n.w
1 / 27
Embed
Share

Explore a groundbreaking method to generate animatable 3D models from monocular photos, revolutionizing applications in games, movies, and communication. This innovative approach allows intuitive control over animations based on kinematics, offering a new frontier in visual computing and computer vision.

  • 3D Models
  • Visual Computing
  • Computer Vision
  • Monocular Data
  • Animation
rayaanacharya
shi_c Follow

Uploaded on | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

Download Presentation

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

E N D

Presentation Transcript

  1. ANIMATABLE 3D MODEL GENERATION FROM 2D MONOCULAR VISUAL DATA P. Fechteler, L. Kausch, A. Hilsmann, P. Eisert IEEE ICIP 2018 9. October 2018 Visual Computing Group Department of Computer Science Computer Vision & Graphics

  2. 2 Motivation Animatable 3D models are omnipresent in today s society, e.g. games, movies, communications ... Our contribution: new method to generate animatable models from monocular photos animation based on kinematics (intuitive animation control) Visual Computing Group Department of Computer Science Computer Vision & Graphics

  3. 3 Overview Input Template Computation Image sequence of rigid object Rigid 3D model Input Monocular image sequence of non-rigid object Non-Rigid Shape Estimation Mesh sequence with consistent topology Articulated Model Calculation Output Animatable model: - vertices - joints - skinning weights Visual Computing Group Department of Computer Science Computer Vision & Graphics

  4. 4 From monocular photos to static 3D reconstruction Input Template Computation Image sequence of rigid object Rigid 3D model Input Monocular image sequence of non-rigid object Non-Rigid Shape Estimation Mesh sequence with consistent topology Articulated Model Calculation Output Animatable model: - vertices - joints - skinning weights Visual Computing Group Department of Computer Science Computer Vision & Graphics

  5. 5 Template computation sparse point cloud SFM [Wu11] dense point cloud CMVS+PMVS [Furukawa10] Poisson surface mesh ?0 ?0 template model - surface mesh - volumetric graph Screened Poisson Surface Reconstruction [Kazhdan13] volumetric graph model [Si15] Visual Computing Group Department of Computer Science Computer Vision & Graphics

  6. 6 From monocular photos to registered 3D Input Template Computation Image sequence of rigid object Rigid 3D model Input Monocular image sequence of non-rigid object Non-Rigid Shape Estimation Mesh sequence with consistent topology Articulated Model Calculation Output Animatable model: - vertices - joints - skinning weights Visual Computing Group Department of Computer Science Computer Vision & Graphics

  7. 7 Non-rigid shape registration Goal: determine locations of template vertices Frame-to-frame approach with global template reference: 1. Estimation of camera orientation and position: from rigid background correspondences 2. Estimation of unknown deformation: minimization of non-linear energy function ??,????????,??+ ????(??,??) min image data fitting controls smoothness of deformation Visual Computing Group Department of Computer Science Computer Vision & Graphics

  8. 8 Non-rigid shape registration: data term Feature point correspondence constraint ?? Interframe 2D-2D SIFT matches [Furch13] ??+1 Visual Computing Group Department of Computer Science Computer Vision & Graphics

  9. 9 Non-rigid shape registration: data term Feature point correspondence constraint Silhouette constraint Visual Computing Group Department of Computer Science Computer Vision & Graphics

  10. 10 Non-rigid shape registration: data term Feature point correspondence constraint Silhouette constraint - region specific via color matching Visual Computing Group Department of Computer Science Computer Vision & Graphics

  11. 11 Non-rigid shape registration: regularization constraint Temporal smoothness penalize strong frame-to-frame deformations Surface detail preservation ARAP functional on surface vertices [Sorkine07] Volume stiffness rotation-invariant volumetric graph Laplacian [Zhou13] Visual Computing Group Department of Computer Science Computer Vision & Graphics

  12. Reconstructed scan registrations of synthetic Pixar Lamp dataset input output output (other view) Visual Computing Group Department of Computer Science Computer Vision & Graphics

  13. 13 From registered 3D models to animatable model Input Template Computation Image sequence of rigid object Rigid 3D model Input Monocular image sequence of non-rigid object Non-Rigid Shape Estimation Mesh sequence with consistent topology Articulated Model Calculation Output Animatable model: - vertices - joints - skinning weights Visual Computing Group Department of Computer Science Computer Vision & Graphics

  14. 14 Components of kinematic model Skeleton control structure (joint locations & orientations) 3D mesh (vertices & topology) Skinning weights (for non-linear LBS/DQB) Pose parameters for animation (joint angles & global similarity) Visual Computing Group Department of Computer Science Computer Vision & Graphics

  15. 15 How to generate such a skinning model? Goal 1: Realism exploit example data (no arbitrary heuristics) Goal 2: Simple usage no sophisticated parameter tuning preprocessing / initialization registration of scan vertices clustering of limbs independent limb alignment optimization of model components skinning weights vertex positions joint locations and orientations shape weight ? Visual Computing Group Department of Computer Science Computer Vision & Graphics

  16. 16 Objective function to optimize Input: registered example 3D reconstructions Goal: resemble input 3D reconstructions in pose and shape Objective ?for S example scans with V vertices and shape weight 2+ ? ?k 2 scani ? ?k,?k,? scani ? ? ?k,?k,? ? ?,?,? = ?k i=1..S k=1..V Laplace term to optimize for shape conformance Data term to optimize for pose conformance scani = vertex k of scan i ?k ?k = vertex k of template model ?() = pose dependent template vertex transformation ?[] = uniform Laplace operator Visual Computing Group Department of Computer Science Computer Vision & Graphics

  17. 17 Optimization of objective function in 3 steps 2+ ? ?k 2 scani ? ?k,?k,? scani ? ? ?k,?k,? ? ?,?,? = ?k i=1..S k=1..V Joint locations and orientations (and pose parameters) Skinning weights Vertices Visual Computing Group Department of Computer Science Computer Vision & Graphics

  18. 18 Optimization of model vertices 2+ ? ?k 2 scani ? ?k,?k,? scani ? ? ?k,?k,? ? ?,?,? = ?k i=1..S k=1..V rearrange skinning function ?(): = ? ?k,?k,? = ?T?k+ ?T ?k with ?T being a 3x3 matrix and ?T being a 3-vector insert into objective function ? ?,?,? solve directly for the linear least squares solution using a sparse solver Visual Computing Group Department of Computer Science Computer Vision & Graphics

  19. 19 Optimization of model skinning weights 2+ ? ?k 2 scani ? ?k,?k,? scani ? ? ?k,?k,? ? ?,?,? = ?k i=1..S k=1..V Block Coordinate Descent algorithm For each vertex vary current skinning weights in all directions determine best gain with respect to ? ?,?,? Loop until convergence take over the skinning weights with best gain update precalculated gains Visual Computing Group Department of Computer Science Computer Vision & Graphics

  20. 20 Optimization of model joints 2+ ? ?k 2 scani ? ?k,?k,? scani ? ? ?k,?k,? ? ?,?,? = ?k i=1..S k=1..V Joint locations and orientations together with pose parameters calculate partial derivatives use exponential map representation for joint rotations to stay inside SO(3) solve using Gauss-Newton algorithm Line Search algorithm for Gauss-Newton update Visual Computing Group Department of Computer Science Computer Vision & Graphics

  21. 21 Optimized skinning model learned from MPI Faust (10 registered example scans) input reconstructed 3D output animated model Visual Computing Group Department of Computer Science Department of Computer Science Visual Computing Group Computer Vision & Graphics Computer Vision & Graphics

  22. 23 ... putting all together ... Visual Computing Group Department of Computer Science Computer Vision & Graphics

  23. 24 Result from synthetic sackboy sequence Input: Output: Examples: reposed model screen captured animatable model Intermediate: reconstructed 3D Visual Computing Group Department of Computer Science Computer Vision & Graphics

  24. 25 Result from synthetic sackboy sequence reconstructed 3D animated model Visual Computing Group Department of Computer Science Computer Vision & Graphics

  25. 26 Result from captured doll sequence Input: 2D photos Output: animatable model Intermediate: reconstructed 3D Visual Computing Group Department of Computer Science Computer Vision & Graphics

  26. 27 Result from captured doll sequence Examples: reposed model Visual Computing Group Department of Computer Science Computer Vision & Graphics

  27. 28 Result from captured doll sequence Result from captured doll sequence Thank you. Questions? Visual Computing Group Department of Computer Science Computer Vision & Graphics

Related


No related presentations.

More Related Content