Revolutionizing 3D Scene Synthesis with Dynamic Lighting and Machine Learning
Our project introduces a cutting-edge view synthesis system using 3D Gaussian splats and a raytracer for real-time dynamic lighting. By leveraging a Machine Learning model, we enable the extraction of material properties from videos for enhanced rendering. We address the challenge of generating new scene images under varying lighting conditions and angles, surpassing existing products like Polycam and Luma.AI Video to 3D API. Our solution bridges the gap in the market with superior performance and innovation.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Lightning Talk: Product Research sdmay25-40
Project Overview Our project is a novel view synthesis system that uses 3D Gaussian splats with a raytracer that allows for real time dynamic lighting and the incorporation of traditional polygon meshes. Use a Machine Learning model to extract material properties from videos to allow for dynamic lighting.
Problem Statement Generate new images of a scene under different lighting conditions with new additional objects from a specific point of view using pictures of the scene from different points of view.
Related product: Polycam A 3D scanning platform with a user friendly interface. Can create a 3D gaussian splat from a smartphone video. Product is very accessible and the app has many other 3D graphics services. Does not use ray tracing to render the images which limits options for reflections, shadows, and other lighting effects. Dependent on Cloud. Can lead to delays and slowdown. Limits on the devices which can be used to take input videos.
Related product 2: Luma AI Video to 3D API A mobile application to capture spatial memories and generate 3D models in seconds. Provides the ability to create 3D scenes from videos using a web API. Simple to use and it doesn t require the user to run the models themselves (Which require large amounts of computing power) Uses NerFS (Neural radiance fields) which are computationally expensive and require neural networks trained on large amounts of data. Long compute times ( 30 minutes for each) Charges users per render.
Related product 3: RealityScan A free-to-download mobile app that enables you to create high-fidelity 3D models using just your phone or tablet. Easy to use. Does not require technical expertise to use. Directly integrates with other 3D graphics tools Uses photogrammetry which is on average Slower than NerF s which are generally slower than Gaussian splats.
Market gap Our solution leverages Gaussian Splats it will outperform most existing products which leverage slower solutions like NeRFs or photogrammetry. Our solution will allow a Gaussian splat to be rendered with dynamically changing lighting by combining a raytracing renderer with a machine learning model which helps extract the material properties.
New ideas from research To make our solution more user friendly we want to integrate it with Unity engine so users can use it in as a solution in other projects. We could also export our created renders to stl files to allow them to be 3D printed.
Conclusion Our solution provides a unique benefit that it allows scenes rendered using 3D gaussian splatting to be dynamically relit.
