Are AI Videos the Right Direction for 3D Asset Generation?

Introduction#

All of us love to play video games. From massive titles like Black Myth: Wukong to indie favorites, game studios spend a significant amount of time and money, designing and creating 3D assets. With the growing adoption of VR and AR, the demand for 3D assets is only going to increase.

Accelerating 3D Asset Creation#

Currently, we have a few techniques to accelerate the process, but not all of them are widely used yet.

Text-to-3D and Image-to-3D#

We have AI tools and models like Meshy.ai, SAM-3D, and others that take text or images as input and generate 3D models.

Problems:

1. Text-to-3D: It is very difficult to properly describe a complex 3D model using text alone.
2. Image-to-3D: While image inputs offer better control allowing us to generate or edit images with AI tools—most tools rely on a single image. If you upload a front view of an object, the model has to guess what the back looks like, and hence lesser control. Multi-view image-to-3D tools are also available but we haven’t got a chance to experiment with it yet.
3. Mesh Quality: A major common problem with these tools is that the generated meshes lack clean geometry and topology, and often have excessively high polygon counts (dense irregular vertices, loose points, and unoptimized geometry). Even if it looks good, it becomes difficult when they want to do edits to it.

Photogrammetry#

It is a technique widely used in AAA game production (your favorite game likely used it). It creates realistic 3D assets by taking multiple photos of an object from different angles and using software to stitch them together into a 3D model.

This technique is so effective that in the video below you can see AEOS Games booked an entire town to capture photogrammetry scans, clean them up, and use them in their game. They explained that photogrammetry is the main reason their small team was able to build a really good game (Unleash the Avatar).

The way it works is that you take multiple photos of an object from different angles. The software tracks key features across images, generates a point cloud, and reconstructs a 3D mesh from that point cloud.

Limitations: Photogrammetry struggles with reflective surfaces and requires stable, consistent lighting. Overall, it’s a pretty useful technique.

Video-to-3D#

Photogrammetry is great for real-world objects, but if we want to create custom sci-fi assets that don’t exist in real life, it won’t work.

This was true until recently. But what if we used AI video generation models to bridge the gap? So we went ahead and tried it, especially since photogrammetry is already integrated into studio pipelines, this could offer a viable workflow for generating quick base meshes.

We decided to test this workflow, and here’s what we did:

1. Concept Image: Used Nano Banana Lite to generate an image of Pikachu.
2. Synthetic Video: Used Omni Flash to generate an orbital 360-degree video from the image.
3. Reconstruction: Used online photogrammetry software to convert the generated video into a 3D model.

Below are the generated video, and the resulting 3D model:

While the model needs some cleanup and topology fixes, it looks like a pretty usable base mesh. We also tried it for a Sci-Fi Helmet and for a character from Dragon Ball Z and the results are below

Current Major Limitation: AI video generation models cannot generate long, perfectly consistent videos yet, making full 360-degree coverage difficult for complex objects, larger assets. However, with the rapid development of world models (like Google DeepMind’s Genie), generating longer and more spatially consistent videos might quickly become a reality.

We also experimented with Gaussian Splatting for a scene, but due to the short duration of the AI-generated video clips we were not able to get the model to generate a proper video, so the results were not good.

A Benchmark for Video Generation Models ?#

The more spatially consistent and detailed an AI video is, the better the final 3D model will be. Because of this, converting generated videos into 3D meshes could serve as a one of many practical benchmark for evaluating the temporal and geometric consistency of video generation models.

Conclusion#

We aren’t 3D technical artists, but we do think this is a cool fun idea to try out and hence we tried. We would love for people to try it out themselves and see it might be helpful for them.

Disclaimer: Pikachu and all associated character names, logos, and designs are trademarks and copyrights of Nintendo, Creatures Inc., and GAME FREAK. All references and generated assets in this article are used strictly for non-commercial, educational, and experimental demonstration purposes.