# 3D Head Modeling To turn a video or photo sequence of your head into a usable 3D model, you will use a process called **Photogrammetry** or a newer AI technique called **3D Gaussian Splatting**. Both methods analyze how the perspective of your face shifts across different frames to calculate geometry and depth. For the highest quality results, running the processing locally using Nvidia CUDA acceleration and terminal-based tools will give you the most control over the final mesh. Here is the exact pipeline to capture and generate the model: 1. **Capture the Footage:** Your face must remain completely static. The biggest enemy of 3D reconstruction is movement. Even blinking or shifting your jaw will confuse the software. * **Lighting:** Ensure your face is evenly lit without harsh, moving shadows. Natural overcast daylight works best. * **The Subject:** Stare at a fixed point on the wall. Do not blink if possible, keep a neutral expression, and do not rotate your own head. * **The Camera:** Move the camera slowly and smoothly in a 360-degree circle around your head. Do two passes: one at eye level, and one looking slightly down from above. 2. **Extract Frames from Video:** If you recorded a video, you need to extract it into individual, high-quality image frames. You can do this quickly in the terminal using `ffmpeg`. Create a folder for your frames, navigate to it, and run: ```bash ffmpeg -i input_video.mp4 -r 2 -qscale:v 2 frames/%04d.jpg ``` *(This extracts 2 frames per second at high quality. Adjust the `-r` value depending on how fast you moved the camera; you want roughly 50 to 150 overlapping photos in total).* 3. **Process the 3D Reconstruction:** To process the images into a 3D model, you have two excellent local, GPU-accelerated options depending on your preferred workflow: **Option A: Traditional Photogrammetry (Meshroom)** [Meshroom](https://alicevision.org/#meshroom) is an open-source, node-based photogrammetry tool that relies heavily on CUDA. It directly outputs a traditional 3D `.obj` mesh with texture maps, which is perfect for VFX pipelines. You simply drag and drop your extracted frames into the interface and click "Start." **Option B: AI Neural Radiance Fields (NeRFstudio)** If you are comfortable in a Python environment, [NeRFstudio](https://docs.nerf.studio/) uses AI to generate highly photorealistic representations of scenes. It is exceptional at handling complex details like hair, which traditional photogrammetry often struggles with. You can install it via your package manager and run the processing through the CLI: ```bash ns-process-data images --data data/frames/ --output-dir data/nerf_output/ ns-train splatfacto --data data/nerf_output/ ``` Once trained, NeRFstudio allows you to export the result as a standard mesh or point cloud. 4. **Clean Up the Mesh in Blender:** Raw photogrammetry scans usually include the background room and some rough geometry. 1. Import the generated `.obj` or `.ply` file into Blender. 2. Enter **Edit Mode** and delete the background geometry (walls, the room behind you). 3. Use the **Smooth** sculpting brush to fix any bumpy artifacts on the skin or hair. 4. Export the isolated, clean head as a `.fbx` or `.obj` file to use in your VFX morphing sequence. # Photogrammetry & AI-Aided Photogrammetry ## What is "-grammetry"? The suffix **"-grammetry"** comes from Greek: - **"gramma"** → written/drawn/recorded - **"-metry"** → measurement So **"-grammetry" = measurement from recorded data**. | Term | Meaning | |------|---------| | **Photo**grammetry | Measurement from **photos** | | **Stereo**grammetry | Measurement from **stereo pairs** | | **Lidar**grammetry | Measurement from **laser scans** | --- ## What is Photogrammetry? **Photogrammetry** is the science of extracting **3D geometric information** (shape, position, size) from **2D photographs**. The core idea: > If you photograph the same object from multiple angles, you can mathematically reconstruct its 3D structure. ### How it works (classical pipeline): ``` 📸 Multiple photos → 🔍 Feature matching → 📐 Triangulation → ☁️ Point cloud → 🧊 3D mesh → 🎨 Textured model ``` 1. **Feature Detection** — Find distinctive keypoints in each image (corners, edges) 2. **Feature Matching** — Match the same keypoints across different photos 3. **Structure from Motion (SfM)** — Estimate camera positions and a sparse 3D point cloud 4. **Multi-View Stereo (MVS)** — Generate a dense point cloud 5. **Mesh Reconstruction** — Connect points into a surface mesh 6. **Texturing** — Project original photos onto the mesh --- ## AI-Aided Photogrammetry Traditional photogrammetry is math-heavy and struggles with: - Textureless/reflective surfaces - Sparse photo coverage - Low-light or blurry images **AI steps in to solve these problems:** --- ### 🔑 1. AI Feature Matching (Replacing SIFT/ORB) | Classical | AI | |-----------|-----| | SIFT, ORB | **SuperPoint + SuperGlue** | | Sensitive to lighting | Robust to lighting/viewpoint changes | | Hand-crafted | Learned from data | **SuperGlue** uses a Graph Neural Network to match features across images with dramatically higher accuracy. --- ### 🌫️ 2. Depth Estimation (Monocular) AI can estimate depth from a **single image** — impossible classically. | Tool | Description | |------|-------------| | **Depth Anything V2** | Meta's foundation model for monocular depth | | **MiDaS** | Multi-scale depth, works on any image | | **ZoeDepth** | Metric (real-world unit) depth estimation | ``` Single photo → AI Depth Model → Depth map → Fused into 3D reconstruction ``` --- ### 🧊 3. Neural Radiance Fields (NeRF) A revolutionary AI approach — instead of explicit geometry, a neural network **learns to represent the scene as a continuous 3D volume**. ``` Photos → Train NeRF network → Render from ANY new viewpoint ``` - **NeRF** (2020) — original paper, slow - **Instant-NGP** (NVIDIA, 2022) — seconds to train - **3D Gaussian Splatting** (2023) — real-time rendering, current SOTA --- ### ✨ 4. Gaussian Splatting (Current SOTA) Instead of a mesh or neural field, the scene is represented as **millions of 3D Gaussians** (fuzzy ellipsoids), each with color and opacity. ``` 📸 Photos → SfM (camera poses) → 3DGS training → Real-time interactive 3D scene ``` **Key tools:** - `gaussian-splatting` (original repo) - **Luma AI** — consumer-friendly 3DGS - **Polycam** — mobile photogrammetry with AI --- ### 🛠️ 5. AI-Enhanced Traditional Pipelines | Tool | What AI Does | |------|-------------| | **RealityCapture** | AI masking, upscaling | | **Metashape (Agisoft)** | AI-based point cloud filtering | | **Meshroom** | Open-source, AliceVision pipeline | | **DroneDeploy** | Aerial AI photogrammetry | | **OpenMVG + OpenMVS** | Classic but pairs well with AI depth | --- ## Summary: Classical vs AI ``` Classical Photogrammetry: AI-Aided Photogrammetry: 📐 Needs many overlapping photos 🤖 Can work with fewer photos ❌ Fails on textureless surfaces ✅ Handles reflective/blank surfaces 🐢 Slow dense matching ⚡ Fast learned features 📦 Outputs explicit mesh 🌊 Can output implicit neural fields ``` --- ## Where to Start? If you want to **try it yourself**: 1. **Easiest**: [Luma AI](https://lumalabs.ai) — just take a video with your phone 2. **Prosumer**: [Polycam](https://poly.cam) — mobile app, excellent results 3. **Open Source**: [Meshroom](https://alicevision.org/) + [COLMAP](https://colmap.github.io/) 4. **Cutting-edge**: [gaussian-splatting](https://github.com/graphdeco-inria/gaussian-splatting) on GitHub Want me to dive deeper into any specific part — like NeRF, Gaussian Splatting, or how to run a pipeline yourself?