Jetson Generative AI – NerfStudio

Vision took a revolutionary leap with Neural Radiance Fields (NeRF)—NerfStudio brings the power of 3D volumetric reconstruction to your scenes, transforming collections of 2D images into immersive 3D neural representations that can be viewed from any angle in real time on your Jetson device.

In this article you’ll learn how to train and visualize high-quality NeRF models using NerfStudio on Jetson, including advanced applications like FruitNeRF for semantic segmentation and fruit counting in agricultural robotics.

Requirements

 

Hardware / Software
Notes
Jetson AGX Orin (64GB/32GB) / Orin NX (16GB)
AGX Orin recommended for best performance
JetPack 6 (L4T r36.x)
Needed for latest pre-built containers
NVMe SSD highly recommended
Storage speed and space critical for training
17.6GB container storage
For nerfstudio container image
≥ 5GB dataset storage
Space for models and datasets

Understanding Neural Radiance Fields

Neural Radiance Fields revolutionize 3D reconstruction by learning to represent scenes as continuous volumetric functions. Unlike traditional photogrammetry, NeRFs can:

  • Synthesize novel viewpoints from sparse input images
  • Handle complex lighting and transparent materials
  • Generate high-quality renderings with photorealistic detail
  • Enable semantic understanding when combined with segmentation (FruitNeRF)

The technology is particularly promising for edge robotics applications, where real-time 3D understanding enables precise navigation, object counting, and environmental mapping.

Step-by-Step Setup

1. Clone and setup jetson-containers

Copy to Clipboard

2. Start the NerfStudio container

Copy to Clipboard

3. Download test data and train your first model

Copy to Clipboard

4. Access the Web UI

When training starts, navigate to:

  • On-device: http://localhost:7007
  • Remote: Replace <jetson-ip> with your board’s address: http://<jetson-ip>:7007

5. Prepare FruitNeRF dataset

For synthetic data (used in this demonstration):

Copy to Clipboard

For real data:

Copy to Clipboard

6. Train a FruitNeRF model

Copy to Clipboard

The training process typically takes 1 hour on Jetson AGX Orin to complete 30,000 training steps.

Troubleshooting

 

Issue Fix
Out of memory during training Reduce camera-res-scale-factor to 0.25 or lower
Slow training performance Use NVMe SSD for dataset storage; reduce image resolution
Web UI not accessible Ensure container ports are exposed; check firewall settings
Dataset format errors Verify nerfstudio dataset format; use COLMAP for pose estimation

NerfStudio in Action

Below is a demonstration of FruitNeRF’s 3D reconstruction capabilities, showcasing how the trained neural radiance field can render novel viewpoints of the synthetic tree dataset with semantic understanding for agricultural applications.

FruitNeRF 3D Render

Experience the power of neural radiance fields with this rendered flythrough of a FruitNeRF model trained on synthetic fruit tree data. The model demonstrates accurate 3D reconstruction with semantic segmentation capabilities for precise fruit counting and agricultural monitoring.

🎥 FruitNeRF Render:

Using Your Own Data

Training on existing datasets is only the beginning. To train NeRF models on your own captured data, you need to process it into the nerfstudio format with known camera poses for each image.

Processing Custom Data

To process your own data, use the following command:

Copy to Clipboard

Supported Data Types

Data Type
Capture Device
Requirements
Processing Speed
Images
Any Camera COLMAP 🐢
Videos Any Camera COLMAP 🐢
360 Data 360 Cameras COLMAP 🐢
 Polycam iOS with LiDAR Polycam App 🐇
KIRI Engine iOS or Android
KIRI Engine App
🐇
Record 3D iOS with LiDAR Record3D app 🐇

 

Images or Video Processing

For standard images or video files, install COLMAP first:

Copy to Clipboard

Process your data:

Copy to Clipboard

Train on your processed data:

Copy to Clipboard

 

The synthetic FruitNeRF data used in this demonstration provides an excellent starting point for learning NerfStudio. For comprehensive guidance on processing your own custom data with various capture methods and devices, refer to the official NerfStudio documentation.

For more information about NerfStudio and advanced configurations, visit the NerfStudio GitHub repository and FruitNeRF project.