extra notes for installing and working with nvidia instant-ngp

instant-ngp_extra_notes.md

instant-ngp Extra Notes

Installing instant-ngp

Installation Instructions: https://github.com/NVlabs/instant-ngp?tab=readme-ov-file#installation

• If you're running a CUDA capable gpu, change the download_colmap.bat scricpt to downlad colmap with CUDA:

echo Downloading COLMAP...
:: powershell -Command "(New-Object Net.WebClient).DownloadFile('https://github.com/colmap/colmap/releases/download/3.7/COLMAP-3.7-windows-no-cuda.zip', 'colmap.zip')"
powershell -Command "(New-Object Net.WebClient).DownloadFile('https://github.com/colmap/colmap/releases/download/3.9.1/COLMAP-3.9.1-windows-cuda.zip', 'colmap.zip')"

From Video

1. Download the video into a instant-ngp\data\exmaple folder.
2. Convert the video into frames using scripts\colmap_to_nerf.py. The goal is to have 50-150 non-blurry images:

python scripts\colmap2nerf.py --video_in data\nerf\example\input.MOV --video_fps 2 --run_colmap --overwrite

3. Use colmap to generate the camera transforms (transforms.json) that instant-ngp needs:

cd data\nerf\examaple
python ..\..\..\scripts\colmap2nerf.py  --colmap_matcher exhaustive --run_colmap --aabb_scale 16 --overwrite

4. Run instant-ngp on the images and camera transforms. It will auto-detect files in the \images folder:

cd ..\..\..
instant-ngp data\nerf\example

From Images

1. Put images in an \images folder within a instant-ngp\data\exmaple folder.
2. Use colmap to generate the camera transforms (transforms.json) that instant-ngp needs:

cd data\nerf\examaple
python ..\..\..\scripts\colmap2nerf.py  --colmap_matcher exhaustive --run_colmap --aabb_scale 16 --overwrite

3. Run instant-ngp on the images and camera transforms. It will auto-detect files in the \images folder:

cd ..\..\..
instant-ngp data\nerf\example

Tips & Tricks

• It's not that quick (on a laptop RTX 3090).
  • Colmap video to images: 10-15 minutes.
  • Colmap camera transforms: 10-15 minutes.
  • Rendering 5-30 minutes, depending on settings.
  • Anticipate ~30-45 minutes for pipeline processing
• The mesh quality is not that great.
  • Good as a reference.
  • Not good enough as a "game ready" asset.
• Keep the camera pointed at the same focal point to help with convergence.
  • instant-ngp bases the position / orientation of the final output on the average focal point
• Image Resolution matters a lot
  • The amount of "zoom-in" detail depends on image resolution.
• Fast shutter speed matters a lot
  • Output is degraded by images with motion blur.
  • Taking "slow motion" high-speed video seems to work better
• Standard photogrammetry stuff:
  • Avoid reflective surfaces.
  • Use consistent lighting.
  • ...
• Useful video tutorial: https://www.youtube.com/watch?v=3TWxO1PftMc
• More useful Ttips: Tips for training NeRF models with Instant Neural Graphics Primitives