Customizing the Viewer

[jettd]

Essential question: Is it possible to modify the viewer to add realtime graphical UI elements?

I am doing some thesis work which involves the nerfstudio viewer. Basically I want to add more tools to help navigate in the scenes, especially large ones or visually noisy ones. My goal is to add:

• minimap (2D birds eye view)
• compass
• waypoint system (to save several location/orientations and immediately 'teleport' to them)

I know that this is possible in theory but the nerfstudio/viser API only allows me to add things like checkboxes and sliders. I expect this would be simple if I was able to add custom HTML elements.

Current hacky solution:

My hack solution is to patch the nerfstudio viewer code. I created a custom class which interacts with the camera to read the position and orientation of the viewer. I use this information to populate basic HTML,JS,CSS elements which are served to a second port during viewing (8008). So the user (me) must open 2 tabs for ports 7007 and 8008 side by side. They can then navigate in the original viewer and the secondary-navigational interface updates as they move.

Here is a screenshot to show how the current implementation is:

I want to better integrate the 2 interfaces, it seems that it would heavily benefit from finding a way to add a new 'Tab' to the Nerfstudio viewer such as 'Navigation' which would contain my 3 primary elements of map, compass, waypoint.

Please help me figure out the best path forward. If i need to post a similar issue in the viser repository let me know, but I expect this is more of a nerfstudio-specific question.

I can also answer any other questions to shed more light on what I am trying to do.