Mock evals reference camera names from local config, not model

[brainwavecoder9]

Starting this as a conversation to confirm approach before submitting a PR.

Conceptually I feel good about this proposal, but I'd like to get confirmation from others who have been around longer than I have to confirm that this is a desired approach. Also mentioning related issues #1095 and #1078.

The Problem

1. In control_robot.py, when you run with --robot.mock=true, the function make_robot_from_config creates a robot with mocked components based on the local configuration.
2. The mocked cameras use the names defined in the local configuration file (configs.py), not the names used in the model.
3. When the policy tries to normalize inputs in normalize.py, it looks for statistics using the observation keys from the mocked robot, but can't find them because they don't match the names used when training the model.

Proposed Solution

Adapt the mock robot creation process to use camera names from the model's normalization statistics when mock mode is enabled and a policy is provided.

Specific Changes:

1. Add a utility function to extract camera names from policy

   • File: lerobot/common/policies/utils.py
   • Add function extract_camera_names_from_policy(policy) that:
     • Takes a policy object
     • Extracts camera names from the normalization buffers
     • Returns a dictionary of camera names found in the policy

2. Modify robot creation to adapt camera names in mock mode

   • File: lerobot/common/robot_devices/robots/utils.py
   • Update make_robot_from_config(config, policy=None) to:
     • Accept an optional policy parameter
     • When in mock mode and a policy is provided, extract camera names
     • Replace the mock camera configuration with cameras matching the policy's expected names
     • Preserve all other robot configuration

3. Update the control flow to load policy before creating the robot

   • File: lerobot/scripts/control_robot.py
   • Modify control_robot(cfg) to:
     • Load policy first when in mock mode with a policy path
     • Pass the policy to make_robot_from_config
     • Ensure we don't reload the policy later in the execution

I expect that this is an easy one to accept because it works with existing framework, only affects mock mode with policies, and uses actual camera names from the model (assuming they are available) but would be easy to fallback to local config if not.

[brainwavecoder9]

FYI was able to prove this out:
https://github.com/brainwavecollective/lerobot/tree/mock_model_cams

However... going to wait on #777 before finalizing any changes. Of note, it looks like the robot.mock still relies on local physical robot config for some reason. I think this is less of a concern given that robot configs are more likely to be consistent, and others are less likely to be running evals for robots that they don't have configured, but it would be nice to have.

Along those lines... probably would be good to think about having configs and cache in gitignore and possibly refactor so that all robots aren't lumped together (haven't looked at #777 in depth so possible this is addressed there).

In any case, will revisit after those updates come through.

PS - Sounds like it's possible that this is at least loosely related to #699 and #694 as well.

[aliberts]

Hi, thanks for your proposal!

Just so that I understand the use case here: you're trying to run a pretrained policy on a mocked robot for testing/debug purposes?

We are removing the mock argument from all hardware code in #777, because we are fundamentally changing the way we're testing our code so this won't be necessary for mock robots specifically. Additionally, that PR introduces significant changes in the abstractions (e.g. ManipulatorRobot is gone). There will still be a MockRobot in the tests directory for testing certain parts of the code that manipulate robots but it will be quite different.

On the more general problem of matching robots features keys with the ones expected by the policies, we're thinking of having a pipeline system -much like in most ML frameworks- to handle these translations. WDYT?

probably would be good to think about having configs and cache in gitignore

Calibration files are written to ~/.cache/huggingface/lerobot/calibration by default in the upcoming redesign ;)

[brainwavecoder9]

Thanks @aliberts - yes this was for testing hardware setup. It was a nice to have but I don't necessarily need it to persist.

This conversation is possibly going beyond the scope of this, but since this was where I was pointed to have a design discussion let's go for it :)

Let's say we have the following categories for related assets:

Operational Cache

These are your temporary files, downloaded models, etc., and they are fully ephemeral  
Wiping this out might slow down operations as things are recreated, but there wouldn't be any impact.

Generated Output

Valuable results, like training, logs, checkpoints, etc.
Expensive to recreate and needs to be protected, but does not necessarily need to persist
Totally up to the user to determine what happens with this, can archive/delete at will

Static User Configuration Files

User/system-specific configuration like USB ports for arms and cameras
In most cases these will be set once and never touched again
Will require templates/placeholders for users, but otherwise not part of repo

Generated Environmental Details / State / Calibration

Like the static config files these are user/system-specific configuration files (e.g., calibration)
However, in most cases the user does not necessarily need to know about or interact with these

Summary

Category	Storage Location	Visibility	Git Status	Durability
Operational Cache	.cache	Hidden	Untracked	Ephemeral
Generated Output	output	Visible	Untracked	User-determined
User Configuration	TBD	Visible	Untracked, but templates necessary	Persistent
Generated State	.TBD	Hidden	Untracked	Persistent

TBD: Generated Calibration

I would consider this to be a type of environmental state file. If we assume the cache is considered to be fully ephemeral then I'm not sure that .cache is necessarily the right place. We might want something like .config, .environment, .state, etc. (not .env since that is taken) to differentiate between these automatic, non-interactive, local cache-like files that have a separate requirement for durability. I didn't include calibration in the list because I think ideally it would be more generic so that other potential state files could be accommodated (e.g., .state/calibration)

TBD: User Configuration

The elephant in the room :) This needs to be revisited separately; it would be a massive improvement for the project and developer experience if the configuration were abstracted from the implementation. I burned a bunch of time when I took a crack at this before I knew about the move to Draccus was happening, another user explored a different solution this week, and it sounds like now there are more design changes coming which would also block potential related improvements.

My initial thought is to align with existing convention and simplify with config (for user-managed configuration files) and .config (for the automatic state/calibration files) because it would be quick and easy to reference mentally, but I do think it's worth a bigger discussion because other people have ideas and there a lot of good ways we could approach this.

TL;DR I would suggest

1. Place the auto-generated calibration data into a new hidden/durable folder (.state/.config/other tbd)
2. Abstract static user config. A solution for this needs to be designed and will require a bigger conversation. If done right it could be a relatively simple solution that would make a lot of people very happy - and I've started Abstraction needed for user-editable configuration #1145 for this

[brainwavecoder9]

PS - I also agree about a pipeline, and I think the above could also serve as as a foundation for mapping models/features to local config