Heterogeneous Agent Reinforcement Learning (HARL) Integration

[Isaacwilliam4]

Contribution

Our team (@some45bucks, @Isaacwilliam4, @Zenif-NIght, @marioharper182) at DIRECTLabs, Utah State University, integrated the Python library HARL into IsaacLab to enable heterogeneous agent reinforcement learning. HARL includes 10 multi-agent reinforcement learning algorithms. Our modified IsaacLab fork with this integration is available at:

IsaacLab-HARL fork: https://github.com/some45bucks/IsaacLab-HARL

	Continuous action space	Discrete action space	Multi-discrete action space
HAPPO	√	√	√
HATRPO	√	√
HAA2C	√	√	√
HADDPG	√
HATD3	√
HAD3QN		√
HASAC	√	√	√
MAPPO	√	√	√
MADDPG	√
MATD3	√

Our current implementation supports HAPPO, HATRPO, HAA2C, and MAPPO.

GTC Presentation

We presented this work at NVIDIA GTC 2025. A recording of the relevant segment can be viewed here:

GTC Video (Segment): https://www.nvidia.com/en-us/on-demand/session/gtc25-S71244/?start=1247&end=1713

Multi-Agent Environments

We created five additional multi-agent environments in IsaacLab to serve as templates for future research. These are fully compatible with the HARL integration.

Environment overview:

Video demonstration:

rl-video-step-80000.mp4

The video shows coordinated ANYmal policies using velocity control in our multi-agent bar environment. The only existing multi-agent environment in IsaacLab prior to our work was the dexterous hand environment.

Paper

Our paper describing this work is currently under review.

@article{haight2024harl,
author    = {Haight, Jacob and Peterson, Isaac and Allred, Christopher and Harper, Mario},
title     = {Heterogeneous Multi-Agent Learning in Isaac Lab: Scalable Simulation for Robotic Collaboration},
journal   = {TBA},
year      = {2024},
}

Acknowledgements

This work builds upon the HARL library developed by PKU-MARL. We used their codebase as the foundation and extended it for integration with IsaacLab.

@article{JMLR\:v25:23-0488,
author  = {Yifan Zhong and Jakub Grudzien Kuba and Xidong Feng and Siyi Hu and Jiaming Ji and Yaodong Yang},
title   = {Heterogeneous-Agent Reinforcement Learning},
journal = {Journal of Machine Learning Research},
year    = {2024},
volume  = {25},
number  = {32},
pages   = {1--67},
url     = {[[http://jmlr.org/papers/v25/23-0488.html}](http://jmlr.org/papers/v25/23-0488.html%7D)](http://jmlr.org/papers/v25/23-0488.html})
}

Original HARL repository: https://github.com/PKU-MARL/HARL
Our modified HARL fork for IsaacLab integration: https://github.com/some45bucks/HARL

[RandomOakForest]

This looks awesome! Thanks for sharing this here!