Memory leak if a test contains the built-in `eval`

[avara1986]

Summary

If I run a test that contains the built-in eval function, a memory leak occurs when executed with pytest-cov.

Reproducer

For example, if I run this code:

File tests_dummy.py

import resource


def test_dummy():
    for i in range(100_000):
        r = eval(f"'a' + '1'")
        assert r == 'a1'

        current_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024

        if i % 500 == 0:
            print(
                f"Round {i} Max RSS: {current_rss}"
            )

When I execute it with --cov="", the memory keeps increasing:

pytest -s --cov="" tests_dummy.py 
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.11.0, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/.../projects/TEST
plugins: cov-6.0.0
collected 1 item                                                                                                                                                                                                  

tests_dummy.py Round 0 Max RSS: 45.87109375
Round 500 Max RSS: 46.12890625
Round 1000 Max RSS: 46.38671875
Round 1500 Max RSS: 46.64453125
Round 2000 Max RSS: 46.64453125
Round 2500 Max RSS: 46.90234375
Round 3000 Max RSS: 47.16015625
Round 3500 Max RSS: 47.41796875
Round 4000 Max RSS: 47.67578125
Round 4500 Max RSS: 47.93359375

However, if I pass --no-cov, the memory remains stable.

pytest -s --no-cov tests_dummy.py
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.11.0, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/.../projects/TEST
plugins: cov-6.0.0
collected 1 item                                                                                                                                                                                                  

tests_dummy.py Round 0 Max RSS: 43.359375
Round 500 Max RSS: 43.359375
Round 1000 Max RSS: 43.359375
Round 1500 Max RSS: 43.359375
Round 2000 Max RSS: 43.359375
Round 2500 Max RSS: 43.359375
Round 3000 Max RSS: 43.359375
Round 3500 Max RSS: 43.359375
Round 4000 Max RSS: 43.359375
Round 4500 Max RSS: 43.359375
Round 5000 Max RSS: 43.359375
...
Round 1647500 Max RSS: 779.58984375

Versions

Python 3.11.0/Python 3.10.5/Python 3.13.1

python -m pip freeze                        
coverage==7.6.10
iniconfig==2.0.0
packaging==24.2
pluggy==1.5.0
pytest==8.3.4
pytest-cov==6.0.0

Config

No .coveragerc
No setup.cfg

[nedbat]

I'm curious if this is causing a problem for you? How did you discover it?

[avara1986]

To give more context, we detected this issue at DataDog in our tracer while running this test:
https://github.com/DataDog/dd-trace-py/blob/main/tests/appsec/iast_aggregated_memcheck/test_aggregated_memleaks.py

This test validates that our tracer detects vulnerabilities (SQLi, CMDi, SSRF...) and ensures that our C++ code doesn’t cause memory leaks. We were trying to include Code Injection detection in the eval function as part of these validations, and that’s when we came across this issue, which we've now reported.

[RonnyPfannschmidt]

Does this happen only on pytest or on plain coverage as well ?

Does it happen with --assert=plain?

[ionelmc]

@avara1986 what is your MALLOC_ARENA_MAX and core count on the test machine? Assuming you running with the defaults and glibc detects a high core count (doesn't matter what is actually available) there is going to be a lot of RSS memory wasted.

[nedbat]

This is an issue with coverage.py. @avara1986 I'm glad to hear that DataDog is getting good use from it. Would they consider some GitHub sponsorship to help with the fix?

[avara1986]

@ionelmc here is the data:

• CPUs: 16
• MALLOC_ARENA_MAX: None

Thanks, @nedbat ! You're right, if I run the script this way, there's still a memory leak using coverage, even without pytest:

def eval_loop():
    for i in range(100_000):
        r = eval(f"'a' + '1'")
        assert r == 'a1'

        current_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024

        if i % 500 == 0:
            print(
                f"Round {i} Max RSS: {current_rss}"
            )
            
if __name__ == "__main__":
    eval_loop()

✅ OK: python tests_dummy.py
❌ LEAK:coverage run tests_dummy.py

Similarly, there’s a leak in these cases with my first pytest example, but not in these others.
✅ OK: pytest -s ...
✅ OK: pytest -s --no-cov ...
❌ LEAK: pytest -s --cov="" tests_dummy.py ...
❌ LEAK: pytest -s --cov="" --assert=plain tests_dummy.py ... <!- @RonnyPfannschmidt there is a leak too
❌ LEAK: coverage run -m pytest -s tests_dummy.py

[ionelmc]

In your case the default for MALLOC_ARENA_MAX would be 128, which is excessive. My suggestion is to set that env var to 1 or 2, then retest. You can read more about it here: https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior

[nedbat]

@avara1986 Any thoughts about sponsorship?

[nedbat]

This is fixed in coverage.py: nedbat/coveragepy@f85d9b7

[avara1986]

Hey @nedbat , we’ve implemented a workaround by disabling coverage for that test since it’s not entirely necessary. However, we’re discussing the possibility of sponsorship internally for the near future. Thanks a lot!

[nedbat]

@avara1986 thanks! ;-)

[nedbat]

This is now released as part of coverage 7.6.11.