Refactor testing console exporter

[ocelotl]

Fixes #2876

[aabmass]

Based on the description in the issue, it sounds like the problem is the test cases are not calling shutdown() on the SDK which causes the background thread to leak in the tests and then export() after the test is ended?

[ocelotl]

Based on the description in the issue, it sounds like the problem is the test cases are not calling shutdown() on the SDK which causes the background thread to leak in the tests and then export() after the test is ended?

We automatically call shutdown() on exit (using atexit in the MeterProvider), so we should not need to call shutdown() manually. The issue here is that (if I am understanding Pytest correctly) when the -s option is not passed to Pytest, it keeps stdout open only during the call phase of the tests (the actual testing) and not during the teardown phase of the tests (when Pytest exits and the shutdown function previously registered via atexit.register is called).

I think we could configure the MeterProvider in our tests not to call shutdown() on exit and add a teardown function for Pytest that calls the shutdown() function manually, but this function end up being called in the teardown phase of the tests too, so I think we should get the same problem because by that moment the Pytest would have closed the stdout file already too if the -s option was not passed. That is just an hypotheses, though.

All this being said, we should still always use devnull to capture the output of the ConsoleExporter when testing it because we want to keep the testing output as clean as possible, even when -s is passed.

[aabmass]

Ah I think you're right that it's the atexit handler. I'm guessing pytest patches sys.stdout during the test.

We automatically call shutdown() on exit (using atexit in the MeterProvider), so we should not need to call shutdown() manually.

That is true, but pytest runs all of the test in a single process. So anything that happens in a test that doesn't get torn down will leak into all the tests. E.g. each PeriodicExportingMetricReader instance has a background thread which will leak and could really slow down our tests. I guess this is a separate issue from this one though.

[ocelotl]

Ah I think you're right that it's the atexit handler. I'm guessing pytest patches sys.stdout during the test.

We automatically call shutdown() on exit (using atexit in the MeterProvider), so we should not need to call shutdown() manually.

That is true, but pytest runs all of the test in a single process. So anything that happens in a test that doesn't get torn down will leak into all the tests. E.g. each PeriodicExportingMetricReader instance has a background thread which will leak and could really slow down our tests. I guess this is a separate issue from this one though.

Good point, but I agree this is another issue for another PR.

[ocelotl]

This test is failing in CI (but not in my laptop 🙄). Marking this PR as draft while I investigate this.