Is PYTHONUNBUFFERED=TRUE a good idea?

[MartinThoma]

I had the experience that printing to STDOUT with Python is way slower than writing to a file. I think this is due to buffering.

Due to AWS Cloudwatch only logging the STDOUT / STDERR, Sagemaker projects have to print instead of writing to files.

Is it really a good idea to have PYTHONUNBUFFERED=TRUE (and is it really TRUE or rather 1?). Will performance suffer from this?

[FarhanTejani]

Setting PYTHONUNBUFFERED=TRUE or PYTHONUNBUFFERED=1 (they are equivalent) allows for log messages to be immediately dumped to the stream instead of being buffered. This is useful for receiving timely log messages and avoiding situations where the application crashes without emitting a relevant message due to the message being "stuck" in a buffer.

As for performance, there can be some (minor) loss that comes with using unbuffered I/O. To mitigate this, I would recommend limiting the number of log messages. If it is a significant concern, one can always leave buffered I/O on and manually flush the buffer when necessary.

[MartinThoma]

I close this now as it is more a question. You are aware of the potential problem, so I don't see a point in leaving this open.