Way to check if containers have booted

[ribalba]

Flows are currently failing because containers are taking longer to boot than we currently wait in the runner.py. Currently I modify the runner to have a custom wait.
I see the following options to fix this:

• Add a variable to the usage_scenario where we can set a custom wait time if we know that containers might take longer to boot
• Add a variable to the config.yml to wait in the boot phase. Not a big fan of this as development with the GMT is already quite time consuming/ annoying based on all the sleeps. And this would be for all runs even the ones we don't need the sleep as the containers don't need the time to boot.
• Offer some checker to see if a container has booted with a timeout[1][2]. I imagine something like:

...
services:
  a:
    container_name: a
    image: a
    healthcheck:
      test: ["CMD-SHELL", "true"]
      interval: 10s
      timeout: 5s
      retries: 5

As soon as we see this in the services we start a routine to check that all containers have started before continuing.

[1] https://docs.docker.com/engine/reference/builder/#healthcheck
[2] https://docs.docker.com/compose/compose-file/compose-file-v3/#healthcheck

[ArneTR]

Nice find with the healthcheck option, was not aware that existed.

To my understanding when a container is not healthy an event is triggered, so we could leverage that directly and not roll on logic on top.
I would really like to understand how that is actually used in the wild then. Is the signal only know to the host, or does the container also get the signal? If so, does the container typically respond with a rountine? Or if only the host gets the signal, is there a default behaviour .... like for instance restarting the container?

However I think the most tricky part is how to integrate it into our measurement model without introducing randomness or overhead. Technically it can also be integrated between the measurement steps to not create randomness in the data. However the question is then, is the lazy startup time not actually part of the boot process?

I think this needs quite some further debate ...

[ArneTR]

Referencing: #567

[ArneTR]

Healthcheck has now been implemented #637