contest: loadavg: wait for dirty page writeback

It looks like AWS machines stall very painfully for some time
after the builds because of pending disk IO. The b/w cap was
increased on our instances but still there is a few minutes
of lag between when we decide the that the load has gone
down and when machine has recovered from the builds.

Wait for dirty memory to go down to ~100MB.

Since we have two conditions now lower the stable count to 3.

Signed-off-by: Jakub Kicinski <[email protected]>

Author: kuba-moo

contest/remote/lib/loadavg.py

@@ -4,21 +4,32 @@
 import time
 
 
-def wait_loadavg(target, check_ival=30, stable_cnt=4):
+def get_dirty_mem():
+    """ Get amount of dirty mem, returns value in MB """
+    with open("/proc/meminfo", "r") as fp:
+        lines = fp.read().split("\n")
+        dirty = list(filter(lambda a: "Dirty" in a, lines))[0]
+        return int(dirty.split(" ")[-2]) / 1000
+
+
+def wait_loadavg(target, dirty_max=100, check_ival=30, stable_cnt=3):
     """
     Wait for loadavg to drop but be careful at the start, the load
     may have not ramped up, yet, so if we ungate early whoever is waiting
     will experience the overload.
     """
+
+    seen_stable = 0
     while target is not None:
         load, _, _ = os.getloadavg()
+        dirty = get_dirty_mem()
 
-        if load <= target:
-            if stable_cnt == 0:
+        if load <= target and dirty <= dirty_max:
+            if seen_stable >= stable_cnt:
                 break
-            stable_cnt -= 1
+            seen_stable += 1
         else:
-            stable_cnt = 0
+            seen_stable = 0
 
-        print(f"Waiting for loadavg to decrease: {load} > {target} ({stable_cnt})")
+        print(f"Waiting for loadavg to decrease: CPU: {load} > {target} Dirty Mem: {dirty} > {dirty_max} MB ({seen_stable})")
         time.sleep(check_ival)