Time drift in Moby VM and containers caused by system sleep

[alanbrent]

Expected behavior

When the Mac wakes from sleep, the Moby VM should resync its local system time with the Mac's system time.

Actual behavior

Moby VM time simply resumes counting from the time the Mac went to sleep. This results in strange and many times difficult to troubleshoot issues with software running in containers.

Information

Docker for Mac: version: win-beta-v1.12.0-rc5-beta21-73-g9cd1dc2
OS X: version 10.11.6 (build: 15G31)
logs: /tmp/1D6BCF46-49C8-45BC-86AC-EF873AB86DFB/20160730-113336.tar.gz
[OK]     docker-cli
[OK]     app
[OK]     moby-syslog
[OK]     disk
[OK]     virtualization
[OK]     system
[OK]     menubar
[OK]     osxfs
[OK]     db
[OK]     slirp
[OK]     moby-console
[OK]     logs
[OK]     vmnetd
[OK]     env
[OK]     moby
[OK]     driver.amd64-linux

Steps to reproduce the behavior

1. put Mac to sleep, wait n seconds/minutes/hours
2. wake Mac from sleep
3. Issue date in container or Moby VM and see that it outputs $(date) - n

[alanbrent]

I was able to reproduce this in the GA release, but it looks like it then caught up. I'll check on this when I leave work and return home, and if it's truly fixed then I'll close the issue.

[alanbrent]

The issue has cropped back up on my Moby VM. I put my Mac to sleep when I left for work, and it's currently running ~47min behind, and the same is the case for any containers I run.

(Mac Shell)->date
Fri Jul 29 19:35:25 PDT 2016
---
moby login: root
Welcome to the Moby alpha, based on Alpine Linux.
moby:~# date
Sat Jul 30 01:48:52 UTC 2016
---
(Mac Shell)->docker run --rm -it ruby bash
root@c9e297c5bef0:/# date
Sat Jul 30 01:49:48 UTC 2016

UPDATE: Moby VM is now 16h behind after the Mac was in sleep mode all night. This is definitely still a problem.

[ijc]

Hi @alanbrent,

The time should resync after a matter of minutes (if not seconds). For how long is it staying out of sync for you?

Do you have NTP on the OSX host? What does ntpq -p show before and after sleep?

It'd also be useful to see the equivalent (chronyc sources) from within the VM, to get a shell use docker run -it --privileged --pid=host alpine:3.4 nsenter -t 1 -m -n sh

There was supposed to be a section in the troubleshooting guide about NTP issues and possible workarounds, but it looks like it somehow got undone by a rebase somewhere along the line. The diff was:

 +* If system does not have access to an NTP server then after a hibernate the time seen by Docker for Mac may be considerably out of sync with the host. Furthermore the time may slowly drift out of sync during use. The time can be manually reset after hibernation by running:
 +
 +        docker run --rm --privileged alpine hwclock -s
 +
 +  Or alternatively to resolve both issues you may like to add the local clock as a low priority (high stratum) fallback NTP time source for the host by editing the host's `/etc/ntp-restrict.conf` to add:
 +
 +        server 127.127.1.1              # LCL, local clock
 +        fudge  127.127.1.1 stratum 12   # increase stratum
 +
 +  And then restarting the NTP service with:
 +
 +        sudo launchctl unload /System/Library/LaunchDaemons/org.ntp.ntpd.plist
 +        sudo launchctl load /System/Library/LaunchDaemons/org.ntp.ntpd.plist

I'll look at getting this section of the docs reinstated.

[alanbrent]

@ijc25

The time should resync after a matter of minutes (if not seconds). For how long is it staying out of sync for you?

It stays out of sync until I manually restart the Moby VM.

Do you have NTP on the OSX host? What does ntpq -p show before and after sleep?

This is what it shows now, which would be "after" sleep. I'd have to reboot to give you a true reading of what it looks like "before" sleep, and I can't stop to do that at the moment.

>ntpq -p localhost: timed out, nothing received

There was supposed to be a section in the troubleshooting guide about NTP issues and possible workarounds, but it looks like it somehow got undone by a rebase somewhere along the line. The diff was:

I ran hwclock -s directly on the Moby VM* (rather than from within a privileged container) and that immediately cleared up the time drift.

* via screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

[ketan]

I can confirm this is still a problem with the latest beta 1.12.0-beta21

[alanbrent]

That's strange @ketan, because I actually came here to report that it appears to be fixed for me in the 1.12.0-beta21 (build: 11019).

[alanbrent]

Update: i had to reboot my machine the other day and currently I am getting a response from ntpq, so perhaps that really is the center point of this issue:

->ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*usdal2-ntp-002. .GPSs.           1 u  254  256  377   50.561  620.549   2.355

[alanbrent]

Update

So I ran into nasty issues with the beta channel, tried reverting to the stable channel but it didn't fix it. I ended up completely uninstalling Docker and reinstalling stable from scratch. At this point, the time drift issue has not recurred. I'm not sure if this should be considered related, because I've yet to see a no-response to ntpq -p since my last reboot.

[dave-tucker]

@alanbrent @ketan are either of you still seeing this issue?

If not, I'm tempted to close as we have now updated the troubleshooting documentation