[SR-11760] SwiftPM builds inside Docker do not re-use existing build artifacts

[swift-ci]

Previous ID	SR-11760
Radar	rdar://problem/73985925
Original Reporter	MrMage (JIRA User)
Type	Bug

Attachment: Download

Environment

Swift 4.2 all the way till Swift 5.2.4.

Additional Detail from JIRA

Votes	19
Component/s	Package Manager
Labels	Bug
Assignee	None
Priority	Medium

md5: 6c4efa086896ce15ed52fa119adc9b54

Issue Description:

See https://forums.swift.org/t/incremental-builds-with-swift-package-manager-and-linux-docker/26538 for a full description. @aciidb0mb3r has asked to file a Jira ticket about this issue.

My suspicion is that SwiftPM for some reason considers build artifacts in lower Docker layers stale and thinks that it needs to rebuild them.

[beccadax]

@aciidb0mb3r

[swift-ci]

Comment by Eric Stern (JIRA)

As a small (or maybe not so small) addition - support for something along the lines of `swift build --dependencies-only` would be hugely beneficial inside of Dockerized builds once this is resolved, in order to maximize the effect of layer caching.

It's quite clumsy and counterintuitive to make "dummy files" (as shown in the linked discussion) just to overwrite them with actual source code later.

[swift-ci]

Comment by Daniel Alm (JIRA)

Fully agree with Firehed (JIRA User). Creating the dummy files is only a workaround; having an option to build only the dependencies would be a much better solution in the long term.

[swift-ci]

Comment by Daniel Alm (JIRA)

I have investigated a bit more and found the following:

1. This seems to affect only non-Swift dependencies, e.g. C and C++ ones. This is still significant, however, as SwiftNIO packages a full copy BoringSSL, which takes long to compile.

2. The output of stat indeed changes when run in a different Docker layer; in particular, fractional timestamp values are removed. Example stat output from the layer the compilation has been run in:

  File: .build/debug/CNIOBoringSSL.build/module.modulemap
  Size: 163         Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 426531      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:29.897272000 +0000
Modify: 2019-11-19 11:13:29.517272000 +0000
Change: 2019-11-19 11:13:29.517272000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.d
  Size: 4546        Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 427653      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:48.345814000 +0000
Modify: 2019-11-19 11:13:48.335814000 +0000
Change: 2019-11-19 11:13:48.335814000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.o
  Size: 8664        Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 427635      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:48.285814000 +0000
Modify: 2019-11-19 11:13:48.335814000 +0000
Change: 2019-11-19 11:13:48.335814000 +0000
 Birth: -

And from a "later" layer:

  File: .build/debug/CNIOBoringSSL.build/module.modulemap
  Size: 163         Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 2930655     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:29.000000000 +0000
Modify: 2019-11-19 11:13:29.000000000 +0000
Change: 2019-11-19 11:14:10.058600000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.d
  Size: 4546        Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 2930545     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:48.000000000 +0000
Modify: 2019-11-19 11:13:48.000000000 +0000
Change: 2019-11-19 11:14:09.948600000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.o
  Size: 8664        Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 2930546     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:48.000000000 +0000
Modify: 2019-11-19 11:13:48.000000000 +0000
Change: 2019-11-19 11:14:09.948600000 +0000
 Birth: -

Unfortunately, even packaging the build archive in a POSIX tar which is supposed to fully preserves timestamps yields different timestamps, as well as causing a rebuild of all non-Swift dependencies:

  File: .build/debug/CNIOBoringSSL.build/module.modulemap
  Size: 163         Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 428027      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:14:15.888600000 +0000
Modify: 2019-11-19 11:13:29.517272000 +0000
Change: 2019-11-19 11:14:15.888600000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.d
  Size: 4546        Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 696234      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:14:15.958600000 +0000
Modify: 2019-11-19 11:13:48.335814000 +0000
Change: 2019-11-19 11:14:15.958600000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.o
  Size: 8664        Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 697210      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:14:16.008600000 +0000
Modify: 2019-11-19 11:13:48.335814000 +0000
Change: 2019-11-19 11:14:16.008600000 +0000
 Birth: -

1. I have uploaded a full build log to https://gist.github.com/MrMage/943f0de9009ee183ae66591eafad6c7a and a sample project to https://github.com/MrMage/docker-caching-test. The Dockerfile is a bit messy with all these experiments, but I think you can see what I had tried to somehow force Swift to use the build cache.

@aciidb0mb3r I would really appreciate if you could take a look. Thanks!

[aciidgh]

Thanks for the investigation! If the timestamp is changing then I suspect we'll have to use file hashing to determine if the manifest changed.

[swift-ci]

Comment by Daniel Alm (JIRA)

@aciidb0mb3r another option might be to use only the integer part of the timestamp, dropping the fractional part.

[swift-ci]

Comment by Pierpaolo Frasa (JIRA)

Is there any update on this issue?

[swift-ci]

Comment by Will Lisac (JIRA)

Something like `swift build --dependencies-only` would be a great addition.

I'm also using the dummy files technique and it becomes increasingly tedious as the number of targets grows.

[swift-ci]

Comment by Bridger Maxwell (JIRA)

Would it be appropriate to round the timestamp here in the Swift Package Manager source?

https://github.com/apple/swift-package-manager/blob/7f35752db78977ff5e2ca3c38cc5427091647698/TSC/Sources/TSCBasic/FileInfo.swift#L55

[swift-ci]

Comment by Daniel Alm (JIRA)

This might help, provided it also gains support for Linux: #2774

[drewcrawford]

I think one cause of detecting artifacts as "stale" is that llbuild looks at the inode number. There is an open PR to change this algorithm swiftlang/swift-llbuild#641

I believe that separately, there is a way to adjust llbuild behavior by supplying a key to the right location. According to the documentation

> A file-system field may be supplied that toggles how the build system handles file system use. The default mode uses detailed stat information for detecting file changes. The device-agnostic mode will ignore device and inode values.

[swift-ci]

Comment by Bridger Maxwell (JIRA)

The blog post for Swift 5.3 mentions better caching https://swift.org/blog/swift-5-3-released :

Faster incremental build times by avoiding duplicated compiler work across source files, and more accurately identifying code that has not changed from the previous build

I can't find any other information about this change. Any chance it relates to this bug? Lately I'm desperate for something to speed up my Swift builds.

[swift-ci]

Comment by Daniel R (JIRA)

+1 on something like `swift build --dependencies-only`

I'm also creating dummy files as a workaround.

[weissi]

@swift-ci create

[weissi]

My colleague Euan Harris just made me aware of a new (experimental) Docker feature in BuildKit which can alleviate this problem. If you create your Dockerfile like so

# syntax = docker/dockerfile:1.2
FROM 5.3-focal AS builder
ADD . /code/
WORKDIR /code
RUN --mount=type=cache,target=/code/.build swift build -c release && cp /code/.build/release/my-binary /app

FROM swift:5.3-focal-slim AS runner
COPY --from=builder /app /app
ENTRYPOINT /app

then it will rebuild fast. You will need to use docker buildx build . and not docker build .. (Partial) docs about BuildKit https://docs.docker.com/buildx/working-with-buildx/ and https://docs.docker.com/develop/develop-images/build_enhancements/ .

As an explainer, the --mount=type=cache,target=/code/.build will create a cache volume that is persisted to /code/.build. It keeps the inode numbers as well as the timestamps intact and therefore SwiftPM will not rebuild unnecessarily.

Long story short, if you can, use BuildKit.

If for whatever reason you cannot use BuildKit, you can contact me directly, I have a hack that lets you use cached builds with plain old stuff (no BuildKit).