cmd: ex: standalone docker garbage collector #16796

sjenning · 2017-10-11T03:06:36Z

This implements a standalone docker GC daemon for use in OpenShift when using a non-docker runtime (e.g. cri-o) but using docker for builds.

https://trello.com/c/iO9d9R1w

$ openshift ex dockergc --help
Perform garbage collection to free space in docker storage 

If the OpenShift node is configured to use a container runtime other than docker, docker will still be used to do
builds.  However OpenShift itself will not manage the docker storage since it is not the container runtime for pods. 

This utility allows garbage collection to do be done on the docker storage.

Usage:
  openshift ex dockergc [NAME] [options]

Examples:
  # Perform garbage collection with the default settings
  openshift ex dockergc

Options:
      --dry-run=false: If true, show the result of the operation without performing it.
      --image-gc-high-threshold=80: The percent of disk usage after which image garbage collection is always run.
      --image-gc-low-threshold=60: The percent of disk usage before which image garbage collection is never run. Lowest
disk usage to garbage collect to.
      --minimum-ttl-duration=1h0m0s: Minimum age for a container or unused image before it is garbage collected.
Examples: '300ms', '10s' or '2h45m'.
  -o, --output='': Output results as yaml or json instead of executing, or use name for succint output (resource/name).
  -a, --show-all=false: When printing, show all resources (default hide terminated pods.)
      --show-labels=false: When printing, show all labels as the last column (default hide labels column)
      --sort-by='': If non-empty, sort list types using this field specification.  The field specification is expressed
as a JSONPath expression (e.g. '{.metadata.name}'). The field in the API resource specified by this JSONPath expression
must be an integer or a string.
      --template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The
template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].

Use "openshift options" for a list of global command-line options (applies to all commands).

@derekwaynecarr @vikaschoudhary16

derekwaynecarr · 2017-10-11T13:59:32Z

can you add a yaml demonstrating how you would run this on cluster?

csrwng · 2017-10-11T14:06:40Z

/unassign

sjenning · 2017-10-17T04:32:44Z

@derekwaynecarr added the example daemonset spec and serviceaccount with required scc

enj · 2017-10-17T14:50:59Z

/unassign

Missing too much context to review 😄

sjenning · 2017-10-17T18:34:30Z

/retest

flake #16414

derekwaynecarr · 2017-10-18T17:33:27Z

examples/dockergc/dockergc-ds.yaml

+          resources:
+            requests:
+              memory: 30Mi
+              cpu: 100m


can we size less? maybe 50m? how did you get to this number?

copied it from node-exporter 😄

derekwaynecarr · 2017-10-18T17:35:06Z

pkg/oc/experimental/dockergc/dockergc.go

+		Perform garbage collection to free space in docker storage
+
+		If the OpenShift node is configured to use a container runtime other than docker,
+		docker will still be used to do builds.  However OpenShift itself will not


derekwaynecarr · 2017-10-18T17:37:30Z

pkg/oc/experimental/dockergc/dockergc.go

+}
+
+func doGarbageCollection(ctx context.Context, client *dockerapi.Client, options *dockerGCConfigCmdOptions, rootDir string) error {
+	capacityBytes, usageBytes, err := getRootDirInfo(rootDir)


can we do a log at each gc interval?

it would be good to have something that logs before gathering root dir info

derekwaynecarr · 2017-10-18T17:39:37Z

@sjenning just a few nits.

derekwaynecarr · 2017-10-18T17:39:52Z

running out of disk space on a cri-o server would be a bug.

/kind bug

liggitt · 2017-10-18T17:44:56Z

pkg/oc/experimental/dockergc/dockergc.go

+
+	dockerapi "github.com/docker/engine-api/client"
+	dockertypes "github.com/docker/engine-api/types"
+	dockerfilters "github.com/docker/engine-api/types/filters"


as long as these dependencies 1) already exist and 2) aren't ones we're trying to delete as part of the oc split (I don't think they are... @deads2k?), hanging this under experimental doesn't bother me

as long as these dependencies 1) already exist and 2) aren't ones we're trying to delete as part of the oc split (I don't think they are... @deads2k?), hanging this under experimental doesn't bother me

We do want to see this leave as part of oc. Is this command logically an oc adm command? If so, then openshift ex is ok and we'll move it to oc ex later.

@deads2k it should actually not be oc but openshift ex. the command is temporary and is avoiding us having to ship an official product image.

@sjenning i think /pkg/cmd/openshift experimental is the right home for this. similar to those other experimental commands.

moved discussion to irc. agree w/ @deads2k that we may want to handle this similar to kubernetes-namespace-reservation

@sjenning -- @deads2k and i spoke in person. when oc moves out of tree, we will move this command with it. at that point, we will need to make sure our ops team has the updated daemonset.

sjenning · 2017-10-18T21:52:12Z

/retest

derekwaynecarr · 2017-10-19T15:23:41Z

/test extended_conformance_install_update

derekwaynecarr · 2017-10-19T15:29:57Z

/approve
/lgtm

openshift-merge-robot · 2017-10-19T15:30:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, sjenning

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~OWNERS~~ [derekwaynecarr]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

sjenning · 2017-10-19T20:20:51Z

/test extended_conformance_install_update

sjenning · 2017-10-19T22:05:54Z

flake #16870
/test extended_conformance_install_update

openshift-ci-robot · 2017-10-19T23:38:56Z

@sjenning: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/experimental/unit	`299f412`	link	`/test origin-ut`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2017-10-20T14:33:59Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-merge-robot · 2017-10-20T18:03:03Z

Automatic merge from submit-queue (batch tested with PRs 16667, 16796, 16960, 16965, 16894).

sjenning added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 11, 2017

openshift-merge-robot assigned csrwng and enj Oct 11, 2017

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 11, 2017

openshift-ci-robot unassigned csrwng Oct 11, 2017

sjenning force-pushed the docker-gc branch from 299f412 to db7289f Compare October 17, 2017 04:31

openshift-ci-robot unassigned enj Oct 17, 2017

openshift-merge-robot assigned abstractj and mfojtik Oct 17, 2017

sjenning removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 17, 2017

sjenning changed the title ~~[WIP] cmd: ex: standalone docker garbage collector~~ cmd: ex: standalone docker garbage collector Oct 17, 2017

sjenning assigned derekwaynecarr and unassigned abstractj and mfojtik Oct 17, 2017

derekwaynecarr reviewed Oct 18, 2017

View reviewed changes

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 18, 2017

liggitt reviewed Oct 18, 2017

View reviewed changes

cmd: ex: standalone docker garbage collector

cbb2f38

sjenning force-pushed the docker-gc branch from e142ba0 to cbb2f38 Compare October 18, 2017 19:31

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 19, 2017

openshift-merge-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 19, 2017

openshift-merge-robot merged commit 4fdb151 into openshift:master Oct 20, 2017

cmd: ex: standalone docker garbage collector #16796

cmd: ex: standalone docker garbage collector #16796

Uh oh!

Conversation

sjenning commented Oct 11, 2017

Uh oh!

derekwaynecarr commented Oct 11, 2017

Uh oh!

csrwng commented Oct 11, 2017

Uh oh!

sjenning commented Oct 17, 2017

Uh oh!

enj commented Oct 17, 2017

Uh oh!

sjenning commented Oct 17, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr commented Oct 18, 2017

Uh oh!

derekwaynecarr commented Oct 18, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr Oct 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sjenning commented Oct 18, 2017

Uh oh!

derekwaynecarr commented Oct 19, 2017

Uh oh!

derekwaynecarr commented Oct 19, 2017

Uh oh!

openshift-merge-robot commented Oct 19, 2017

Uh oh!

sjenning commented Oct 19, 2017

Uh oh!

sjenning commented Oct 19, 2017

Uh oh!

openshift-ci-robot commented Oct 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Oct 20, 2017

Uh oh!

openshift-merge-robot commented Oct 20, 2017

Uh oh!

Uh oh!

derekwaynecarr Oct 18, 2017 •

edited

Loading

openshift-ci-robot commented Oct 19, 2017 •

edited

Loading