Extended test for registry garbage collector #14509

miminar · 2017-06-07T15:48:51Z

Waiting for #14585
Resolves #15452

miminar · 2017-06-07T15:49:21Z

dmage · 2017-06-08T07:10:15Z

test/extended/images/hardprune.go

+		cleanUp.addImage(baseImg4, imageId, "")
+		//baseImg4Spec := fmt.Sprintf("%s/%s/A:new", registryURL, oc.Namespace())
+
+		childImg1, imageId, err := BuildAndPushChildImage(oc, dClient, baseImg1Spec, "A", "latest", 1, outSink, true)


s/"A"/"C"/? In the next table childImg1 is mentioned in the image stream C.

dmage · 2017-06-08T07:13:41Z

test/extended/images/hardprune.go

+
+})
+
+func expectDeletions(deletedPaths []string, namespace string, expect expectedDeletions) error {


How are you going to get these paths?

I'd like to read them from the standard output of dockerregistry -prune command. Will something like this be available? Or will it be more like:

Deleted blob sha256:XXXX Deleted layer link sha256:XXXX from repository NAMESPACE/REPO

?

You can see it there: master...dmage:prune

dmage · 2017-06-08T07:27:30Z

test/extended/images/helper.go

 	}

-	return "", nil
+	return "", "", nil


No result and no error? Looks like there is something wrong.

Maybe it could be better structured. Nevertheless, the returned values are not wrong. This line is reached when a push failure is expected and the push really failed - so there's no usable output for the caller. Well, maybe imageID is of some use unless removeAfterPush is true.

miminar · 2017-06-16T17:27:08Z

test/extended/images/helper.go

+		out, err := oc.SetNamespace(metav1.NamespaceDefault).AsAdmin().
+			Run("exec").Args("--stdin", podName, "--", "/bin/sh", "-s").
+			InputString(fmt.Sprintf(registryGCLauncherScript, dockerRegistryBinary)).Output()
+		if exitError, ok := err.(*exutil.ExitError); ok && strings.Contains(exitError.StdErr, "unable to upgrade connection") {


@stevekuznetsov have you seen this before?

Jun 16 12:27:17.529: INFO: Running 'oc exec --config=/tmp/openshift/core/openshift.local.config/master/admin.kubeconfig --namespace=default --stdin docker-registry-1-z3hj1 -- /bin/sh -s' Jun 16 12:27:17.771: INFO: Error running &{/data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/oc [oc exec --config=/tmp/openshift/core/openshift.local.config/master/admin.kubeconfig --namespace=default --stdin docker-registry-1-z3hj1 -- /bin/sh -s] [] error: unable to upgrade connection: container not found ("registry") error: unable to upgrade connection: container not found ("registry") [] <nil> 0xc420d582d0 exit status 1 <nil> <nil> true [0xc420030000 0xc420030040 0xc420030040] [0xc420030000 0xc420030040] [0xc420030020 0xc420030038] [0xde31c0 0xde32c0] 0xc421682240 <nil>}: error: unable to upgrade connection: container not found ("registry") failed to execute into registry pod docker-registry-1-z3hj1: exit status 1 Jun 16 12:27:17.822: INFO: Running 'oc exec --config=/tmp/openshift/core/openshift.local.config/master/admin.kubeconfig --namespace=default --stdin docker-registry-1-z3hj1 -- /bin/sh -s' Jun 16 12:27:18.065: INFO: Error running &{/data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/oc [oc exec --config=/tmp/openshift/core/openshift.local.config/master/admin.kubeconfig --namespace=default --stdin docker-registry-1-z3hj1 -- /bin/sh -s] [] error: unable to upgrade connection: container not found ("registry") error: unable to upgrade connection: container not found ("registry") [] <nil> 0xc4212150b0 exit status 1 <nil> <nil> true [0xc4202b76d8 0xc4202b7780 0xc4202b7780] [0xc4202b76d8 0xc4202b7780] [0xc4202b76f8 0xc4202b7760] [0xde31c0 0xde32c0] 0xc42168ef00 <nil>}: error: unable to upgrade connection: container not found ("registry")

Do I use the exec wrong? The interesting part is that this fails roughly in one out of three runs - the first time this code is executed.
If this succeeds once, it will succeed next time as well.

Looks like there isn't a container registry in the docker-registry-1-z3hj1 pod? I haven't seen this before but it may mean the registry failed to start.

@stevekuznetsov my bad. The command attempts to execute into an old pod just after a redeployment. I just need to make sure to choose the most recent one.

miminar · 2017-06-16T17:28:27Z

test/extended/images/hardprune.go

+		// this shouldn't delete anything
+		o.Expect(deleted.Len()).To(o.Equal(0))
+
+		/* TODO: use a persistent storage for the registry to preserve data across re-deployments


@stevekuznetsov Would it be possible to use persistent volume for the registry for the extended tests at least?

What's the context here? PV for test smells bad -- are there cross-test dependencies?

During a single test, I'd like to:

upload some images

redeploy the registry into a different configuration

check whether the registry is usable with the images uploaded before

In this case the modified configuration is read-only, therefore I cannot upload any other images to test with.

Sounds reasonable. You'll want to figure out how to get claimable PVs on the CI environment - what OpenShift Ansible options need to be set? Do we need to set up the filesystem in a special way? We've got logic to set up a VG for EmptyDir mounts already on XFS so we've got a place in origin-ci-tool to add more set up in case we need to. OpenShift Ansible inventory changes would land in aos-cd-jobs.

Thanks for the pointers I'll look into it.

miminar · 2017-06-26T18:26:21Z

@dmage this is ready for a cherry-pick on top of #14585 if you want.

miminar · 2017-06-27T15:00:58Z

Hopefully resolved possible flake caused by #13428.

miminar · 2017-06-28T14:39:48Z

@stevekuznetsov is something wrong with selinux policy:

+ make test-extended SUITE=core 'FOCUS=registry|imageapi|ImagePrune|ImageQuota'
test/extended/core.sh --ginkgo.focus="registry|imageapi|ImagePrune|ImageQuota"
[WARNING] No compiled `ginkgo` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh vendor/github.com/onsi/ginkgo/ginkgo
++ Building go targets for linux/amd64: vendor/github.com/onsi/ginkgo/ginkgo
/data/src/github.com/openshift/origin/hack/build-go.sh took 71 seconds
[WARNING] No compiled `junitmerge` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/junitmerge
++ Building go targets for linux/amd64: tools/junitmerge
/data/src/github.com/openshift/origin/hack/build-go.sh took 3 seconds
[INFO] [CLEANUP] Cleaning up temporary directories
[INFO] Starting server
chcon: invalid context: /var/lib/openshift/openshift.local.volumes	system_u:object_r:openshift_var_lib_t:s0
[ERROR] PID 6461: test/extended/setup.sh:88: `${sudo} chcon "${label}" ${VOLUME_DIR}` exited with status 1.
[INFO] 		Stack Trace: 
[INFO] 		  1: test/extended/setup.sh:88: `${sudo} chcon "${label}" ${VOLUME_DIR}`
[INFO] 		  2: test/extended/core.sh:8: os::test::extended::setup
[INFO]   Exiting with code 1.
/data/src/github.com/openshift/origin/hack/lib/log/system.sh: line 31:  9141 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to _output/scripts/core/artifacts/events.txt
error: A server URL must be specified
[INFO] [CLEANUP] Dumping etcd contents to _output/scripts/core/artifacts/etcd

?

stevekuznetsov · 2017-06-28T14:56:06Z

@miminar #14937

eparis · 2017-06-28T15:03:33Z

[test]

Signed-off-by: Oleg Bulatov <[email protected]>

Signed-off-by: Michal Minář <[email protected]>

openshift-bot · 2017-07-03T10:15:16Z

Evaluated for origin testextended up to b1c1518

openshift-bot · 2017-07-03T11:15:25Z

continuous-integration/openshift-jenkins/testextended SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended/809/) (Base Commit: fc34104) (PR Branch Commit: b1c1518) (Extended Tests: core(registry|imageapi|ImagePrune|ImageQuota))

miminar · 2017-07-03T12:40:51Z

Flake #15015 re-[test]

miminar · 2017-07-03T15:25:46Z

Flake #12072. re-[test]

openshift-bot · 2017-07-03T15:31:21Z

Evaluated for origin test up to b1c1518

openshift-bot · 2017-07-03T19:24:59Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/2937/) (Base Commit: 9c5e7da) (PR Branch Commit: b1c1518)

openshift-merge-robot · 2017-07-25T19:01:12Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: miminar
We suggest the following additional approver: mfojtik

Assign the PR to them by writing /assign @mfojtik in a comment when ready.

Associated issue: 14585

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

openshift-merge-robot · 2017-07-28T22:08:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: miminar
We suggest the following additional approver: mfojtik

Assign the PR to them by writing /assign @mfojtik in a comment when ready.

Associated issue: 14585

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

openshift-merge-robot · 2017-07-28T22:36:14Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: miminar
We suggest the following additional approver: mfojtik

Assign the PR to them by writing /assign @mfojtik in a comment when ready.

Associated issue: 14585

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

openshift-merge-robot · 2017-08-01T00:05:14Z

@miminar PR needs rebase

miminar · 2017-08-01T15:08:29Z

Closing in favor of #14585 which now embeds the extended test.

miminar force-pushed the hard-prune-extended-test branch from b0518d4 to b03fec2 Compare June 7, 2017 15:51

dmage reviewed Jun 8, 2017

View reviewed changes

miminar force-pushed the hard-prune-extended-test branch 3 times, most recently from d84e614 to f8563b6 Compare June 16, 2017 15:33

miminar commented Jun 16, 2017

View reviewed changes

miminar force-pushed the hard-prune-extended-test branch 4 times, most recently from 2db7dd1 to b0a12a4 Compare June 20, 2017 18:24

miminar changed the title ~~[WIP] Extended test for registry garbage collector~~ [DO_NOT_MERGE] Extended test for registry garbage collector Jun 20, 2017

miminar force-pushed the hard-prune-extended-test branch 2 times, most recently from 72d2ac8 to 3ef2fc1 Compare June 26, 2017 15:39

miminar changed the title ~~[DO_NOT_MERGE] Extended test for registry garbage collector~~ Extended test for registry garbage collector Jun 26, 2017

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 27, 2017

miminar force-pushed the hard-prune-extended-test branch from 3ef2fc1 to 749bd88 Compare June 27, 2017 08:26

miminar mentioned this pull request Jun 27, 2017

Persistent storage for docker-registry extended tests #14900

Closed

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 27, 2017

miminar force-pushed the hard-prune-extended-test branch 2 times, most recently from 4b475f4 to 7d916b7 Compare June 27, 2017 15:00

miminar mentioned this pull request Jun 27, 2017

registry: allow to override the DOCKER_REGISTRY_URL and default to in… #14882

Merged

miminar force-pushed the hard-prune-extended-test branch 2 times, most recently from 783a16e to 6808038 Compare June 28, 2017 10:54

miminar force-pushed the hard-prune-extended-test branch 2 times, most recently from 8a50257 to be2cc4d Compare June 29, 2017 13:24

Oleg Bulatov and others added 2 commits July 3, 2017 12:10

Add -prune option to dockerregistry

b6edc91

Signed-off-by: Oleg Bulatov <[email protected]>

Extended test for registry garbage collector

b1c1518

Signed-off-by: Michal Minář <[email protected]>

miminar force-pushed the hard-prune-extended-test branch from be2cc4d to b1c1518 Compare July 3, 2017 10:10

openshift deleted a comment from openshift-bot Jul 3, 2017

openshift deleted a comment from eparis Jul 4, 2017

openshift-merge-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jul 24, 2017

miminar mentioned this pull request Jul 25, 2017

extended test flake: expected different manifest version for image pruning #15452

Closed

openshift-merge-robot assigned smarterclayton and mfojtik Jul 25, 2017

openshift-merge-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 28, 2017

openshift-merge-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 28, 2017

miminar mentioned this pull request Jul 31, 2017

Add -prune option to dockerregistry #14585

Merged

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 1, 2017

miminar closed this Aug 1, 2017

miminar deleted the hard-prune-extended-test branch October 10, 2017 09:56


		})

		func expectDeletions(deletedPaths []string, namespace string, expect expectedDeletions) error {

Extended test for registry garbage collector #14509

Extended test for registry garbage collector #14509

Uh oh!

Conversation

miminar commented Jun 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

miminar commented Jun 7, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miminar Jun 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miminar commented Jun 26, 2017

Uh oh!

miminar commented Jun 27, 2017

Uh oh!

miminar commented Jun 28, 2017

Uh oh!

stevekuznetsov commented Jun 28, 2017

Uh oh!

eparis commented Jun 28, 2017

Uh oh!

openshift-bot commented Jul 3, 2017

Uh oh!

openshift-bot commented Jul 3, 2017

Uh oh!

miminar commented Jul 3, 2017

Uh oh!

miminar commented Jul 3, 2017

Uh oh!

openshift-bot commented Jul 3, 2017

Uh oh!

openshift-bot commented Jul 3, 2017

Uh oh!

openshift-merge-robot commented Jul 25, 2017

Uh oh!

openshift-merge-robot commented Jul 28, 2017

Uh oh!

openshift-merge-robot commented Jul 28, 2017

Uh oh!

openshift-merge-robot commented Aug 1, 2017

Uh oh!

miminar commented Aug 1, 2017

Uh oh!

Uh oh!

miminar commented Jun 7, 2017 •

edited

Loading

miminar Jun 8, 2017 •

edited

Loading