-
Notifications
You must be signed in to change notification settings - Fork 4.7k
ParseDockerImageReference use docker/distribution reference parser #11245
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ParseDockerImageReference use docker/distribution reference parser #11245
Conversation
[test] |
This parser have a few issues:
|
ref.Registry, ref.Name = reference.SplitHostname(named) | ||
} else if named, ok := dockerRef.(reference.Tagged); ok { | ||
ref.Tag = named.Tag() | ||
} else if named, ok := dockerRef.(reference.Digested); ok { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe Tagged
and Digested
can be considered invalid.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, but in this case type switch no longer needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ignore my previous comment. They are valid references.
|
||
repoParts := strings.Split(ref.Name, "/") | ||
|
||
if !isRegistryName(ref.Registry) && len(repoParts) == 1 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A note for isRegistryName()
: use strings.ContainsAny
. Also document that it needs to be called with just one path component (no slash allowed).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You forget about localhost
. So I think this function useful here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is. This is a suggestion to modify the isRegistryName()
function itself and annotate it with a godoc.
ref.Registry = "" | ||
} | ||
|
||
if ref.Namespace == "" && len(repoParts) >= 2 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: len(ref.Namespace) == 0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make sure repoParts[0]
doesn't look like a registry?
ref.Namespace, ref.Name = repoParts[0], strings.Join(repoParts[1:], "/") | ||
} | ||
|
||
if ref.Namespace == "" && IsRegistryDockerHub(ref.Registry) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: len(ref.Namespace) == 0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
Tag: "tag", | ||
}, | ||
{ | ||
// docker.io/namespace/name == 255 chars without implicit namespace |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/implicit/explicit/
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
@@ -196,7 +196,9 @@ func ValidateImageStreamTagReference(tagRef api.TagReference, fldPath *field.Pat | |||
} | |||
switch tagRef.From.Kind { | |||
case "DockerImage": | |||
if ref, err := api.ParseDockerImageReference(tagRef.From.Name); err == nil && tagRef.ImportPolicy.Scheduled && len(ref.ID) > 0 { | |||
if ref, err := api.ParseDockerImageReference(tagRef.From.Name); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
... && len(tagRef.From.Name) > 0
, which is already covered above.
I'd rather do it right than stick with the old behavior. Another example is parsing of |
@@ -297,6 +298,10 @@ func TestParseDockerImageReference(t *testing.T) { | |||
Err: true, | |||
}, | |||
{ | |||
From: "abc@badid", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you add From: "@sha256:3c87c572822935df60f0f5d3665bd376841a7fcfeb806b5f212de6a00e9a7b25"
, From: "sha256:3c87c572822935df60f0f5d3665bd376841a7fcfeb806b5f212de6a00e9a7b25"
and From: ":tag"
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
@miminar This behavior right according |
package main
import ("fmt"; "github.com/docker/docker/reference")
func main() {
ref, err := reference.ParseNamed("bar/foo/baz")
if err != nil {
fmt.Println("failed to parse reference: %v", err)
}
fmt.Printf("hostname=%q, remotename=%q\n", ref.Hostname(), ref.RemoteName())
}
Which makes more sense to me. Although if you replace |
@miminar Using
I'm not sure we want it. |
@miminar i don't like vendoring 15 more deps just to get this helper, not saying that we can't even use the helper as it is as we need to get the namespace still.. |
a1b8c4b
to
e3afe0b
Compare
I took the function to separate the host name and repository from the docker.
|
@miminar I think this is unacceptable as well. we will break everyone. |
By everyone you mean us? 😄 This PR attempts to get the parsing right. Different output for the same input is expected if the previous parsing was wrong. We just need to make sure that everything works as expected. I wouldn't default to |
OK :)
If we do not want to use |
e3afe0b
to
2ea0593
Compare
repoParts := strings.Split(ref.Name, "/") | ||
|
||
if len(repoParts) >= 2 { | ||
if isRegistryName(repoParts[0]) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this even possible to satisfy? It is with localhost
in name (e.g. registry.com/app/localhost/image
). But it looks to me like a valid reference.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
0613aae
to
d9b7feb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Just a note: the pkg/image/api/utils
modules is intended to be imported in openshift/source-to-image
to use the same parser and avoid duplicating the same hacks in different way.
@ncdc would you take a look as well?
break | ||
} | ||
return ref, fmt.Errorf("the docker pull spec %q must be two or three segments separated by slashes", spec) | ||
// It's not enough just to use the reference.ParseNamed(). We have the fill |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: s/We have the/We have to/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, typo. I'll fix it.
@@ -0,0 +1,53 @@ | |||
package utils |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In kube we try to avoid generic "utils" packages. Not a deal breaker, but I'd recommend a more specific package name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
github.com/openshift/origin/pkg/image/reference
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WFM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
@@ -488,7 +488,7 @@ func TestValidateImageStream(t *testing.T) { | |||
}, | |||
}, | |||
expected: field.ErrorList{ | |||
field.Invalid(field.NewPath("spec", "tags").Key("badid").Child("from", "name"), "abc@badid", "only tags can be scheduled for import"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like we lost a test case where we validate that you can't specify a digest for import where the error returned is "only tags can be scheduled for import"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"abc@badid" was always wrong, but previously the error was ignored.
what do you mean "we lost" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no test case that validates it receives "only tags can be scheduled for import" as an error any more. Could you please add a test case that tries to schedule a valid pull-by-digest spec?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ncdc Fixed
return ref, err | ||
} | ||
|
||
if named, ok := dockerRef.(reference.Named); ok { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't reference.ParseNamed
return a reference.Named
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
Tag: "tag", | ||
}, | ||
{ | ||
// docker.io/namespace/name == 255 chars without explicit namespace |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/without/with
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
78c9a44
to
148e83a
Compare
bf5259e
to
8b073e8
Compare
@ncdc please review again. |
8b073e8
to
a9c2463
Compare
It sure if I'll have time to review today. Feel free to find another On Tuesday, October 11, 2016, Alexey Gladkov [email protected]
|
@ncdc Then please take off lock on this PR. |
@liggitt @smarterclayton please review |
a9c2463
to
e9843b1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did another pass
@@ -346,6 +347,8 @@ func validImageStreamImage(imageNode *imagegraph.ImageStreamImageNode, imageStre | |||
} | |||
} | |||
return false, dockerImageReference.ID | |||
} else { | |||
fmt.Fprintf(os.Stderr, "ERR(%s): %#+v\n", imageNode.Name, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this for debugging purposes? If not, maybe something a bit gentler than "ERR" would be better?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ncdc I forgot to remove debugging. Fixed.
) | ||
|
||
// NamedDockerImageReference points to a Docker image. | ||
type NamedDockerImageReference struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This type is identical in everything except name to DockerImageReference
in pkg/image/api/types.go
. And ParseNamedDockerImageReference
below is effectively wrapped by pkg/image/api/helpers.go#ParseDockerImageReference
. Can we collapse the two types and functions into just 1 type and 1 function?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I split it to able use this parser in S2I as mentioned in #11245 (comment). S2I can't use ParseDockerImageReference
directly because origin
imports it. To avoid differences in the parser behavior we need to use one parser everywhere. So I move it into separate module with minimal dependencies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok
e9843b1
to
0cf14aa
Compare
flake: #8571 |
0cf14aa
to
2c9ba48
Compare
We can't remove ParseDockerImageReference and just use docker/distribution/reference to parse DockerImageReference because we have to support few special cases related to docker.io. Keeping it in mind, we can reimplement ParseDockerImageReference to use new parser. In addition we got new limits on the length of the DockerImageReference. So, after refactoring our restrictions are pretty the same as in docker/distribution. Signed-off-by: Gladkov Alexey <[email protected]>
2c9ba48
to
caa9083
Compare
Evaluated for origin test up to caa9083 |
continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/10015/) (Base Commit: 6b0457c) |
@mfojtik merge ? |
[merge] Michal Fojtik On 13 October 2016 at 19:13:19, Alexey Gladkov ([email protected])
|
continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/10015/) (Image: devenv-rhel7_5179) |
Evaluated for origin merge up to caa9083 |
We can't remove ParseDockerImageReference and just use docker/distribution/reference
to parse DockerImageReference because we have to support few special cases related
to docker.io. Keeping it in mind, we can reimplement ParseDockerImageReference
to use new parser.
In addition we got new limits on the length of the DockerImageReference.
So, after refactoring our restrictions are pretty the same as in docker/distribution.
Fix #11211
@ncdc @miminar please review.