Enable federation for origin clusters #14239

marun · 2017-05-18T08:08:12Z

TODO

build openshift-specific server image
build kubefed that targets the openshift-specific server image
update kubefed to report openshift version (e.g. 3.6) instead of kube version (e.g. 1.6)
update kubefed init to use rh-specific etcd image
test compatibility of kubefed init's PV annotation with openshift

derekwaynecarr · 2017-05-20T22:04:01Z

@marun -- so far looks fine. i thought the second commit would have a dockerfile?

marun · 2017-05-23T02:18:22Z

@stevekuznetsov I can't seem to get ldflags to work, the federation image names end up blank. Help?

stevekuznetsov · 2017-05-23T13:56:07Z

hack/common.sh

@@ -669,6 +669,8 @@ readonly -f os::build::save_version_vars
 function os::build::get_product_vars() {
  export OS_BUILD_LDFLAGS_IMAGE_PREFIX="${OS_IMAGE_PREFIX:-"openshift/origin"}"
  export OS_BUILD_LDFLAGS_DEFAULT_IMAGE_STREAMS="${OS_BUILD_LDFLAGS_DEFAULT_IMAGE_STREAMS:-"centos7"}"
+  export OS_BUILD_LDFLAGS_FEDERATION_SERVER_IMAGE_NAME="${OS_BUILD_LDFLAGS_FEDERATION_SERVER_IMAGE_openshift:-"openshift/origin-federation"}"


This should be "${OS_BUILD_LDFLAGS_IMAGE_PREFIX}-federation"

stevekuznetsov · 2017-05-23T13:57:35Z

hack/common.sh

@@ -716,6 +718,8 @@ function os::build::ldflags() {
  ldflags+=($(os::build::ldflag "${OS_GO_PACKAGE}/vendor/k8s.io/client-go/pkg/version.gitVersion" "${KUBE_GIT_VERSION}"))
  ldflags+=($(os::build::ldflag "${OS_GO_PACKAGE}/vendor/k8s.io/client-go/pkg/version.buildDate" "${buildDate}"))
  ldflags+=($(os::build::ldflag "${OS_GO_PACKAGE}/vendor/k8s.io/client-go/pkg/version.gitTreeState" "clean"))
+  ldflags+=($(os::build::ldflag "${OS_GO_PACKAGE}/cmd/kubefed.serverImageName" "${OS_BUILD_LDFLAGS_FEDERATION_SERVER_IMAGE_NAME}"))


This looks like it should work -- maybe it's weird since it's also the entrypoint? @smarterclayton thoughts?

I've switched to target the pkg instead of the entrypoint. Not that I can test it - the new release build mechanism has hosed what was formerly a working dev setup.

stevekuznetsov · 2017-05-23T13:58:37Z

images/federation/Dockerfile

+#
+# The standard name for this image is openshift/origin-federation
+#
+FROM openshift/origin-base


If you expect the hyperkube binary to be shipped to customers it will be in the RPM so install using yum a la origin-pod.

@smarterclayton I'm not sure there's a point in packaging hyperkube, given that this is unlikely to be the method for shipping the federation servers in the future. What's the simplest acceptable solution to getting this image built for 3.6?

What's the simplest acceptable solution to getting this image built for 3.6?

To be clear... my comment is prefixed with that "IF" for a reason. If we are shipping this, we need to RPM install it. If we are not, just keep doing what you're doing.

Is hyperkube a symlink?

No, it is not a symlink.

marun · 2017-05-24T06:32:59Z

@stevekuznetsov Where are the docs for the new build mechanisms? In addition to requiring the manual installation of a bunch of dependencies via yum and imagebuilder via go get, there is apparently a dependency on being able to an authorized pull of the openshift/source image?

openshift/origin-pod: /home/dev/src/os/bin/imagebuilder
openshift/origin-pod: --> Image openshift/origin-source was not found, pulling ...
openshift/origin-pod: unable to pull image (from: openshift/origin-source, tag: latest): unauthorized: authentication required

detiber · 2017-05-24T12:30:22Z

@marun I needed to run hack/build-base-images.sh before I could run make release (with the same error you are seeing).

stevekuznetsov · 2017-05-24T13:29:54Z

@marun you should be able to divorce yourself from any dependencies with our release container:

OS_BUILD_ENV_PRESERVE="_output/local" hack/env OS_ONLY_BUILD_PLATFORMS="linux/amd64" make release

I'll open a PR to see if we can make at least some of those env vars default so you don't have to set them.

marun · 2017-05-24T14:38:34Z

[test]

openshift-bot · 2017-05-24T18:06:48Z

Evaluated for origin test up to cac7f93

openshift-bot · 2017-05-24T19:45:49Z

continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/1708/) (Base Commit: 483738e)

stevekuznetsov

Changes LGTM

derekwaynecarr · 2017-05-24T20:51:48Z

LGTM

derekwaynecarr · 2017-05-24T20:52:05Z

[merge][severity: bug]

eparis: I'm really sorry if this does break the actual build...

openshift-bot · 2017-05-24T21:05:57Z

Evaluated for origin merge up to cac7f93

detiber · 2017-05-25T02:28:56Z

@marun seeing the following error when trying to join clusters to the federation:

I0525 02:24:42.705278       1 clustercontroller.go:122] It's a new cluster, a cluster client will be created
W0525 02:24:42.709866       1 cluster_util.go:121] error in fetching secret: User "system:serviceaccount:federation-system:federation-controller-manager" cannot get secrets in project "federation-system"
...
W0525 02:25:42.716880       1 cluster_util.go:121] error in fetching secret: User "system:serviceaccount:federation-system:federation-controller-manager" cannot get secrets in project "federation-system"
E0525 02:25:42.716908       1 clustercontroller.go:125] Failed to create cluster client, err: timed out waiting for secret: timed out waiting for the condition
I0525 02:25:42.716914       1 clustercontroller.go:163] Failed to Get the status of cluster: jdetiber-fed-us-west1-a

detiber · 2017-05-25T04:04:52Z

@marun I was able to resolve by running:

oc adm policy add-role-to-user admin system:serviceaccount:federation-system:federation-controller-manager -n federation-system

I suspect there are more limited permissions rather than admin that I could have added, but this unblocked my testing.

openshift-bot · 2017-05-26T04:21:53Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/801/) (Base Commit: 1d30fc4) (Extended Tests: bug) (Image: devenv-rhel7_6273)

derekwaynecarr · 2017-05-26T14:34:30Z

woot!

smarterclayton · 2017-05-26T15:17:07Z

well done Maru

…

On Fri, May 26, 2017 at 10:34 AM, Derek Carr ***@***.***> wrote: woot! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14239 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p-LOXPZb461mc9S3xgdfEsYzSqCSks5r9uL5gaJpZM4Ne3vA> .

smarterclayton · 2017-06-08T04:50:19Z

Follow up item: the federation image is 800M of unshared layers with the rest of OpenShift (i.e. if you download origin onto a node, to get federation you have to get 800M of additional content). That sucks and we need to fix it in a follow up.

marun requested a review from derekwaynecarr May 18, 2017 08:08

derekwaynecarr self-assigned this May 20, 2017

derekwaynecarr mentioned this pull request May 22, 2017

UPSTREAM: 46020: Enable customization of federation image #14238

Closed

marun force-pushed the enable-federation branch from 57c1f35 to 90e993a Compare May 23, 2017 02:16

UPSTREAM: 46020: Enable customization of federation image

4775b28

marun force-pushed the enable-federation branch from 90e993a to 55bf2a9 Compare May 23, 2017 02:22

stevekuznetsov suggested changes May 23, 2017

View reviewed changes

marun added 2 commits May 23, 2017 22:57

UPSTREAM: 46247: Enable customization of federation etcd image

a457d5f

UPSTREAM: 46323: Use beta annotation for fed etcd pvc storage class

2bb8fc4

marun force-pushed the enable-federation branch from 55bf2a9 to a0c834a Compare May 24, 2017 06:08

Enable preliminary support for origin federation

cac7f93

marun force-pushed the enable-federation branch from a0c834a to cac7f93 Compare May 24, 2017 18:00

stevekuznetsov approved these changes May 24, 2017

View reviewed changes

marun mentioned this pull request May 24, 2017

DO NOT MERGE: Add script to smoke test federation #14336

Closed

openshift-bot merged commit 827f454 into openshift:master May 26, 2017

Enable federation for origin clusters #14239

Enable federation for origin clusters #14239

Uh oh!

Conversation

marun commented May 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

derekwaynecarr commented May 20, 2017

Uh oh!

marun commented May 23, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marun commented May 24, 2017

Uh oh!

detiber commented May 24, 2017

Uh oh!

stevekuznetsov commented May 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marun commented May 24, 2017

Uh oh!

openshift-bot commented May 24, 2017

Uh oh!

openshift-bot commented May 24, 2017

Uh oh!

stevekuznetsov left a comment

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr commented May 24, 2017

Uh oh!

derekwaynecarr commented May 24, 2017 • edited by eparis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented May 24, 2017

Uh oh!

detiber commented May 25, 2017

Uh oh!

detiber commented May 25, 2017

Uh oh!

openshift-bot commented May 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

derekwaynecarr commented May 26, 2017

Uh oh!

smarterclayton commented May 26, 2017 via email

Uh oh!

smarterclayton commented Jun 8, 2017

Uh oh!

Uh oh!

marun commented May 18, 2017 •

edited

Loading

stevekuznetsov commented May 24, 2017 •

edited

Loading

derekwaynecarr commented May 24, 2017 •

edited by eparis

Loading

openshift-bot commented May 26, 2017 •

edited

Loading