baremetal: discuss options for managing baremetal-operator

The Bare Metal Operator is currently launched by the Machine API
Operator when the cluster is running on the baremetal infrastructure
platform.

Discuss options for evolving this design so that the Machine API
Operator no longer needs to have quite so much bare metal specific
knowledge.

Author: markmc

enhancements/baremetal/baremetal-operator-management.md

@@ -0,0 +1,166 @@
+---
+title: baremetal-operator-management
+authors:
+  - "@markmc"
+reviewers:
+  - "@abhinavdahiya"
+  - "@dhellmann"
+  - "@enxebre"
+  - "@eparis"
+  - "@hardys"
+  - "@sadasu"
+  - "@smarterclayton"
+  - "@stbenjam"
+approvers:
+  - TBD
+creation-date: 2020-02-13
+last-updated: 2020-02-13
+status: provisional
+see-also:
+  - https://github.com/openshift/enhancements/pull/200
+replaces:
+superseded-by:
+---
+
+# Managing the Bare Metal Operator
+
+## Release Signoff Checklist
+
+- [ ] Enhancement is `implementable`
+- [ ] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)
+
+## Open Questions
+
+1. Should the Bare Metal Operator be managed by the Cluster Version
+   Operator or the Machine API Operator?
+2. How to handle upgrades
+3. Which release this change should target
+
+## Summary
+
+The Bare Metal Operator provides bare metal machine management
+capabilities needed for the Machine API provider on the `baremetal`
+platform, and there is no equivalent for this component on other
+platforms.
+
+The Bare Metal Operator is currently managed by the Machine API
+Operator in a fashion that requires the Machine API Operator to have
+significant bare metal specific knowledge. This can be resolved either
+by:
+
+1. The Machine API Operator can add a generic operator management
+   framework (similar to some Cluster Version Operator capabilities)
+   for platform specific operators, and the Bare Metal Operator would
+   integrate with this framework.
+2. The Cluster Version Operator can gain the ability to manage
+   operators that are platform-specific, and the Bare Metal Operator
+   would move to being managed by the Cluster Version Operator.
+
+## Motivation
+
+In order to bring up a cluster using the baremetal platform:
+
+1. The installer needs to capture bare metal host information and
+   provisioning configuration for later use by baremetal-operator
+2. Something needs to install the CRDs for these bare metal specific
+   resources created by the installer
+3. Something needs to launch baremetal-operator
+
+Currently (1) is achieved by the installer creating:
+
+* A Provisioning resource manifest
+* Manifests for BareMetalHost resources and their associated secrets
+
+and these manifests are applied by the cluster-bootstrap component
+towards the end of the cluster bootstrapping process.
+
+This resource creation step does not succeed until the step (2)
+completes - i.e. the relevant CRDs are applied - and this is currently
+done by the CVO as it applies the manifests for the MAO.
+
+Finally, (3) happens when the MAO detects that it is running on the
+baremetal infrastructure platform and instantiates the BMO deployment.
+
+There are two problems emerging with this design:
+
+* A sense that launching the BMO is outside of the scope of the MAO,
+  particularly since it does not manage operators on any other
+  platform
+
+* Expanding needs for bare metal specific manifests - for example, new
+  CRDs used to drive new BMO capabilities - to be installed early in
+  the cluster bring-up means introducing yet more bare metal specific
+  concerns into the MAO
+
+Steps (2) and (3) are aspects of cluster bring-up which the CVO is
+clearly well-suited. However, the CVO has no concept of managing
+operators that are specific to an infrastructure platform. Therefore,
+to use the CVO capabilities in this case would mean installing base
+metal CRDs and launching the bare metal operator even on platforms
+where it is not needed.
+
+In contrast, the way (2) and (3) is currently achieved by the MAO is
+not at all generic and results in the MAO having substantial built-in
+knowledge of the bare metal platform. A more generic mechanism where
+it could discover and apply any required manifests from BMO would
+mean the addition of operator management capabilities that look very
+much like the CVO's capabilities.
+
+### Goals
+
+Allow bare metal machine management capabilities to be fully
+encapsulated in the Bare Metal Operator.
+
+Ensure that the Bare Metal Operator is only deployed on the
+`baremetal` platform.
+
+### Non-Goals
+
+### Proposal
+
+A choice should be made between the two options listed above:
+
+1. The Machine API Operator adds a generic framework for
+   platform-specific operators - e.g. a platform should be able to
+   specify an operator image, and the Machine API Operator would
+   extract and apply the necessary manifests from that operator image.
+2. The Cluster Version Operator adds the ability to have Cluster
+   Operators that are specific to certain platforms - e.g. an operator
+   image would be marked as conditional on a platform name, and the
+   Cluster Version Operator would ignore it unless it detected that the
+   cluster was configured with that platform.
+
+### Risks and Mitigations
+
+
+## Design Details
+
+### Test Plan
+
+### Graduation Criteria
+
+
+### Upgrade / Downgrade Strategy
+
+
+### Version Skew Strategy
+
+
+## Implementation History
+
+
+## Drawbacks
+
+
+## Alternatives
+
+## References
+
+- "/enhancements/baremetal/baremetal-provisioning-config.md"
+- https://github.com/openshift/machine-api-operator/pull/302
+- https://github.com/metal3-io/baremetal-operator/issues/227
+- https://github.com/openshift/enhancements/pull/90
+- https://github.com/openshift/enhancements/pull/102