Skip to content

Commit ba4754d

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #3055 from jakobmoellerdev/submodule-layouts
📖 docs updates for external types and submodule-layouts
2 parents 1105363 + f208114 commit ba4754d

File tree

5 files changed

+532
-151
lines changed

5 files changed

+532
-151
lines changed

‎docs/book/src/SUMMARY.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,10 @@
106106
- [completion](./reference/completion.md)
107107
- [Artifacts](./reference/artifacts.md)
108108
- [Platform Support](./reference/platform.md)
109+
110+
- [Sub-Module Layouts](./reference/submodule-layouts.md)
111+
- [Using an external Type / API](./reference/using_an_external_type.md)
112+
109113
- [Configuring EnvTest](./reference/envtest.md)
110114

111115
- [Metrics](./reference/metrics.md)

‎docs/book/src/reference/reference.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,10 +31,12 @@
3131
- [completion](completion.md)
3232
- [Artifacts](artifacts.md)
3333
- [Platform Support](platform.md)
34-
- [Writing controller tests](writing-tests.md)
35-
- [Metrics](metrics.md)
3634

35+
- [Sub-Module Layouts](submodule-layouts.md)
36+
- [Using an external Type / API](using_an_external_type.md)
37+
38+
- [Metrics](metrics.md)
3739
- [Reference](metrics-reference.md)
3840

3941
- [Makefile Helpers](makefile-helpers.md)
40-
- [CLI plugins](../plugins/cli-plugins.md)
42+
- [CLI plugins](../plugins/plugins.md)

‎docs/book/src/reference/submodule-layouts.md

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# Sub-Module Layouts
2+
3+
This part describes how to modify a scaffolded project for use with multiple `go.mod` files for APIs and Controllers.
4+
5+
Sub-Module Layouts (in a way you could call them a special form of [Monorepo's][monorepo]) are a special use case and can help in scenarios that involve reuse of APIs without introducing indirect dependencies that should not be available in the project consuming the API externally.
6+
7+
<aside class="note">
8+
<h1>Using external Types</h1>
9+
10+
If you are looking to do operations and reconcile via a controller a Type(CRD) which are owned by another project then, please see [Using an external Type](/reference/using_an_external_type.md) for more info.
11+
12+
</aside>
13+
14+
## Overview
15+
16+
Separate `go.mod` modules for APIs and Controllers can help for the following cases:
17+
18+
- There is an enterprise version of an operator available that wants to reuse APIs from the Community Version
19+
- There are many (possibly external) modules depending on the API and you want to have a more strict separation of transitive dependencies
20+
- If you want to reduce impact of transitive dependencies on your API being included in other projects
21+
- If you are looking to separately manage the lifecycle of your API release process from your controller release process.
22+
- If you are looking to modularize your codebase without splitting your code between multiple repositories.
23+
24+
They introduce however multiple caveats into typical projects which is one of the main factors that makes them hard to recommend in a generic use-case or plugin:
25+
26+
- Multiple `go.mod` modules are not recommended as a go best practice and [multiple modules are mostly discouraged][multi-module-repositories]
27+
- There is always the possibility to extract your APIs into a new repository and arguably also have more control over the release process in a project spanning multiple repos relying on the same API types.
28+
- It requires at least one [replace directive][replace-directives] either through `go.work` which is at least 2 more files plus an environment variable for build environments without GO_WORK or through `go.mod` replace, which has to be manually dropped and added for every release.
29+
30+
<aside class="note warning">
31+
<h1>Implications on Maintenance efforts</h1>
32+
33+
When deciding to deviate from the standard kubebuilder `PROJECT` setup or the extended layouts offered by its plugins, it can result in increased maintenance overhead as there can be breaking changes in upstream that could break with the custom module structure described here.
34+
35+
Splitting your codebase to multiple repos and/or multiple modules incurs costs that will grow over time. You'll need to define clear version dependencies between your own modules, do phased upgrades carefully, etc. Especially for small-to-medium projects, one repo and one module is the best way to go.
36+
37+
Bear in mind, that it is not recommended to deviate from the proposed layout unless you know what you are doing.
38+
You may also lose the ability to use some of the CLI features and helpers. For further information on the project layout, see the doc [What's in a basic project?][basic-project-doc]
39+
40+
</aside>
41+
42+
## Adjusting your Project
43+
44+
For a proper Sub-Module layout, we will use the generated APIs as a starting point.
45+
46+
For the steps below, we will assume you created your project in your `GOPATH` with
47+
48+
```shell
49+
kubebuilder init
50+
```
51+
52+
and created an API & controller with
53+
54+
```shell
55+
kubebuilder create api --group operator --version v1alpha1 --kind Sample --resource --controller --make
56+
```
57+
58+
### Creating a second module for your API
59+
60+
Now that we have a base layout in place, we will enable you for multiple modules.
61+
62+
1. Navigate to `api/v1alpha1`
63+
2. Run `go mod init` to create a new submodule
64+
3. Run `go mod tidy` to resolve the dependencies
65+
66+
Your api go.mod file could now look like this:
67+
68+
```go.mod
69+
module YOUR_GO_PATH/test-operator/api/v1alpha1
70+
71+
go 1.21.0
72+
73+
require (
74+
k8s.io/apimachinery v0.28.4
75+
sigs.k8s.io/controller-runtime v0.16.3
76+
)
77+
78+
require (
79+
github.com/go-logr/logr v1.2.4 // indirect
80+
github.com/gogo/protobuf v1.3.2 // indirect
81+
github.com/google/gofuzz v1.2.0 // indirect
82+
github.com/json-iterator/go v1.1.12 // indirect
83+
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
84+
github.com/modern-go/reflect2 v1.0.2 // indirect
85+
golang.org/x/net v0.17.0 // indirect
86+
golang.org/x/text v0.13.0 // indirect
87+
gopkg.in/inf.v0 v0.9.1 // indirect
88+
gopkg.in/yaml.v2 v2.4.0 // indirect
89+
k8s.io/klog/v2 v2.100.1 // indirect
90+
k8s.io/utils v0.0.0-20230406110748-d93618cff8a2 // indirect
91+
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
92+
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
93+
)
94+
```
95+
96+
As you can see it only includes apimachinery and controller-runtime as dependencies and any dependencies you have
97+
declared in your controller are not taken over into the indirect imports.
98+
99+
### Using replace directives for development
100+
101+
When trying to resolve your main module in the root folder of the operator, you will notice an error if you use a VCS path:
102+
103+
```shell
104+
go mod tidy
105+
go: finding module for package YOUR_GO_PATH/test-operator/api/v1alpha1
106+
YOUR_GO_PATH/test-operator imports
107+
YOUR_GO_PATH/test-operator/api/v1alpha1: cannot find module providing package YOUR_GO_PATH/test-operator/api/v1alpha1: module YOUR_GO_PATH/test-operator/api/v1alpha1: git ls-remote -q origin in LOCALVCSPATH: exit status 128:
108+
remote: Repository not found.
109+
fatal: repository 'https://YOUR_GO_PATH/test-operator/' not found
110+
```
111+
112+
The reason for this is that you may have not pushed your modules into the VCS yet and resolving the main module will fail as it can no longer
113+
directly access the API types as a package but only as a module.
114+
115+
To solve this issue, we will have to tell the go tooling to properly `replace` the API module with a local reference to your path.
116+
117+
You can do this with 2 different approaches: go modules and go workspaces.
118+
119+
#### Using go modules
120+
121+
For go modules, you will edit the main `go.mod` file of your project and issue a replace directive.
122+
123+
You can do this by editing the `go.mod` with
124+
``
125+
```shell
126+
go mod edit -require YOUR_GO_PATH/test-operator/api/[email protected] # Only if you didn't already resolve the module
127+
go mod edit -replace YOUR_GO_PATH/test-operator/api/[email protected]=./api/v1alpha1
128+
go mod tidy
129+
```
130+
131+
Note that we used the placeholder version `v0.0.0` of the API Module. In case you already released your API module once,
132+
you can use the real version as well. However this will only work if the API Module is already available in the VCS.
133+
134+
<aside class="note warning">
135+
<h1>Implications on controller releases</h1>
136+
137+
Since the main `go.mod` file now has a replace directive, it is important to drop it again before releasing your controller module.
138+
To achieve this you can simply run
139+
140+
```shell
141+
go mod edit -dropreplace YOUR_GO_PATH/test-operator/api/v1alpha1
142+
go mod tidy
143+
```
144+
145+
</aside>
146+
147+
#### Using go workspaces
148+
149+
For go workspaces, you will not edit the `go.mod` files yourself, but rely on the workspace support in go.
150+
151+
To initialize a workspace for your project, run `go work init` in the project root.
152+
153+
Now let us include both modules in our workspace:
154+
```shell
155+
go work use . # This includes the main module with the controller
156+
go work use api/v1alpha1 # This is the API submodule
157+
go work sync
158+
```
159+
160+
This will lead to commands such as `go run` or `go build` to respect the workspace and make sure that local resolution is used.
161+
162+
You will be able to work with this locally without having to build your module.
163+
164+
When using `go.work` files, it is recommended to not commit them into the repository and add them to `.gitignore`.
165+
166+
```gitignore
167+
go.work
168+
go.work.sum
169+
```
170+
171+
When releasing with a present `go.work` file, make sure to set the environment variable `GOWORK=off` (verifiable with `go env GOWORK`) to make sure the release process does not get impeded by a potentially commited `go.work` file.
172+
173+
#### Adjusting the Dockerfile
174+
175+
When building your controller image, kubebuilder by default is not able to work with multiple modules.
176+
You will have to manually add the new API module into the download of dependencies:
177+
178+
```dockerfile
179+
# Build the manager binary
180+
FROM golang:1.20 as builder
181+
ARG TARGETOS
182+
ARG TARGETARCH
183+
184+
WORKDIR /workspace
185+
# Copy the Go Modules manifests
186+
COPY go.mod go.mod
187+
COPY go.sum go.sum
188+
# Copy the Go Sub-Module manifests
189+
COPY api/v1alpha1/go.mod api/go.mod
190+
COPY api/v1alpha1/go.sum api/go.sum
191+
# cache deps before building and copying source so that we don't need to re-download as much
192+
# and so that source changes don't invalidate our downloaded layer
193+
RUN go mod download
194+
195+
# Copy the go source
196+
COPY cmd/main.go cmd/main.go
197+
COPY api/ api/
198+
COPY internal/controller/ internal/controller/
199+
200+
# Build
201+
# the GOARCH has not a default value to allow the binary be built according to the host where the command
202+
# was called. For example, if we call make docker-build in a local env which has the Apple Silicon M1 SO
203+
# the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore,
204+
# by leaving it empty we can ensure that the container and binary shipped on it will have the same platform.
205+
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go
206+
207+
# Use distroless as minimal base image to package the manager binary
208+
# Refer to https://github.com/GoogleContainerTools/distroless for more details
209+
FROM gcr.io/distroless/static:nonroot
210+
WORKDIR /
211+
COPY --from=builder /workspace/manager .
212+
USER 65532:65532
213+
214+
ENTRYPOINT ["/manager"]
215+
```
216+
217+
### Creating a new API and controller release
218+
219+
Because you adjusted the default layout, before releasing your first version of your operator, make sure to [familiarize yourself with mono-repo/multi-module releases][multi-module-repositories] with multiple `go.mod` files in different subdirectories.
220+
221+
Assuming a single API was created, the release process could look like this:
222+
223+
```sh
224+
git commit
225+
git tag v1.0.0 # this is your main module release
226+
git tag api/v1.0.0 # this is your api release
227+
go mod edit -require YOUR_GO_PATH/test-operator/[email protected] # now we depend on the api module in the main module
228+
go mod edit -dropreplace YOUR_GO_PATH/test-operator/api/v1alpha1 # this will drop the replace directive for local development in case you use go modules, meaning the sources from the VCS will be used instead of the ones in your monorepo checked out locally.
229+
git push origin main v1.0.0 api/v1.0.0
230+
```
231+
232+
After this, your modules will be available in VCS and you do not need a local replacement anymore. However if youre making local changes,
233+
make sure to adopt your behavior with `replace` directives accordingly.
234+
235+
### Reusing your extracted API module
236+
237+
Whenever you want to reuse your API module with a separate kubebuilder, we will assume you follow the guide for [using an external Type](/reference/using_an_external_type.md).
238+
When you get to the step `Edit the API files` simply import the dependency with
239+
240+
```shell
241+
go get YOUR_GO_PATH/test-operator/[email protected]
242+
```
243+
244+
and then use it as explained in the guide.
245+
246+
[monorepo]: https://en.wikipedia.org/wiki/Monorepo
247+
[replace-directives]: https://go.dev/ref/mod#go-mod-file-replace
248+
[multi-module-repositories]: https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories

0 commit comments

Comments
 (0)