Skip to content

Commit 0ccbe54

Browse files
pecameronpecameron
committed
Document a/b deployment
A route can front up to 4 services that handle the requests. The load balancing strategy governs which endpoint gets each request. When roundrobin is chosen, the portion of the requests that each service handles is governed by the weight assigned to the service. Each endpoint in the service gets a fraction of the service's requests bug 1470350 https://bugzilla.redhat.com/show_bug.cgi?id=1470350 Code change is in origin PR 15309 openshift/origin#15309
1 parent fb8300a commit 0ccbe54

File tree

5 files changed

+407
-178
lines changed

5 files changed

+407
-178
lines changed

‎architecture/core_concepts/routes.adoc

Lines changed: 116 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -197,14 +197,14 @@ from other connections, or turn off stickiness entirely.
197197

198198
By default, sticky sessions for passthrough routes are implemented using the
199199
`source` xref:load-balancing[load balancing strategy]. The default can be
200-
changed for all passthrough routes by using the `*ROUTER_TCP_BALANCE_SCHEME*`
200+
changed for all passthrough routes by using the `ROUTER_TCP_BALANCE_SCHEME`
201201
xref:env-variables[environment variable], and for individual routes by using the
202202
`*haproxy.router.openshift.io/balance*` xref:route-specific-annotations[route
203203
specific annotation].
204204

205205
Other types of routes use the `leastconn` xref:load-balancing[load balancing
206206
strategy] by default, which can be changed by using the
207-
`*ROUTER_LOAD_BALANCE_ALGORITHM*` xref:env-variables[environment variable]. It
207+
`ROUTER_LOAD_BALANCE_ALGORITHM` xref:env-variables[environment variable]. It
208208
can be changed for individual routes by using the
209209
`*haproxy.router.openshift.io/balance*` xref:route-specific-annotations[route
210210
specific annotation].
@@ -261,53 +261,53 @@ $ oc set env dc/router ROUTER_SYSLOG_ADDRESS=127.0.0.1 ROUTER_LOG_LEVEL=debug
261261
[cols="2,2,6", options="header"]
262262
|===
263263
|Variable | Default | Description
264-
|`*DEFAULT_CERTIFICATE*` | | The contents of a default certificate to use for routes that don't expose a TLS server cert; in PEM format.
265-
|`*DEFAULT_CERTIFICATE_DIR*` | | A path to a directory that contains a file named *_tls.crt_*. If *_tls.crt_* is not a PEM file which also contains a private key, it is first combined with a file named tls.key in the same directory. The PEM-format contents are then used as the default certificate. Only used if `DEFAULT_CERTIFICATE` or `DEFAULT_CERTIFICATE_PATH` are not specified.
266-
|`*DEFAULT_CERTIFICATE_PATH*` | | A path to default certificate to use for routes that don't expose a TLS server cert; in PEM format. Only used if `DEFAULT_CERTIFICATE` is not specified.
267-
|`*EXTENDED_VALIDATION*` | `true` | If `true`, the router confirms that the certificate is structurally correct. It does not verify the certificate against any CA. Set `false` to turn off the tests.
268-
|`*NAMESPACE_LABELS*` | | A label selector to apply to namespaces to watch, empty means all.
269-
|`*PROJECT_LABELS*` | | A label selector to apply to projects to watch, emtpy means all.
270-
|`*RELOAD_SCRIPT*` | | The path to the reload script to use to reload the router.
271-
|`*ROUTER_ALLOWED_DOMAINS*` | | A comma-separated list of domains that the host name in a route can only be part of. Any subdomain in the domain can be used. Option `ROUTER_DENIED_DOMAINS` overrides any values given in this option. If set, everything outside of the allowed domains will be rejected.
272-
|`*ROUTER_BACKEND_CHECK_INTERVAL*` | 5000ms | Length of time between subsequent "liveness" checks on backends. xref:time-units[(TimeUnits)]
273-
|`*ROUTER_COMPRESSION_MIME*` | "text/html text/plain text/css" | A space separated list of mime types to compress.
274-
|`*ROUTER_DEFAULT_CLIENT_TIMEOUT*`| 30s | Length of time within which a client has to acknowledge or send data. xref:time-units[(TimeUnits)]
275-
|`*ROUTER_DEFAULT_CONNECT_TIMEOUT*`| 5s | The maximum connect time. xref:time-units[(TimeUnits)]
276-
|`*ROUTER_DEFAULT_SERVER_TIMEOUT*`| 30s | Length of time within which a server has to acknowledge or send data. xref:time-units[(TimeUnits)]
277-
|`*ROUTER_DEFAULT_TUNNEL_TIMEOUT*` | 1h | Length of time till which TCP or WebSocket connections will remain open. If you have websockets/tcp
264+
|`DEFAULT_CERTIFICATE` | | The contents of a default certificate to use for routes that don't expose a TLS server cert; in PEM format.
265+
|`DEFAULT_CERTIFICATE_DIR` | | A path to a directory that contains a file named *_tls.crt_*. If *_tls.crt_* is not a PEM file which also contains a private key, it is first combined with a file named tls.key in the same directory. The PEM-format contents are then used as the default certificate. Only used if `DEFAULT_CERTIFICATE` or `DEFAULT_CERTIFICATE_PATH` are not specified.
266+
|`DEFAULT_CERTIFICATE_PATH` | | A path to default certificate to use for routes that don't expose a TLS server cert; in PEM format. Only used if `DEFAULT_CERTIFICATE` is not specified.
267+
|`EXTENDED_VALIDATION` | `true` | If `true`, the router confirms that the certificate is structurally correct. It does not verify the certificate against any CA. Set `false` to turn off the tests.
268+
|`NAMESPACE_LABELS` | | A label selector to apply to namespaces to watch, empty means all.
269+
|`PROJECT_LABELS` | | A label selector to apply to projects to watch, emtpy means all.
270+
|`RELOAD_SCRIPT` | | The path to the reload script to use to reload the router.
271+
|`ROUTER_ALLOWED_DOMAINS` | | A comma-separated list of domains that the host name in a route can only be part of. Any subdomain in the domain can be used. Option `ROUTER_DENIED_DOMAINS` overrides any values given in this option. If set, everything outside of the allowed domains will be rejected.
272+
|`ROUTER_BACKEND_CHECK_INTERVAL` | 5000ms | Length of time between subsequent "liveness" checks on backends. xref:time-units[(TimeUnits)]
273+
|`ROUTER_COMPRESSION_MIME` | "text/html text/plain text/css" | A space separated list of mime types to compress.
274+
|`ROUTER_DEFAULT_CLIENT_TIMEOUT`| 30s | Length of time within which a client has to acknowledge or send data. xref:time-units[(TimeUnits)]
275+
|`ROUTER_DEFAULT_CONNECT_TIMEOUT`| 5s | The maximum connect time. xref:time-units[(TimeUnits)]
276+
|`ROUTER_DEFAULT_SERVER_TIMEOUT`| 30s | Length of time within which a server has to acknowledge or send data. xref:time-units[(TimeUnits)]
277+
|`ROUTER_DEFAULT_TUNNEL_TIMEOUT` | 1h | Length of time till which TCP or WebSocket connections will remain open. If you have websockets/tcp
278278
connections (and any time HAProxy is reloaded), the old HAProxy processes
279279
will "linger" around for that period. xref:time-units[(TimeUnits)]
280-
|`*ROUTER_DENIED_DOMAINS*` | | A comma-separated list of domains that the host name in a route can not be part of. No subdomain in the domain can be used either. Overrides option `ROUTER_ALLOWED_DOMAINS`.
281-
|`*ROUTER_ENABLE_COMPRESSION*`| | If `true` or `TRUE`, compress responses when possible.
282-
|`*ROUTER_LOG_LEVEL*` | warning | The log level to send to the syslog server.
283-
|`*ROUTER_MAX_CONNECTIONS*`| 20000 | Maximum number of concurrent connections.
284-
|`*ROUTER_OVERRIDE_HOSTNAME*`| | If set `true`, override the spec.host value for a route with the template in `ROUTER_SUBDOMAIN`.
285-
|`*ROUTER_SERVICE_HTTPS_PORT*` | 443 | Port to listen for HTTPS requests.
286-
|`*ROUTER_SERVICE_HTTP_PORT*` | 80 | Port to listen for HTTP requests.
287-
|`*ROUTER_SERVICE_NAME*` | public | The name that the router identifies itself in the in route status.
288-
|`*ROUTER_CANONICAL_HOSTNAME*` | | The (optional) host name of the router shown in the in route status.
289-
|`*ROUTER_SERVICE_NAMESPACE*` | | The namespace the router identifies itself in the in route status. Required if `ROUTER_SERVICE_NAME` is used.
290-
|`*ROUTER_SERVICE_NO_SNI_PORT*` | 10443 | Internal port for some front-end to back-end communication (see note below).
291-
|`*ROUTER_SERVICE_SNI_PORT*` | 10444 | Internal port for some front-end to back-end communication (see note below).
292-
| `*ROUTER_SLOWLORIS_HTTP_KEEPALIVE*`| 300s | Set the maximum time to wait for a new HTTP request to appear. If this is set too low, it can confuse browsers and applications not expecting a small `keepalive` value. xref:time-units[(TimeUnits)]
293-
|`*ROUTER_SLOWLORIS_TIMEOUT*` | 10s | Length of time the transmission of an HTTP request can take. xref:time-units[(TimeUnits)]
294-
|`*ROUTER_SUBDOMAIN*`| | The template that should be used to generate the host name for a route without spec.host (e.g. ${name}-${namespace}.myapps.mycompany.com).
295-
|`*ROUTER_SYSLOG_ADDRESS*` | | Address to send log messages. Disabled if empty.
296-
|`*ROUTER_SYSLOG_FORMAT*` | | If set, override the default log format used by underlying router implementation. Its value should conform with underlying router implementation's specification.
297-
|`*ROUTER_TCP_BALANCE_SCHEME*` | source | xref:load-balancing[Load-balancing strategy] for multiple endpoints for pass-through routes. Available options are `source`, `roundrobin`, or `leastconn`.
298-
|`*ROUTER_LOAD_BALANCE_ALGORITHM*` | leastconn | xref:load-balancing[Load-balancing strategy] for routes with multiple endpoints. Available options are `source`, `roundrobin`, and `leastconn`.
280+
|`ROUTER_DENIED_DOMAINS` | | A comma-separated list of domains that the host name in a route can not be part of. No subdomain in the domain can be used either. Overrides option `ROUTER_ALLOWED_DOMAINS`.
281+
|`ROUTER_ENABLE_COMPRESSION`| | If `true` or `TRUE`, compress responses when possible.
282+
|`ROUTER_LOG_LEVEL` | warning | The log level to send to the syslog server.
283+
|`ROUTER_MAX_CONNECTIONS`| 20000 | Maximum number of concurrent connections.
284+
|`ROUTER_OVERRIDE_HOSTNAME`| | If set `true`, override the spec.host value for a route with the template in `ROUTER_SUBDOMAIN`.
285+
|`ROUTER_SERVICE_HTTPS_PORT` | 443 | Port to listen for HTTPS requests.
286+
|`ROUTER_SERVICE_HTTP_PORT` | 80 | Port to listen for HTTP requests.
287+
|`ROUTER_SERVICE_NAME` | public | The name that the router identifies itself in the in route status.
288+
|`ROUTER_CANONICAL_HOSTNAME` | | The (optional) host name of the router shown in the in route status.
289+
|`ROUTER_SERVICE_NAMESPACE` | | The namespace the router identifies itself in the in route status. Required if `ROUTER_SERVICE_NAME` is used.
290+
|`ROUTER_SERVICE_NO_SNI_PORT` | 10443 | Internal port for some front-end to back-end communication (see note below).
291+
|`ROUTER_SERVICE_SNI_PORT` | 10444 | Internal port for some front-end to back-end communication (see note below).
292+
|`ROUTER_SLOWLORIS_HTTP_KEEPALIVE`| 300s | Set the maximum time to wait for a new HTTP request to appear. If this is set too low, it can confuse browsers and applications not expecting a small `keepalive` value. xref:time-units[(TimeUnits)]
293+
|`ROUTER_SLOWLORIS_TIMEOUT` | 10s | Length of time the transmission of an HTTP request can take. xref:time-units[(TimeUnits)]
294+
|`ROUTER_SUBDOMAIN`| | The template that should be used to generate the host name for a route without spec.host (e.g. ${name}-${namespace}.myapps.mycompany.com).
295+
|`ROUTER_SYSLOG_ADDRESS` | | Address to send log messages. Disabled if empty.
296+
|`ROUTER_SYSLOG_FORMAT` | | If set, override the default log format used by underlying router implementation. Its value should conform with underlying router implementation's specification.
297+
|`ROUTER_TCP_BALANCE_SCHEME` | source | xref:load-balancing[Load-balancing strategy] for multiple endpoints for pass-through routes. Available options are `source`, `roundrobin`, or `leastconn`.
298+
|`ROUTER_LOAD_BALANCE_ALGORITHM` | leastconn | xref:load-balancing[Load-balancing strategy] for routes with multiple endpoints. Available options are `source`, `roundrobin`, and `leastconn`.
299299
//|`*ROUTE_FIELDS*` | | A field selector to apply to routes to watch, empty means all. (FUTURE: it does not have complete support we need in upstream/k8s.)
300-
|`*ROUTE_LABELS*` | | A label selector to apply to the routes to watch, empty means all.
301-
|`*STATS_PASSWORD*` | | The password needed to access router stats (if the router implementation supports it).
302-
|`*STATS_PORT*` | | Port to expose statistics on (if the router implementation supports it). If not set, stats are not exposed.
303-
|`*STATS_USERNAME*` | | The user name needed to access router stats (if the router implementation supports it).
304-
|`*TEMPLATE_FILE*` | `/var/lib/haproxy/conf/custom/` `haproxy-config-custom.template` | The path to the HAProxy template file (in the container image).
305-
|`*RELOAD_INTERVAL*` | 12s | The minimum frequency the router is allowed to reload to accept new changes. xref:time-units[(TimeUnits)]
306-
|`*ROUTER_USE_PROXY_PROTOCOL*`| | When set to `true` or `TRUE`, HAProxy expects incoming connections to use the `PROXY` protocol on port 80 or port 443. The source IP address can pass through a load balancer if the load balancer supports the protocol, for example Amazon ELB.
307-
|`*ROUTER_ALLOW_WILDCARD_ROUTES*`| | When set to `true` or `TRUE`, any routes with a wildcard policy of `Subdomain` that pass the router admission checks will be serviced by the HAProxy router.
308-
|`*ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK*` | | Set to `true` to relax the namespace ownership policy.
309-
|`*ROUTER_STRICT_SNI*` | | xref:strict-sni[strict-sni]
310-
|`*ROUTER_CIPHERS*` | intermediate | Specify the set of xref:ciphers[ciphers] supported by bind.
300+
|`ROUTE_LABELS` | | A label selector to apply to the routes to watch, empty means all.
301+
|`STATS_PASSWORD` | | The password needed to access router stats (if the router implementation supports it).
302+
|`STATS_PORT` | | Port to expose statistics on (if the router implementation supports it). If not set, stats are not exposed.
303+
|`STATS_USERNAME` | | The user name needed to access router stats (if the router implementation supports it).
304+
|`TEMPLATE_FILE` | `/var/lib/haproxy/conf/custom/` `haproxy-config-custom.template` | The path to the HAProxy template file (in the container image).
305+
|`RELOAD_INTERVAL` | 12s | The minimum frequency the router is allowed to reload to accept new changes. xref:time-units[(TimeUnits)]
306+
|`ROUTER_USE_PROXY_PROTOCOL`| | When set to `true` or `TRUE`, HAProxy expects incoming connections to use the `PROXY` protocol on port 80 or port 443. The source IP address can pass through a load balancer if the load balancer supports the protocol, for example Amazon ELB.
307+
|`ROUTER_ALLOW_WILDCARD_ROUTES`| | When set to `true` or `TRUE`, any routes with a wildcard policy of `Subdomain` that pass the router admission checks will be serviced by the HAProxy router.
308+
|`ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK` | | Set to `true` to relax the namespace ownership policy.
309+
|`ROUTER_STRICT_SNI` | | xref:strict-sni[strict-sni]
310+
|`ROUTER_CIPHERS` | intermediate | Specify the set of xref:ciphers[ciphers] supported by bind.
311311
|===
312312

313313
[NOTE]
@@ -353,11 +353,11 @@ number of running servers changing, many clients will be
353353
directed to different servers. This algorithm is generally
354354
used with xref:routes-sticky-sessions[passthrough routes].
355355

356-
The `*ROUTER_TCP_BALANCE_SCHEME*` environment variable sets the default
357-
for passthorugh routes. The `*ROUTER_LOAD_BALANCE_ALGORITHM*` environment
358-
variable sets the default strategy for the router for the remaining routes.
356+
The `ROUTER_TCP_BALANCE_SCHEME` xref:env-variables[environment variable] sets the default
357+
strategy for passthorugh routes. The `ROUTER_LOAD_BALANCE_ALGORITHM` xref:env-variables[environment
358+
variable] sets the default strategy for the router for the remaining routes.
359359
A xref:route-specific-annotations[route specific annotation],
360-
`*haproxy.router.openshift.io/balance*` may be use to control specific routes.
360+
`*haproxy.router.openshift.io/balance*`, may be used to control specific routes.
361361

362362
[[strict-sni]]
363363
== HAProxy Strict SNI
@@ -897,6 +897,70 @@ criteria, it will replace the existing route based on the above mentioned
897897
resolution order (oldest route wins).
898898
====
899899

900+
[[alternateBackends]]
901+
== Alternate Backends and Weights
902+
903+
A route is usually associated with one service through the `to:` token with `kind: Service`.
904+
All of the requests to the route are handled by endpoints in the service based on the
905+
xref:load-balancing[load balancing strategy].
906+
907+
It is possible to have as many as 4 services supporting the route. The portion of
908+
requests that are handled by each service is governed by the service `weight`.
909+
910+
The first service is entered using the `to:` token as before, and up to three additional
911+
services can be entered using the `alternateBackend:` token. Each service must be
912+
`kind: Service` which is the default.
913+
914+
Each service can have a `weight` associated with it. The portion of requests handled
915+
by the service is `weight` / `sum_of_all_weights`. When a service has more than one
916+
endpoint, the service's weight is distributed among the endpoints with each endpoint
917+
getting at least 1. If the service `weight` is 0 each endpoint will get 0.
918+
919+
The `weight` must be in the range 0-256. The default is 1.
920+
When the `weight` is 0 no requests are passd to the service. If all services have
921+
`weight` 0, requests are returned with a 503 error.
922+
923+
When using `alternateBackends` also use the `roundrobin` xref:load-balancing[load balancing strategy]
924+
to ensure requests are distributed as expected to the services based on `weight`.
925+
`roundrobin` can be set for a route using a xref:route-specific-annotations[route annotation],
926+
or for the router in general using an xref:env-variables[environment variable].
927+
928+
Here is an example of using alternate backends for
929+
xref:../../dev_guide/deployments/advanced_deployment_strategies.adoc#advanced-deployment-a-b-deployment[a/b deployments].
930+
931+
.A Route with alternateBackends and weights:
932+
====
933+
934+
[source,yaml]
935+
----
936+
apiVersion: v1
937+
kind: Route
938+
metadata:
939+
name: route-alternate-service
940+
annotations:
941+
haproxy.router.openshift.io/balance: roundrobin <1>
942+
spec:
943+
host: www.example.com
944+
to:
945+
kind: Service
946+
name: service-name <2>
947+
weight: 20 <4>
948+
alternateBackends:
949+
- kind: Service
950+
name: service-name2 <3>
951+
weight: 10 <4>
952+
kind: Service
953+
name: service-name3 <3>
954+
weight: 10 <4>
955+
----
956+
957+
<1> This route uses `roundrobin` xref:load-balancing[load balancing strategy]
958+
<2> The first service name is `service-name` which may have 0 or more pods
959+
<3> The alternateBackend services may also have 0 or more pods
960+
<4> The total `weight` is 40. `service-name` will get 20/40 or 1/2 of the requests,
961+
`service-name2` and `service-name3` will each get 1/4 of the requests.
962+
====
963+
900964

901965
[[route-specific-annotations]]
902966
== Route-specific Annotations
@@ -1045,8 +1109,8 @@ the host names in a route using the `ROUTER_DENIED_DOMAINS` and
10451109
[cols="2"]
10461110
|===
10471111

1048-
|`*ROUTER_DENIED_DOMAINS*` | Domains listed are not allowed in any indicated routes.
1049-
|`*ROUTER_ALLOWED_DOMAINS*` | Only the domains listed are allowed in any indicated routes.
1112+
|`ROUTER_DENIED_DOMAINS` | Domains listed are not allowed in any indicated routes.
1113+
|`ROUTER_ALLOWED_DOMAINS` | Only the domains listed are allowed in any indicated routes.
10501114

10511115
|===
10521116

0 commit comments

Comments
 (0)