Skip to content

Amend the endpoint picker protocol to support multiple fallback endpoints #761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
+15 −16
Open

Amend the endpoint picker protocol to support multiple fallback endpoints #761

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c62d3b3
Amend the endpoint picker protocol to support multiple fallback endpo…
wbpcode Apr 30, 2025
1047cb6
address comments
wbpcode May 9, 2025
f7d96a8
add optionally
wbpcode May 9, 2025
c204cf7
address comments
wbpcode May 13, 2025
6a3c8f6
address comments
wbpcode May 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Amend the endpoint picker protocol to support multiple fallback endpo… 
…ints

Signed-off-by: wangbaiping(wbpcode) <[email protected]>
  • Loading branch information
@wbpcode
wbpcode committed Apr 30, 2025
commit c62d3b3026abef07cca4a91d38572020ffd17088
18 changes: 15 additions & 3 deletions docs/proposals/004-endpoint-picker-protocol/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,23 @@ Constraints:
- If the EPP did not communicate the server endpoint via these two methods, it MUST return an error as follows:
- [ImmediateResponse](https://github.com/envoyproxy/envoy/blob/f2023ef77bdb4abaf9feef963c9a0c291f55568f/api/envoy/service/ext_proc/v3/external_processor.proto#L195) with 503 (Serivce Unavailable) HTTP status code if there are no ready endpoints.
- [ImmediateResponse](https://github.com/envoyproxy/envoy/blob/f2023ef77bdb4abaf9feef963c9a0c291f55568f/api/envoy/service/ext_proc/v3/external_processor.proto#L195) with 429 (Too Many Requests) HTTP status code if the request should be dropped (e.g., a Sheddable request, and the servers under heavy load).
- The EPP MUST not set two different values in the header and the inner response metadata value.
- The EPP MUST not set two different values in the header and the inner response metadata value.
- Setting different value leads to unpredictable behavior because proxies aren't guaranteed to support both paths, and so this protocol does not define what takes precedence.

### Destination endpoint fallback
A single fallback endpoint CAN be set using the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows:

For each HTTP request, if destination endpoint fallback is necessary or possible, the EPP CAN set the `x-gateway-destination-endpoint` HTTP header or metadata entry with multiple addresses in `<ip:port>,<ip:port>,...` format. Multiple addresses are separated by commas. The first valid endpoint in the addresses list will be used as the primary endpoint. And if retrying is happening, the proxy will try the endpoints after the selected endpoint in order.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we continue to have this subsection or update the ## Destination Endpoint section to state that one address MUST be set or multiple addresses CAN be set for fallback purposes? Thoughts @ahg-g @kfswain?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated content of the ## Destination Endpoint section.

The value of the header or metadata entry MUST contains at least one endpoint in `<ip:port>` format or multiple endpoints in `<ip:port>,<ip:port>,...` format. Multiple endpoints are separated by commas. The first valid endpoint in the value will be used. And if retrying is happening, the proxy will try the endpoints after the previously selected endpoint in order.

The x-gateway-destination-endpoint-fallback is still be kept for backward compatibility.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this is a breaking change that will impact the Istio implementation at least, which is based on Envoy LB subsetting and requires a single value to be specified.

Adding @LiorLieberman for feedback.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From API's perspective, this is not a break change. But if take the implementation detail into account, this actually may break the istio implementation. But ideally, we should have a flag to enable the new format or use legacy format in the EPP.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a comment below


For example:
```go
dynamicMetadata: {
"envoy.lb" {
"x-gateway-destination-endpoint": "<ip:port>,<ip:port>,..."
}
}
```

Single fallback endpoint also CAN be set using the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the EPP still support x-gateway-destination-endpoint-fallback if x-gateway-destination-endpoint now supports endpoint fallback by listing multiple addresses? cc: @ahg-g @kfswain

Copy link
Author

@wbpcode wbpcode May 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the API break change acceptable or is there a way to deprecate the x-gateway-destination-endpoint-fallback? Ideally, the EPP should has a flag to enable the new format or use the legacy format. And then we can deprecate the legacy format gradually.


```go
dynamicMetadata: {
Expand All @@ -58,7 +70,7 @@ dynamicMetadata: {
}
```

### Why envoy.lb namespace as a default?
### Why envoy.lb namespace as a default?
The `envoy.lb` namespace is a predefined namespace. One common way to use the selected endpoint returned from the server, is [envoy subsets](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/subsets) where host metadata for subset load balancing must be placed under `envoy.lb`. Note that this is not related to the subsetting feature discussed above, this is an enovy implementation detail.

## Matching An InferenceModel
Expand Down