InferenceExtension in kgateway

[artberger]

lands in the kgateway release around 3/14

Resources:

• Contact Daneyon
• InferenceExtension project docs: https://gateway-api-inference-extension.sigs.k8s.io/
• PR about the endpoint picker inference: Makes the Inference Extension Deployer Optional kgateway#10903
• Eitan S.'s lab: https://play.instruqt.com/manage/soloio/tracks/kgateway-inference-extension
• Another thread from Posta: https://solo-io-corp.slack.com/archives/C085A044BHT/p1743128023815909

[danehans]

In support of kgateway-dev/kgateway#10411

[danehans]

Docs TODOs:

1. Quickstart modeled after existing kgtw/inference extension quickstarts.
2. At the end of the quickstart, link to appropriate next steps. One of the links should be for troubleshooting.
3. A troubleshooting doc- kubectl get/describe/logs the appropriate k8s resources, port-forward the control plane and proxy (kubectl port-forward --address 0.0.0.0 deploy/$GATEWAY_NAME -n $GATEWAY_NS 19000 and kubectl port-forward --address 0.0.0.0 -n kgateway-system deploy/kgateway 9097 to get config snapshot), etc.
4. Caveats- InferencePool is only supported as a backendRef from an HTTProute rule, No failureMode support for an InferencePool (xref), no status support, etc.

[artberger]

@danehans we published an initial doc page in the Kgateway docs for Inference Extension. It is mostly just a summary of where kgateway fits in, and then points out to the Inference Extension docs (instead of copying its content). We were planning to work on a more in-depth guide post-release, so I will leave this issue open for that.

https://kgateway.dev/docs/integrations/inference-extension/

[danehans]

Sounds great. Thanks for the update.