[BUG] IMDS token endpoint timeout was changed from 3s to 1s without consulting the IMDS IdentityProvider plugin owners

[isaacbanner]

Library name and version

Azure.Identity 1.7.0

Describe the bug

At some point over the last two years, the IMDS token acquisition flow was updated to the current model used by ImdsManagedIdentitySource and, as a part of this new model, the token acquisition timeout was updated to 1 second, as configured in DefaultAzureCredentialFactory (see InitialImdsConnectionTimeout on line 105).

This is in direct contrast to the three-second timeout that the Managed Identity team previously updated this sdk to observe back in 2020 (see AzureVmImdsProbeTimeoutInSeconds). As far as we're aware, the MI/IMDS team were not involved in the conversation to change this timeout and now we're receiving tickets from partners complaining that these calls are timing out and failing to acquire a token.

Expected behavior

The timeout for calls to the IMDS token endpoint should be three seconds.

Actual behavior

The timeout for calls to the IMDS token is one second.

Reproduction Steps

N/A

Environment

Azure VM/VMSS

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[christothes]

Hi @isaacbanner - Is the 3 second timeout guidance documented somewhere? If so, a link would be greatly appreciated.

[isaacbanner]

There isn't a published SLA for token requests on IMDS that I know of - I can engage the IMDS team if you'd like a more concrete promise on that latency. I mention 3 seconds above because that's what we changed it to intentionally two years ago, and that configuration from our team is no longer being observed by the sdk. Is there documentation anywhere for why this was pulled down to 1s from 3?

[christothes]

The timeout is an implementation detail rather than a guarantee, so this would not be formally documented. However, here is the PR that explains it a bit. The intent is to make it as short as possible while allowing enough time for the endpoint to make a connection (which normally should be very fast). Connection attempts are also retried 3 times, so it should be rare that the endpoint fails to respond after all attempts.

This short timeout should only be used when in the context of the DefaultAzureCredential credential chain. In this scenario, there is tension between wanting the connection to be resilient to failures to connect and failing as quickly as possible so that the developer experience is good when running outside of the cloud environment.