[BUG] Azure.Identity 1.5.0 freezes up ChainedTokenCredential with ManagedIdentityCredential listed first in local dev

[mikequ-taggysoft]

Describe the bug

Immediately after upgrading Azure.Identity from 1.4.1 to 1.5.0, I noticed all my web projects freeze up at startup in local dev (VS Kestrel).

In my host builder inside Program.cs, I have

var tokenCred = new ChainedTokenCredential(new ManagedIdentityCredential(), new AzureCliCredential());

var secretClient = new SecretClient(
                        new Uri($"https://my-keyvault.vault.azure.net/"),
                        tokenCred);

var certificatesClient = new CertificateClient(
                        new Uri($"https://my-keyvault.vault.azure.net/"),
                            tokenCred);

config.AddAzureKeyVault(secretClient, new KeyVaultSecretManager());

...//load some necessary secret/certs etc

Note that I use ChainedTokenCredential with ManagedIdentityCredential listed first, followed by AzureCliCredential. This would ensure that when the project runs in Azure, managed identity is immediately used. In local dev, managed identity is attempted first which would fail quickly, then AzureCliCredential is successfully used next.

Expected behavior

Normally, the ManagedIdentityCredential should fail quickly (within a second or so) when running in local dev environment, which allows the chained credential to fall through to the next available credential.

Actual behavior

Something changed in Azure.Identity 1.5.0, which makes the program freeze up at ManagedIdentityCredential in local dev for a minute+. No exception/error messages (except Kestrel would time out, saying host is unable to start). But eventually, AzureCliCredential hits and code flows through. Maybe the timetout on ManagedIdentityCredential was misconfigured in the newer package.

Environment:

• Azure.Identity 1.5.0
• Visual Studio 2022 RC1
• ASP.NET Core Web API and Razor projects set to start up via Kestrel

[jsquire]

Thank you for your feedback. Tagging and routing to the team members best able to assist.

[schaabs]

@mikequ-taggysoft sorry you're running into this trouble. This delay was introduced by this refactoring to make the ManagedIdentityCredential endpoint discovery more reliable. Unfortunately, depending on your systems network configuration, failures may not be immediate, and fail due to timeout after the full duration of RetryOptions.NetworkTimeout which I believe defaults to 90 seconcds.

After discovering this timeout behavior in some dev configurations we added this fix which limits the initial network connection timeout when the ManagedIdentityCredential is used in the DefaultAzureCredential. Unfortunately this only fixes the issue in ManagedIdentityCredential when used in the DefaultAzureCredential chain, not when used with a custom chain with the ChainedTokenCredential as in your case.

You should be able to work around this issue by configuring the NetworkTimeout on the ManagedIdentityCredential to limit the time it will wait on a network response. I would start by configuring it to 1 second as we have done in the DefaultAzureCredential, but you can experiment to see what values work best for your development environment and deployed environment. Keep in mind setting too short NetworkTimeout durations will lead to artificial network timeouts when deployed to a managed identity enabled host, causing the ManagedIdentityCredential to throw a CredentialUnavailableException. Below is an example of how you can update your code to configure the NetworkTimeout.

   var miCredOptions = new TokenCredentialOptions { Retry = { NetworkTimeout = TimeSpan.FromSeconds(1) }};

   var tokenCred = new ChainedTokenCredential(new ManagedIdentityCredential(options: miCredOptions), new AzureCliCredential());

We're currently exploring options of how to fix this long delay in some development environments when using ManagedIdentityCredential inside a ChainedTokenCredential, but hopefully this work-around will unblock you in the meantime. Please let me know if you have any trouble with the work-around. I'll update this issue once we have a better idea of what the fix might be, and when it will be available.

[mikequ-taggysoft]

@schaabs Thanks for the clarification! Although I've never noticed it myself, are you basically suggesting that ManagedIdentityCredential can sometimes have a longer latency (potentially >= 1s) while running inside Azure, despite it using Microsoft-managed private network? By any chance you can share some insights on some probability/known services/scenarios where this is more prone to happen (even if anecdotal)?

BTW, the only reason I'm using ChainedTokenCredential over DefaultAzureCredential is because the latter places VisualStudioCredential in a higher priority, and back then I found it to be very slow. AzureCliCredential on the other hand is always very fast.

Now after seeing your post, I did test the VisualStudioCredential again (running VS 2022 RC1) and it appears fast so far. Has the performance of this credential been tuned and can be expected to be consistently speedy nowadays?

[christothes]

BTW, the only reason I'm using ChainedTokenCredential over DefaultAzureCredential is because the latter places VisualStudioCredential in a higher priority, and back then I found it to be very slow. AzureCliCredential on the other hand is always very fast.

You can also skip the VisualStudioCredential with DefaultAzureCredential by setting this to true:

azure-sdk-for-net/sdk/identity/Azure.Identity/src/DefaultAzureCredentialOptions.cs

Line 98 in f993f9d

public bool ExcludeVisualStudioCredential { get; set; }

Now after seeing your post, I did test the VisualStudioCredential again (running VS 2022 RC1) and it appears fast so far. Has the performance of this credential been tuned and can be expected to be consistently speedy nowadays?

There have been no changes in the SDK but perhaps there were some improvements shipped with VS's token utility.

[schaabs]

@mikequ-taggysoft

are you basically suggesting that ManagedIdentityCredential can sometimes have a longer latency (potentially >= 1s) while running inside Azure, despite it using Microsoft-managed private network?

No, a 1 second timeout should be safe since the managed identity endpoint is generally a non-addressable local endpoint. We did some extensive performance benchmarking on various managed identity hosts and found this to be an acceptable limit. What I was trying to suggest is that setting NetworkTimeout to a considerably shorter amount of time, say 300ms, will further speed up your ChainedTokenCredential in your development enviornment, but depending on the host and your application you might see some artificial timeouts when deployed. Sorry for the confusion.