Unable to configure S3 Client when using DynamoDB S3Link

[Danny-UKDM]

Describe the bug

When implementing S3Link for both S3Link.Create() and myLinkedProperty.UploadStreamAsync() there appears to be no method for configuring the AmazonS3Client which is created under the hood, or for providing your own configured instance of an AmazonS3Client.

This appears fine for a production environment, but prevents you from being able to ensure the underlying AmazonS3Client is correctly configured for end-to-end integration testing via a service like localstack.

Despite best efforts to ensure the DynamoDBContext passed into S3Link.Create() is configured for localstack in a "Development" environment (which functions as expected elsewhere), the AmazonS3Client ultimately throws an error of:

"The AWS Access Key Id you provided does not exist in our records"

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Either:

1 - When correctly configuring the DI registration of DynamoDBContext for Development, which is passed into S3Link.Create(), the underlying functionality correctly clones the configuration and creates the correctly Development-configured AmazonS3Client under the hood when performing operations.

2 - S3Link allows the caller to pass their own configured instance(s) of AmazonS3Client which is known to be correctly configured for Development and end-to-end integration testing via a service like localstack.

Current Behavior

1 - Once calling myLinkedProperty.UploadStreamAsync() after using S3Link.Create() with a localstack-configured instance of DynamoDBContext, the following exception is ultimately thrown:

Amazon.S3.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records.
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.InvokeAsync[T](IExecutionContext executionContext) in /_/sdk/src/Handlers/AwsSdk/Internal/XRayPipelineHandler.cs:line 699
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Transfer.Internal.SimpleUploadCommand.ExecuteAsync(CancellationToken cancellationToken)
   at UD.Application.Vedrock.Persistence.Entities.Converse.Command.InsertConverseEntityCommand.Handler.CreateStorageLink(Message[] messages, ConverseEntity entity, CancellationToken cancellationToken) in D:\git\verification\apps\vedrock-service\src\UD.Application.Vedrock\Persistence\Entities\Converse\Command\InsertConverseEntityCommand.cs:line 75

The above happens regardless of manual AmazonDynamoDBClient instance creation with:

serviceCollection
    .AddSingleton<IAmazonDynamoDB>(_ =>
        new AmazonDynamoDBClient(new BasicAWSCredentials("localstack", "localstack"),
            new AmazonDynamoDBConfig
            {
                ServiceURL = "http://localhost:4566/",
                AuthenticationRegion = "eu-west-1"
            }));
            
serviceCollection
       .AddScoped<IDynamoDBContext>(provider =>
           new DynamoDBContext(
               provider.GetRequiredService<IAmazonDynamoDB>(),
               new DynamoDBContextConfig
               {
                   Conversion = DynamoDBEntryConversion.V2,
                   RetrieveDateTimeInUtc = true,
                   ConsistentRead = true,
                   IsEmptyStringValueEnabled = true
               }
           ));          

2 - There appears to be no specific configuration options or opportunities to pass my own configured AmazonS3Client to S3Link

Reproduction Steps

Minimal reproduction via .NET 8 Console App:

using Amazon;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.Runtime;
using SomeNamespace;

const string profile = "localstack";
const string serviceUrl = "http://localhost:4566/";
const string authenticationRegion = "eu-west-1";

var localstackCredentials = new BasicAWSCredentials("localstack", "localstack");
var dynamoDbConfig = new AmazonDynamoDBConfig
{
    Profile = new Profile(profile),
    ServiceURL = serviceUrl,
    AuthenticationRegion = authenticationRegion
};

// There is no opportunity to configure any facets of `S3Link` during the client or context construction, or to pass in your own `AmazonS3Client`
var dynamoDbClient = new AmazonDynamoDBClient(localstackCredentials, dynamoDbConfig);
var dynamoDbContext = new DynamoDBContext(dynamoDbClient,
    new DynamoDBContextConfig
    {
        Conversion = DynamoDBEntryConversion.V2,
        RetrieveDateTimeInUtc = true,
        ConsistentRead = true,
        IsEmptyStringValueEnabled = true
    });

var item = new SomeClass
{
    // There is no opportunity to configure the `AmazonS3Client` here, or to pass in your own
    Prop = S3Link.Create(dynamoDbContext, "bucket", "key", RegionEndpoint.EUWest1)
};

// This then throws "AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records."
await item.Prop.UploadStreamAsync(new MemoryStream("Hello World"u8.ToArray()));

namespace SomeNamespace
{
    public class SomeClass
    {
        public required S3Link Prop { get; set; }
    }
}

csproj:

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net8.0</TargetFramework>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="AWSSDK.DynamoDBv2" Version="3.7.400.21" />
        <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.301" />
        <PackageReference Include="AWSSDK.S3" Version="3.7.403" />
    </ItemGroup>

</Project>

docker-compose.yml:

services:
  localstack:
    image: localstack/localstack:3.7.2
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - DEBUG=${DEBUG:-0}
      - DEFAULT_REGION=eu-west-1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

Possible Solution

From attempting to step through the AWS SDK call stack after attempting to configure for localstack:

• S3Link CreatClientCacheFromContext(DynamoDBContext context) (typo here btw) did not seem to add the created new S3ClientCache to the S3Link.Caches dictionary despite not throwing an exception. This seemed to cause S3ClientCache GetClient to always fall in to if (!this.clientsByRegion.TryGetValue(region.SystemName, out output)) when S3Link operations are performed, where ServiceClientHelpers.CreateServiceFromAssembly() is then used.

• AmazonServiceClient CloneConfig(ClientConfig newConfig), which looks to construct the configuration for the AmazonS3Client created under the hood by S3Link, does not seem to respect client configuration which is necessary for localstack (e.g. maintaining the "localstack" accessKey and secreyKey, maintaining the localstack serviceUrl over using an AWS Region and allowing the necessary setting of ForcePathStyle = true for integration testing)

I believe offering the ability to either configure the created AmazonS3Client via S3Link or to provide your own configured instance of AmazonS3Client to S3Link would allow for end-to-end testing via localstack as expected.

Additional Information/Context

appsettings.Development.json read in as config for my specific code experiencing this issue:

{
    "AWS": {
        "ServiceURL": "http://localhost:4566/",
        "AuthenticationRegion": "eu-west-1",
        "Profile": "localstack"
    },
    "Logging": {
        "LogLevel": {
            "Default": "Information",
            "Microsoft.AspNetCore": "Warning"
        }
    }
}

and my service registrations (called in order):

    private static IServiceCollection ConfigureDelivery(this IServiceCollection serviceCollection, IConfiguration config)
    {
        serviceCollection.AddAWSService<IAmazonSQS>();

        if (string.Equals(Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"), "Development", StringComparison.OrdinalIgnoreCase) ||
            string.Equals(Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT"), "Development", StringComparison.OrdinalIgnoreCase))
        {
            Console.WriteLine("Configuring S3 for 'Development'");

            serviceCollection
                .AddSingleton<IAmazonS3>(_ => new AmazonS3Client(new AmazonS3Config
                {
                    AuthenticationRegion = config.GetValue<string>("AWS:AuthenticationRegion"),
                    ServiceURL = config.GetValue<string>("AWS:ServiceURL"),
                    ForcePathStyle = true
                }));
        }
        else
        {
            serviceCollection.AddAWSService<IAmazonS3>();
        }

        serviceCollection.AddSingleton<AmazonSQSExtendedClient>(provider =>
        {
            var commonConfig = provider.GetRequiredService<IOptions<CommonConfig>>().Value;

            return new AmazonSQSExtendedClient(
                provider.GetRequiredService<IAmazonSQS>(),
                new ExtendedClientConfiguration()
                    .WithLargePayloadSupportEnabled(provider.GetRequiredService<IAmazonS3>(), commonConfig.BucketName)
                    .WithS3KeyProvider(new PrefixedGuidS3KeyProvider())
            );
        });

        return serviceCollection;
    }

    private static IServiceCollection ConfigurePersistence(this IServiceCollection serviceCollection)
    {
        if (string.Equals(Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"), "Development", StringComparison.OrdinalIgnoreCase) ||
            string.Equals(Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT"), "Development", StringComparison.OrdinalIgnoreCase))
        {
            serviceCollection
                .AddSingleton<IAmazonDynamoDB>(_ =>
                    new AmazonDynamoDBClient(new BasicAWSCredentials("localstack", "localstack"),
                        new AmazonDynamoDBConfig
                        {
                            ServiceURL = "http://localhost:4566/",
                            AuthenticationRegion = "eu-west-1"
                        }));
        }
        else
        {
            serviceCollection
                .AddAWSService<IAmazonDynamoDB>();
        }

        serviceCollection
               .AddScoped<IDynamoDBContext>(provider =>
                   new DynamoDBContext(
                       provider.GetRequiredService<IAmazonDynamoDB>(),
                       new DynamoDBContextConfig
                       {
                           Conversion = DynamoDBEntryConversion.V2,
                           RetrieveDateTimeInUtc = true,
                           ConsistentRead = true,
                           IsEmptyStringValueEnabled = true
                       }
                   ))
               .AddKeyedSingleton<IDataRepository, DynamoDataRepository>(DataEngine.DynamoDb);

        return serviceCollection;
    }

NB - I have confirmed that it is indeed the Development routes which are invoked as part of my end-to-end testing

AWS .NET SDK and/or Package version used

AWSSDK.DynamoDBv2 3.7.400.21
AWSSDK.Extensions.NETCore.Setup 3.7.301
AWSSDK.S3 3.7.403

Targeted .NET Platform

.NET 8

Operating System and version

Windows 11 (with WSL)

[peterrsongg]

@Danny-UKDM We added a new configuration option called service-specific endpoints where you can set an environment variable or a profile for a specific service and that will set the serviceURL for that service. Can you try adding this in your application?

AWS_ENDPOINT_URL_S3 = "http://localhost:4566/";

[Danny-UKDM]

@Danny-UKDM We added a new configuration option called service-specific endpoints where you can set an environment variable or a profile for a specific service and that will set the serviceURL for that service. Can you try adding this in your application?

AWS_ENDPOINT_URL_S3 = "http://localhost:4566/";

@peterrsongg Thanks for leading me to that 👍 it seems to have resolved the issue with ServiceUrl property on the AmazonS3Client that get's created. However, the issue remains that I seem to be unable to set ForcePathStyle to true on the AmazonS3Client which is created under-the-hood; this is typically needed to make any AmazonS3Client work with localstack.

I managed to break here on an e2e test run and manually change ForcePathStyle to true to force the returned client from the S3ClientCache to be configured for localstack as typical; the e2e tests ran against S3 in localstack as expected.

Could there be a way to override the ForcePathStyle configuration setting for any instance of AmazonS3Client created, similar to what the AWS_ENDPOINT_URL_S3 environment variable achieves? I believe that would allow localstack users to complete e2e testing when S3Links are involved 🤔

[peterrsongg]

@Danny-UKDM There is no way to do that via environment variables, only the service-specific endpoints work with that at the moment. However, you should be able to do something like

s3 = 
   force_path_style = true

in your config file but I see that we haven't added that as a reserved property in our SharedCredentialsFile, so even if you do, we won't read that value. Maybe there is a reason for this. I will bring this up with the team and see if this is something we could support or if there are other opportunites here for alternative solutions

[Danny-UKDM]

@Danny-UKDM There is no way to do that via environment variables, only the service-specific endpoints work with that at the moment. However, you should be able to do something like

s3 = 
   force_path_style = true

in your config file but I see that we haven't added that as a reserved property in our SharedCredentialsFile, so even if you do, we won't read that value. Maybe there is a reason for this. I will bring this up with the team and see if this is something we could support or if there are other opportunites here for alternative solutions

@peterrsongg that would be greatly appreciated, thank you! 👍

I must say, with a few years of experience working with AmazonS3Client, I did determine a few moons back that (probably for what you're hinting at here) ForcePathStyle, specifically, would never be read from a config file when using GetAWSOptions() from AWSSDK.Extensions.NETCore.Setup - I have typically always needed to swap the registration of IAmazonS3 in DI when performing integration testing for an instance where I explicitly set ForcePathStyle in the AmazonS3Client object initializer 🤔

If AWS client construction takes client configuration from IConfiguration with precedence over how they're sometimes constructed under-the-hood by the SDK, then enabling ForcePathStyle to be read from configuration certainly sounds like a potential solution 👍

[peterrsongg]

@Danny-UKDM Spoke with the team and decided this is most likely just a miss when implementing ForcePathStyle and we forgot to add it as an option in the config file as well. We have created a backlog item to address this. Thank you for bringing this to our attention