container_namespace should infer project_id from provider but doesn't

[bennycornelissen]

Terraform Version

Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/scaleway/scaleway v2.53.0

Affected Resource(s)

• scaleway_container_namespace

Terraform Configuration Files

provider "scaleway" {
  #should be the ID of a non-default project
  project_id = "REDACTED"
  region     = "nl-ams"
  zone       = "nl-ams-1"
}

resource "scaleway_container_namespace" "foo" {
  name       = "foo"
}

Expected Behavior

Create a Serverless Containers namespace in the project matching the ID from the provider.

Actual Behavior

Creates a Serverless Containers namespace in the default project.

Steps to Reproduce

1. Apply
2. Destroy
3. Update the scaleway_container_namespace resource, adding in the project_id argument
4. Apply again. It will now create the namespace in the correct place.

[remyleone]

Hello thanks for filling up the issue. Could you try to set it explicitly to see if it can solve the problem in the meantime we prepare a fix? https://registry.terraform.io/providers/scaleway/scaleway/latest/docs/resources/container_namespace

[bennycornelissen]

Hi @remyleone I did that (see step 3 in "Steps to reproduce"). Setting the project_id explicitly as part of the resource works fortunately.

[bennycornelissen]

Another weird, possibly related thing here. My workaround adding the project_id argument is only necessary for creating the scaleway_container_namespace. Once created, I can remove it and subsequent Terraform runs will be fine.

[bennycornelissen]

I was just deleting a container namespace that was created in the wrong project (due to this bug) and I noticed that the project_id and organization_id attributes for the container namespace are identical. As if it somehow ignored the project_id in the provider and falls back to the organization ID?

[bennycornelissen]

Hi again, it seems this bug is not constrained to this specific resource. I'm currently in the process of testing out Scaleway specifically, and my first 'demo project' was a serverless container setup, but now I'm delving into other things and I'm seeing the same issue.

Code:

provider "scaleway" {
  project_id = "REDACTED PROJECT UUID"
  region = "nl-ams"
  zone   = "nl-ams-1"
}

# Get info by Region key
data "scaleway_availability_zones" "main" {
  region = "nl-ams"
}

locals {
  environment_name = "k8s-cluster-basic"

  # Public Gateways are a zonal resource. Select how many you want (should not be more than the amount of zones)
  public_gateway_count = 1
  public_gateway_zones = toset(slice(data.scaleway_availability_zones.main.zones, 0, local.public_gateway_count))
}

resource "scaleway_vpc" "main" {
  name       = local.environment_name
}

resource "scaleway_vpc_private_network" "k8s" {
  name       = local.environment_name
  vpc_id     = scaleway_vpc.main.id

  ipv4_subnet {
    subnet = "10.0.1.0/24"
  }
}

resource "scaleway_vpc_public_gateway" "main" {
  for_each = local.public_gateway_zones

  name       = "${local.environment_name}-gw-${each.key}"
  type       = "VPC-GW-M"

  bastion_enabled = true
  bastion_port    = 2222

}

The project_id in my provider block is set to the UUID of a non-default project in my organization. If I apply this code I should end up with the VPC, private network, and public gateway (so far) in that specific project. Instead, it ends up in my 'default' project.

Similar to the issue with the container namespace, if I add the project_id argument explicitly in the resource definitions, everything works fine.

This leads me to believe that some internal shared logic used to determine the project ID if it isn't set as a resource argument is broken.

[Gnoale]

Hello @bennycornelissen
Thanks for reporting it, the fix for container should be merged soon
we will prepare fixes for other resources as well
Best

[bennycornelissen]

Update: after some back-and-forth with @remyleone on Slack, it seems we may have found the issue: provider config precedence, in a way that I simply hadn't considered.

In my environment, I had the following variables set:

❯ hasenv SCW
SCW_SECRET_KEY
SCW_DEFAULT_ORGANIZATION_ID
SCW_ACCESS_KEY
SCW_DEFAULT_PROJECT_ID

The SCW_DEFAULT_PROJECT_ID environment variable was set to the UUID of the default project (same UUID as the org).

In my Terraform code, I tested 2 types of provider config; one simple, and one a bit more elaborate:

Flavor one:

provider "scaleway" {
  project_id = "<UUID here>"
  region     = "nl-ams"
  zone       = "nl-ams-1"
}

Flavor two:

locals {
  scw_provider_region       = "nl-ams"
  scw_provider_zone         = "nl-ams-1"
  scw_provider_project_name = "<my project name here>"

  environment_name = "k8s-cluster-basic"
}

# This is the 'root' provider. We use this to dynamically get the UUID for the Scaleway project we actually
# want to create resources in
provider "scaleway" {
  alias = "scaleway-root"

  # NOTE: Authentication secret, Org ID, and default project are expected to be supplied through the environment.
  # make sure you set SWC_ACCESS_KEY, SWC_SECRET_KEY, SCW_DEFAULT_ORGANIZATION_ID, and SCW_DEFAULT_PROJECT_ID
  region = local.scw_provider_region
  zone   = local.scw_provider_zone
}

# Look up the Scaleway project
data "scaleway_account_project" "project" {
  name     = local.scw_provider_project_name
  provider = scaleway.scaleway-root
}

# This is the 'main' scaleway provider that we will use for all other resources
provider "scaleway" {
  project_id = data.scaleway_account_project.project.id
  region     = local.scw_provider_region
  zone       = local.scw_provider_zone
}

Then, if I would create a project-scoped resource without specifying a project_id argument, I was expecting it to be created in my non-default project, because that's what I set in the provider "scaleway" {} block. However, I didn't consider that it would be overridden by the SCW_DEFAULT_PROJECT_ID environment variable.

Unsetting that variable works. If my environment looks like below, my project-scoped resources are created in the project where I expect them, and even my 'Flavor two' approach with a 'scaleway-root provider' to look up the actual project UUID based on name works perfectly.

❯ hasenv SCW
SCW_SECRET_KEY
SCW_DEFAULT_ORGANIZATION_ID
SCW_ACCESS_KEY

So it may very well be that this 'bug' is a massive case of PEBKAC on my end 😅 Unless this behaviour is not intentional.. then it's a bug. I'm not sure if the docs are clear enough on this particular provider config precedence, or that I just did a poor job of reading them (very possible). I'll leave that to you to decide.

TL;DR: if you are setting project_id in your provider block, don't use the SCW_DEFAULT_PROJECT_ID environment variable at the same time...

[Gnoale]

Hello, ahah ok exactly, I was writing you and saw your comment 🥇

Hello, there is a precedence over the environment variables when building the scaleway client in the provider.

This is probably why your provider config is not taken in account here.

We will not change the order at the moment because it could cause unexpected behavior for existing configuration.

Could you make sure that you don't have any SCW_DEFAULT_PROJECT_ID in your environment ? and your provider config should work as expected

[bennycornelissen]

@Gnoale I got a Slack message from @remyleone earlier today, asking me to check my environment variables... 😉 It's been a few years since provider config precedence caught me. Good call as well on not changing the order either 👍

[remyleone]

It would still be helpful to provide more logs about the origin of the credentials we get. So that way it is clear to identify the provenance. CLI is already doing it. We might implement it in the logs of terraform

[bennycornelissen]

Also, as I mentioned on Slack as well, you can't really tell at the plan stage currently in which project resources will be created as it's a computed value.

While I understand that's not easily fixed, it would be helpful to offer a provider level argument as a safeguard, similar to the allowed_account_ids attribute in the AWS provider.

Example:

provider scaleway {
  allowed_project_ids = [XYZABCDEF]
}

If I were to run Terraform with any other project ID (no matter where it comes from), it would stop because the project ID is not allowed.

[Gnoale]

#3099

[Gnoale]

We will improve logging #3099 (comment)