do not store tailscale_tailnet_key's sensitive information in terraform state.

[james-lawrence]

Is your feature request related to a problem? Please describe.
terraform isn't a secret storage system. sensitive information should ideally not be stored in it. this is related to #144 and tailscale/tailscale#3243. I understand this might not be feasible (depends on the backend for the apis involved among other things). just thought I'd point this out.

Describe the solution you'd like
ideally a way to fetch the sensitive information at runtime and only store a checksum in the terraform state for verifying the value.

[DentonGentry]

There are several Hashicorp vault providers which can handle tailscale key state:
https://github.com/bloominlabs/vault-plugin-secrets-tailscale
https://github.com/davidsbond/vault-plugin-tailscale

The most recent updates in tailscale/tailscale#3243 describe the intended solution: an OAuth 2.0 Client Credential flow with a long-lived credential. That credential can be used to issue short lifetime access tokens which would presumably not be stored anywhere.

[dgivens]

I tried using vault-plugin-secrets-tailscale, but it didn't work out for me. When a Vault lease is revoked or expires, the secrets associated with the lease are revoked.

I am using Nomad to run Terraform. When it runs a job, the job is given a Vault token that is revoked upon completion of the job. In the instance of Terraform being run as a batch job, when I used the Tailscale Vault plugin to generate a preauth key that would be inserted into the instance userdata, it ended up being revoked before cloud-init could be run.

The only way I found to work around it was to switch to this provider.

[james-lawrence]

@DentonGentry the existence of other solution doesn't negate the problems with this one.

[DentonGentry]

@DentonGentry the existence of other solution doesn't negate the problems with this one.

If not pulling secrets from Hashicorp vault or similar solutions, then what would you propose?

[davidsbond]

ideally a way to fetch the sensitive information at runtime and only store a checksum in the terraform state for verifying the value.

I think part of the problem with this solution is that the Tailscale API only returns the secret key data once on creation and it cannot be retrieved again. So that secret value has to go in the state for now. You're welcome to try out the Vault plugin that I implemented https://github.com/davidsbond/vault-plugin-tailscale which is admittedly very lightweight, but I'd be happy to handle contributions or feature requests on it.

[james-lawrence]

If not pulling secrets from Hashicorp vault or similar solutions, then what would you propose?

I proposed a solution to the problem already in the issue. as I said I understand technical underpinning of tailscales credential management make solving this problematic atm. but the terraform integration is problematic as it opens up sensitive values to be exposed inadvertently. alternative options don't absolve this terraform resource from being problematic in that regard is all I was asserting.

I mainly opened this so this problem is discoverable for others.

[dgivens]

To be fair, this is a known issue with Terraform. It's very explicitly stated in the Terraform docs about avoiding secrets in the state. It's not something specific to the tailscale provider. There are mitigation steps you can take, such as restricting access to the location where states are stored, ensuring the states are kept in a location that encrypts the data at rest, etc.

For my implementation, I decided to keep the Tailscale API creds in a simple kv Vault secret, which the Vault Terraform provider reads, then uses to configure the Tailscale provider. The Tailscale keys created by Terraform are single use and ephemeral, so they aren't really of any use to anyone who might get access to the state once they've been used.

provider "vault" {}

data "vault_kv_secret_v2" "tailscale" {
  mount = "secret"
  name  = "tailscale"
}

locals {
  tailscale_data = jsondecode(data.vault_kv_secret_v2.tailscale.data_json)
}

provider "tailscale" {
  api_key = local.tailscale_data.api_key
  tailnet = local.tailscale_data.tailnet
}

resource "tailscale_tailnet_key" "foo" {
  reusable      = false
  ephemeral     = true
  preauthorized = true
  tags = [
    "tag:foo"
  ]
}

[james-lawrence]

@dgivens it very much is specific to this provider if this provider is doing the exact thing terraform says not to do. ;)

[davidsbond]

The terraform SDK does have a helper for modifying the value before it enters the state:

https://github.com/hashicorp/terraform-plugin-sdk/blob/3495894b26ae61cf11da2669170ff9d1c655896b/helper/schema/schema.go#L204

However, I think it would cause issues for a reusable key on a subsequent apply. Unless you can keep obtaining the secret value from the API I'm not sure what else the provider can do to keep reusable keys working.

[dgivens]

@james-lawrence I can see your point of view, but it applies equally to the Vault provider as well. The first several paragraphs of the overview for that provider is very clear about the risk of secrets in state and plan files, including this highlighted block:

Important

Interacting with Vault from Terraform causes any secrets that you read and write to be persisted in both Terraform's state file and in any generated plan files. For any Terraform module that reads or writes Vault secrets, these files should be treated as sensitive and protected accordingly.

[james-lawrence]

@dgivens yes i don't use the vault provider for that reason. not sure what you're trying to get at here. if one person jumps off a bridge should you?

[kevcube]

@james-lawrence you're not addressing the fact that terraform is known to store secrets in state. Almost every provider has this "issue" with certain resources.

This provider is not particularly engineered to store these secrets, but they are an attribute of a resource - something that is always stored in terraform state. This is a requirement for them to be consumed by any other terraform code.

The only way to avoid this is for terraform to never set the value tailscale_tailnet_key.key. Which makes it useless, as it can only be read once.

Your solution of "fetch the sensitive information at runtime" from where? You're welcome to use terraform to write the key into vault or wherever else, but ultimately terraform needs to be able to read it from tailscale_tailnet_key.

Please re-read the official HashiCorp view on secrets in state and treat your state as if it is confidential. https://developer.hashicorp.com/terraform/language/state/sensitive-data

I don't see what you're describing as an issue with this provider.

[james-lawrence]

@kevcube you don't need to store sensitive information to validate sensitive data. you can store a checksum instead of the sensitive data; as I pointed out in the top level issue. this issue is to act as a warning to people using the provider and to maybe prod the developers to stop leaking secrets when it isn't necessary.

[kevcube]

@james-lawrence this package is not a validator, it is a provider. It provides the key after it is created. This is not at all a fault of this packages' developers, and this is not secrets leakage. Honestly: have you used terraform before this? And also, have you read the documentation I linked regarding secrets in terraform state?