[Bug] Idempotency issue while having multiple inputs using elasticstack_fleet_package_policy

[mathieu-letribot]

Describe the bug
The bug occurs when I tried to manage Kubernetes package policy using the new elasticstack_fleet_package_policy resource. With kubernetes, there are multiple intputs to configure. The elasticstack_fleet_package_policy is working fine (it can create/update/delete the package policy), but the plan is always showing some differences in the inputs (and not in the same order). I suppose that inputs order in terraform code are not the same as the input order returned by the API.

Example:

After creating the policy, I execute then plan and can see some differences:

  # module.ec.elasticstack_fleet_package_policy.kubernetes will be updated in-place
  ~ resource "elasticstack_fleet_package_policy" "kubernetes" {
        id              = "555443a0-b129-4cf9-8124-9c31b5182cef"
        name            = "kubernetes"
        # (7 unchanged attributes hidden)

      ~ input {
          ~ input_id     = "kube-scheduler-kubernetes/metrics" -> "kube-apiserver-kubernetes/metrics"
          ~ streams_json = (sensitive value)
            # (1 unchanged attribute hidden)
        }
      ~ input {
          ~ input_id     = "events-kubernetes/metrics" -> "kube-state-metrics-kubernetes/metrics"
          ~ streams_json = (sensitive value)
            # (1 unchanged attribute hidden)
        }
      ~ input {
          ~ input_id     = "audit-logs-filestream" -> "kube-proxy-kubernetes/metrics"
          ~ streams_json = (sensitive value)
            # (1 unchanged attribute hidden)
        }
      ~ input {
          ~ enabled      = true -> false
          ~ input_id     = "kube-state-metrics-kubernetes/metrics" -> "kube-controller-manager-kubernetes/metrics"
          ~ streams_json = (sensitive value)
        }
      ~ input {
          ~ enabled      = false -> true
          ~ input_id     = "kube-apiserver-kubernetes/metrics" -> "container-logs-filestream"
          ~ streams_json = (sensitive value)
        }
      ~ input {
          ~ enabled      = false -> true
          ~ input_id     = "kube-proxy-kubernetes/metrics" -> "events-kubernetes/metrics"
          ~ streams_json = (sensitive value)
        }
      ~ input {
          ~ input_id     = "kube-controller-manager-kubernetes/metrics" -> "kube-scheduler-kubernetes/metrics"
          ~ streams_json = (sensitive value)
            # (1 unchanged attribute hidden)
        }
      ~ input {
          ~ enabled      = true -> false
          ~ input_id     = "container-logs-filestream" -> "audit-logs-filestream"
          ~ streams_json = (sensitive value)
        }

        # (1 unchanged block hidden)
    }

Executing the plan again will show changes again but in different order:

  # module.ec.elasticstack_fleet_package_policy.kubernetes will be updated in-place
  ~ resource "elasticstack_fleet_package_policy" "kubernetes" {
        id              = "555443a0-b129-4cf9-8124-9c31b5182cef"
        name            = "kubernetes"
        # (7 unchanged attributes hidden)

      ~ input {
          ~ enabled      = false -> true
          ~ input_id     = "kube-controller-manager-kubernetes/metrics" -> "kubelet-kubernetes/metrics"
          ~ streams_json = (sensitive value)
        }
      ~ input {
          ~ enabled      = true -> false
          ~ input_id     = "events-kubernetes/metrics" -> "kube-apiserver-kubernetes/metrics"
          ~ streams_json = (sensitive value)
        }
      ~ input {
          ~ input_id     = "container-logs-filestream" -> "kube-state-metrics-kubernetes/metrics"
          ~ streams_json = (sensitive value)
            # (1 unchanged attribute hidden)
        }
      ~ input {
          ~ input_id     = "audit-logs-filestream" -> "kube-proxy-kubernetes/metrics"
          ~ streams_json = (sensitive value)
            # (1 unchanged attribute hidden)
        }
      ~ input {
          ~ enabled      = true -> false
          ~ input_id     = "kubelet-kubernetes/metrics" -> "kube-controller-manager-kubernetes/metrics"
          ~ streams_json = (sensitive value)
        }
      ~ input {
          ~ enabled      = false -> true
          ~ input_id     = "kube-apiserver-kubernetes/metrics" -> "container-logs-filestream"
          ~ streams_json = (sensitive value)
        }
      ~ input {
          ~ enabled      = false -> true
          ~ input_id     = "kube-proxy-kubernetes/metrics" -> "events-kubernetes/metrics"
          ~ streams_json = (sensitive value)
        }
      ~ input {
          ~ enabled      = true -> false
          ~ input_id     = "kube-state-metrics-kubernetes/metrics" -> "audit-logs-filestream"
          ~ streams_json = (sensitive value)
        }

        # (1 unchanged block hidden)
    }

To Reproduce
Steps to reproduce the behavior:

1. Create a package policy with multiple inputs

resource "elasticstack_fleet_package_policy" "kubernetes" {
  name            = "kubernetes"
  namespace       = "default"
  description     = "Kubernetes package policy"
  agent_policy_id = elasticstack_fleet_agent_policy.kubernetes.policy_id
  package_name    = elasticstack_fleet_package.kubernetes.name
  package_version = elasticstack_fleet_package.kubernetes.version
  force           = true

  input {
    enabled  = true
    input_id = "kubelet-kubernetes/metrics"
    streams_json = jsonencode({...})
  }
  input {
    enabled  = false
    input_id = "kube-apiserver-kubernetes/metrics"
    streams_json = jsonencode({...})
  }
  input {
    enabled  = true
    input_id = "kube-state-metrics-kubernetes/metrics"
    streams_json = jsonencode({...})
  }
  input {
    enabled  = false
    input_id = "kube-proxy-kubernetes/metrics"
    streams_json = jsonencode({...})
  }
  input {
    enabled  = false
    input_id = "kube-controller-manager-kubernetes/metrics"
    streams_json = jsonencode({...})
  }
  input {
    enabled  = true
    input_id = "container-logs-filestream"
    streams_json = jsonencode({...})
  }
  input {
    enabled  = true
    input_id = "events-kubernetes/metrics"
    streams_json = jsonencode({...})
  }
  input {
    enabled  = false
    input_id = "kube-scheduler-kubernetes/metrics"
    streams_json = jsonencode({...})
  }
  input {
    enabled  = false
    input_id = "audit-logs-filestream"
    streams_json = jsonencode({...})
  }

}

2. terraform plan will then show some differences

Expected behavior
I except this resource to be idempotent to avoid confusion while applying some (real) changes.

Versions :

• OS: Mac OS Sonoma (M2)
• Terraform Version 1.6.3
• Provider version v0.10.0
• Elasticsearch Version 8.11.0

[markjandejong]

I'm no Go expert but is it because Inputs is defined as a map instead of a list here?

terraform-provider-elasticstack/generated/fleet/fleet.gen.go

Lines 664 to 677 in 499eead

	type PackagePolicy struct {
	Description *string `json:"description,omitempty"`
	Enabled *bool `json:"enabled,omitempty"`
	Id string `json:"id"`
	Inputs map[string]PackagePolicyInput `json:"inputs"`
	Name string `json:"name"`
	Namespace *string `json:"namespace,omitempty"`
	// Deprecated:
	OutputId *string `json:"output_id,omitempty"`
	Package *PackagePolicyPackageInfo `json:"package,omitempty"`
	PolicyId *string `json:"policy_id,omitempty"`
	Revision float32 `json:"revision"`
	Vars *map[string]interface{} `json:"vars,omitempty"`
	}

[markjandejong]

@taylor-swanson , can you please look at this when you have a moment? I'm asking based on previous requests to you on this resource. ;) Thanks!

[taylor-swanson]

This is seems to be caused by a map vs list/slice issue. The Fleet API hands back a map of the inputs, but the Terraform schema defines the inputs as a list. The order of maps is undefined in Go and the resource populates the list of inputs based on the "order" of the map, which is more or less random every time.

In the short term, I could sort the inputs to match the order of the existing list (if applicable), and place new entries at the end.

In the long term, the provider was recently updated to support protocol v6, which provides much better schema support for map structures with nested objects/attributes. Unfortunately, this would mean a breaking change for the schema, but would better represent the data we receive from Fleet.

[markjandejong]

Thanks @taylor-swanson ! That would be fantastic! The resource is otherwise not useful in its current state.

[breml]

@taylor-swanson , @tobio I am affected by this bug as well so thank you for the quick fix. Do you mind to ship a new release. This would be highly appreciated.

[markjandejong]

@breml, You can build the module locally following the build instructions here. Be forewarned that the new module will require some manual state file intervention if using fleet_package* resources as the resource names have changed to fleet_integration*.

[breml]

@dejongm building locally worked and I was able to successfully test the new version of the provider. But in order to ship to code internally, I need a proper release.

[tobio]

@breml we'll get a release out here shortly

[psypuff]

@tobio this seems to still happen. I was initially using TF 1.6.6 and elasticstack 0.11.6 but later also tested 0.11.2 (couldn't get earlier versions to work due to other issues). Later also tried TF 1.9.5 but ran into the same behavior.
Worth mentioning I was importing existing resources and tried matching the existing inputs order. Though it seems the order randomly changes with each tf apply execution.