Proposal: State Encryption

[StephanHCB]

Suggest an existing issue in legacy Terraform to fix in OpenTF.

I already made a PR for full client side state encryption for all backends except the extended backends 2 years ago.

I would be more than happy to either submit it as a stale PR, or just make a PR. Whichever works better for you. I actually have a local patch based on 1.5.5 which I'd just need to commit.

Technical description / Proposal - see comments below - to be added here when at least some consensus is achieved and people have had a chance to comment.

Draft PR see in the comments below, but note that the implementation will likely differ from this PR. I have tried it extensively with state stored on Azure storage accounts, and it is known to work in this scenario. The PR includes documentation as an experimental feature and extensive unit tests.

I would love not to have to maintain an internal fork for this any more :)

[cube2222]

@StephanHCB Hey!

We'd appreciate it a lot if you could resubmit this as an RFC on the OpenTF project, once the repo is open, as it seems you've thought this through a lot.

I did a rudimentary working PoC of this during the last weekend as well with pbkdf2 and aead, to better understand the problem domain.

Anyway, one idea we've had and discussed, was possibly using SOPS under the hood, as it already solves the use case of encrypting json/yaml files really well (and what is a state file if not that). The advantages would be

1. encrypting only the sensitive values has a lower chance of breaking state backends that actually try to parse your state (and I suppose there might be some that do), as opposed to sending an encrypted blob of the whole json
2. out of the box supporting all encryption backends that sops integrates with.

We could encrypt plan files, too, for safer storage if you're only applying at a later time.

I'd love to hear your thoughts on that.

[StephanHCB]

We'd appreciate it a lot if you could resubmit this as an RFC on the OpenTF project, once the repo is open, as it seems you've thought this through a lot.

Will do. I am happy this issue with terraform's security will finally get addressed.

I did a rudimentary working PoC of this during the last weekend as well with pbkdf2 and aead, to better understand the problem domain.

There are indeed a number of lifecycle events to consider:

• initial encryption
• key rotation
• moving away from state encryption
• switching state encryption providers

Then there are the manual operations that need access to the state, where tf state pull gets new, rare, edge cases:

• state doctoring in the presence of a truncated state blob

Anyway, one idea we've had and discussed, was possibly using SOPS under the hood, as it already solves the use case of encrypting json/yaml files really well (and what is a state file if not that). The advantages would be

1. encrypting only the sensitive values has a lower chance of breaking state backends that actually try to parse your state (and I suppose there might be some that do), as opposed to sending an encrypted blob of the whole json

2. out of the box supporting all encryption backends that sops integrates with.

Generally, my solution supports pluggable encryption backends via a simple interface they have to implement, one implementation could very easily be one using SOPS. You select which encryption backend you want via a pair of environment variables, one for the current encryption configuration and one fallback used only for decryption (for key rotation, switching encryption backends, or altogether moving away from state encryption).

SOPS was brought up on the original issue. You've summarized the upsides quite well, the biggest downside was in my reply back then (hashicorp/terraform#9556 (comment)):

"It's certainly a lot more complex than what this PR does, at least if you want to do it in a secure fashion. TF providers keep adding fields in the state, and if you miss one that's sensitive, you are again leaking credentials. This would have a high probability of giving a false sense of security IMHO, which is often actually very bad for security."

Where I work, this was one of the reasons we decided to encrypt the entire state.

We could encrypt plan files, too, for safer storage if you're only applying at a later time.

Good point. That is especially relevant for shipping / storing / processing plan files "in the cloud". On premises, there is less point to it, because you have to have access to both the encryption and the decryption keys anyway for every tf run.

[0x91]

We could encrypt plan files, too,
for safer storage if you're only applying at a later time.

This is only somewhat related but with plan files it's also worthwhile to look at solutions that prevent tampering as gaining the ability to modify a plan file is quite problematic as something executing OpenTF will generally have highly privileged access.

I'm not a cryptographer, but using AEAD should satisfy both the requirement to prevent sensitive values and ensuring the integrity of plans before applying them?

[cube2222]

Alright, now that we have the repo in place I think we can continue the discussion on this.

@StephanHCB would you be able to rewrite the issue in a way that doesn't link to the Terraform repo? We're trying to keep any issues here self-contained and generally detach from the Terraform issue tracker.

Then I think we can discuss this a bit more here and possibly get an RFC going?

[roni-frantchi]

Hey @StephanHCB thanks for submitting!

To help us maintain a clear separation between opentf and hashicorp's offerings, we're asking that people describe issues that are in other repositories rather than linking those directly.
I've thus scrubbed out any links to said issues.

We'd love to get an RFC going of course.

It is also worth mentioning that as an author of a PR to legacy your are permitted under legacy Terraform CLA to resubmit your code elsewhere as well.

Thanks!

[ekini]

For additional reference, Pulumi uses https://gocloud.dev/howto/secrets/ under the hood to provide this functionality. That allows to easily cover all major clouds and even keep a CMK in a different cloud to the state file.

[StephanHCB]

Alright, let's get started. Let's talk motivation and requirements.

The Case for State Encryption

OpenTF state contains lots of sensitive information.

The most obvious example are credentials such as primary access keys to storage, but even ignoring any credentials state often includes a full map of your network, including every VM, kubernetes cluster, database, etc. That is a treasure trove for an attacker who wishes to orient themselves in your private network.

Unlike runtime information processed by OpenTF, which only lives in memory and is discarded when the tf run ends, state must by its very nature be persisted. In large installations, tf state is not (just) stored in local files because multiple users need access to it. Remote state backend options include simple storage (such as storage accounts, various databases, ...), meaning these storage options do not "understand" the state, but there are also extended backends, which do wish to gain information from state. The persistent nature and (often) cloud storage of state increases the risk of it falling into the wrong hands.

Large corporations and financial institutions have compliance requirements for storage of sensitive information. One frequent requirement is encryption at rest using a customer managed key. From a security perspective, ideally such a key is not even made available to a cloud provider storing the state, for if the key is available and encryption is transparent, encryption at rest reduces to little more than a proxy for RBAC.

Requirements for State Encryption

Ability to encrypt

• local state files
• remote state
• plan files

using a key supplied to terraform at runtime, either directly or through a key retrieval mechanism.

Off by default: Default to no encryption, thus maintaining full compatibility with existing tf versions.

The following lifecycle events need to be supported

• initial encryption
• moving away from state encryption
• key rotation
• switching state encryption methods

The following manual operations need to be supported

• state pull with/without decryption
• state push
• state doctoring in the presence of a truncated state blob

Encryption methods should be pluggable to be able to support

• incremental feature implementation (e.g. have a full encryption option out quickly and later add other methods)
• full state encryption
• partial state encryption
• easy localized implementation changes to a single method
• key retrieval from typical vault products
• forks that just add a key retrieval / encryption method (we will never be able to support everything out there, especially some in-house solutions)

Partial state encryption should retain the structure of the state json and just mask sensitive values, so that the extended backends can still work with the state but not see any credentials, while full state encryption helps prevent information disclosure from state.

Encryption should be transparent

Encrypted state needs to be json, because many of the current backend implementations expect it to be. How much the json resembles the structure of the unencrypted state will depend on the encryption method.

Note: asymmetric encryption algorithms are of little use given the way tf needs to both read and write state in the same run for many operations. You will most likely always need a symmetric session key, but where that key comes from is a different matter. It is entirely feasible to have a state encryption method that obtains its key from some kind of key vault.

Any comments are welcome.

[StephanHCB]

Anyway, one idea we've had and discussed, was possibly using SOPS under the hood, as it already solves the use case of encrypting json/yaml files really well (and what is a state file if not that).

SOPS seems to fulfill a significant portion of the above requirements, so I think we should take a closer look at it.

I haven't worked with SOPS before, but just looking at their github repo I gather:

• License is Mozilla 2.0, so that's good
• It pulls in a pretty significant dependency tree - is there a process for vetting dependency licenses?
• It exports a stable library interface for decryption, but not for encryption as far as I can tell. The idea seems to be that you use their editor to write config files, and use their library for decrypting them. This does not match our use case, do we have a good solution to that?

Do we already have an idea how we're going to decide which field values need encrypting? That seems to me to be the biggest hurdle we'd have to overcome, to not impose on the community a need to maintain a huge list of fields to encrypt.

[StephanHCB]

For additional reference, Pulumi uses https://gocloud.dev/howto/secrets/ under the hood to provide this functionality. That allows to easily cover all major clouds and even keep a CMK in a different cloud to the state file.

This is also a very promising candidate, at least for the full state encryption use case.

• License is Apache 2.0
• Pulls in a pretty significant dependency tree - same question as for SOPS: how to vet them?
• Supports GCP Key Mgmt, AWS Key Mgmt, Azure Key Vault, Hashicorp Vault and local keys
• Used by Pulumi for a very similar purpose, so it's real world tested

Personally this will be the first library I'll look at for a prototype implementation.

[StephanHCB]

How should state encryption be configured?

In my previous implementation, I introduced two environment variables:

• TF_REMOTE_STATE_ENCRYPTION: configuration used for encryption and decryption
• TF_REMOTE_STATE_DECRYPTION_FALLBACK: fallback configuration for decryption, tried if decryption fails with the first choice

You set each of these to a json document with two fields

• implementation: select a state encryption method by name
• parameters: configure the state encryption method

I used environment variables because those do not show up in process lists, and also can be centrally provided for all users on a machine. Also, works with terraform wrappers such as terragrunt with no code changes, because environment variables are "just present".

Turns out this is also a good choice for CI systems, works with Spacelift for example, you can build it so the key never leaves your local private worker deployment.

I will create a draft PR with my old code for everyone to look at, hopefully I can get around to that on Monday, although with the additional requirements I doubt the end result will actually include much of that code other than the interface for pluggable state encryption methods.

[cube2222]

It exports a stable library interface for decryption, but not for encryption as far as I can tell. The idea seems to be that you use their editor to write config files, and use their library for decrypting them. This does not match our use case, do we have a good solution to that?

Yeah, I think SOPS would need to be used by shelling out to the CLI, not great.

I do agree that the Go Cloud library looks very promising here, esp. since Pulumi is using them already.

Do we already have an idea how we're going to decide which field values need encrypting? That seems to me to be the biggest hurdle we'd have to overcome, to not impose on the community a need to maintain a huge list of fields to encrypt.

The schema contains info about which fields are sensitive, it's currently also included in the plan file, so in other words the info should already be there.

How should state encryption be configured?

That's a good question! I'm not sure, open to hearing more opinions here. Environment variables are the obvious solution, but we could also think about .terraformrc or even a custom file in the .terraform directory in the pwd (similar to how "current workspace" is stored). Obviously that would most likely not include the secret itself, if there is one (for the most simple implementation that involves a passphrase), just config details like the KMS key id. Secrets I believe shouldn't be stored in any plain-text config files.

I will create a draft PR with my old code for everyone to look at, hopefully I can get around to that on Monday, although with the additional requirements I doubt the end result will actually include much of that code other than the interface for pluggable state encryption methods.

Sounds good, and thanks a lot for this! Btw. no rush, I don't think we'll be able to merge this prior to the 1.6 stable release, which is now our main priority. A PoC would however be great grounds for further discussion.