Changing NOMAD_TOKEN causes no-op changes to create deployment (due to token accessor for MRD)

[regner]

Nomad version

nomad version
Nomad v1.10.4
BuildDate 2025-08-12T20:48:32Z
Revision 62b195aaa535b2159d215eaf89e6f4a455d6f686

Operating system and Environment details

Nomad cluster is a mix of Linux and Windows machines. When deploying jobs Nomad CLI is run from a Linux environment.

Issue

An execution of nomad job run example.hcl will create a new deployment when there is no change.

Reproduction steps

Create an example job spec, as far as I can tell nothing about it matters, example provided below. Then execute the following commands:

# Login to Nomad
nomad login
export NOMAD_TOKEN=#####

# Create the first deployment
nomad job run example.hcl

# As expected that will create a deployment, once that is completed continue

# This will return immediately and nothing will happen as there is no change 
nomad job run example.hcl

# Generate a new token
nomad login
export NOMAD_TOKEN=#####

# Deploy the job spec again
nomad job run example.hcl

# That will cause a new deployment even though it should not and there
# are not any changes

You can see in this screenshot that the Nomad UI even knows that there is no change between the two versions, yet there is a version bump and a new deployment.

Job file (if appropriate)

job "example-bad-things" {
  region      = "global"
  datacenters = ["dcspecial"]
  node_pool   = "specialpool"
  type        = "service"
  namespace   = "production"

  group "bad-things" {
    task "postgres" {
      driver = "docker"

      config {
        image = "postgres:17.6"
      }
      resources {
        cpu    = 500
        memory = 512
      }

      template {
        destination = "secrets/file.env"
        env         = true
        data        = <<EOF
PGDATA=/var/lib/postgresql/17/docker
POSTGRES_USER=grafana
POSTGRES_PASSWORD=badbadbadbad
POSTGRES_DB=grafana
EOF
      }
    }
  }
}

[regner]

I was initially running into this with nomad-pack and thought the issue was there, but as you can see above I can reproduce this with just the nomad CLI. I was in the middle of writing this issue up for the nomad-pack repo when I figured that out. >.<

[tgross]

Hi @regner! This is expected because we store the accessor ID for the token on the job. You can see this with the job inspect command:

$ nomad job inspect example | jq '.Job.NomadTokenID'
"390ca100-2651-216c-d12a-4073db5b7357"

The reason for this is that it's used as part of the Enterprise-only Multi-Region Deployments feature to ensure that the token used to submit the job is valid in every region. It doesn't look like it's used anywhere else. That you're seeing this in Community Edition is definitely annoying though, and it would be nice if we could do that conditionally. I'll mark this for roadmapping.

(Internal ref https://hashicorp.atlassian.net/browse/NMD-1006)

[regner]

It's super annoying when using nomad-pack with multiple job specs.

We have node pools specific to stateful data for example. Since node pool must be at the job level, a pack that wants to deploy to multiple node pools needs to generate multiple job specs. So our pack for Grafana has two job specs, one for Postgres and one for Grafana itself. If we make a change to Grafana and deploy that it forces a deployment of Postgres even though there was no change.

[regner]

Also Thank you for the explanation.

[vky5]

Hey @tgross Looked into this. Root cause is Job.SpecChanged() in nomad/structs/structs.go it does a reflect.DeepEqual on the whole job struct but never excludes NomadTokenID. diff.go already excludes that same field for the plan-diff display (#8773), this just never got applied here.

Fix: add NomadTokenID to SpecChanged's existing scrub list, next to Status, Version, ModifyIndex, etc. That function is also used in deploymentwatcher.handleRollbackValidity to decide if a rollback target is actually different from the current version, so this fixes that blind spot too.

One thing I can't check from the OSS side: does MRD ever rely on a no-op resubmit to refresh the stored NomadTokenID. Say after a token rotation with nothing else changed? If I exclude the field, that resubmit becomes a true no-op and the new token never gets saved. If that's not a real case, I'll open a PR.

[tgross]

Fix: add NomadTokenID to SpecChanged's existing scrub list

👍

One thing I can't check from the OSS side: does MRD ever rely on a no-op resubmit to refresh the stored NomadTokenID. Say after a token rotation with nothing else changed?

No, the deployment actually fails if you rotate the token before it's complete. The leader of a region has a temporary ACL token created when it becomes the leader (ref leader.go#L381-L383) but that token is only valid within a given region and for a single term, so we can't use it for cross-region work. So that's why we use the user's token, which has to be a global token. If that token is rotated out in the middle of the deployment, the leader of one region has no token it can use to driven the deployment forward in another region.

In retrospect, I don't love this design and we should instead give it a new token scoped to the deployment from the authoritative region (edit: tracking that in NMD-1687 internally now). But that doesn't block anything you're suggesting here.