Skip to content

Gate cluster-name resolution behind KubernetesMode - #2259

Merged
Paamicky merged 5 commits into
mainfrom
mcommey/ec2describe-fix
Aug 28, 2026
Merged

Paamicky merged 5 commits into
mainfrom
mcommey/ec2describe-fix

Conversation

@Paamicky

Copy link
Copy Markdown
Contributor

Description of the issue

When the agent can't reach the EC2 API (e.g. a private VPC), starting it with an OpenTelemetry config hangs. The agent calls EC2 for instance tags to resolve a cluster name on non-Kubernetes hosts, and that call retries until config setup stalls.

Description of changes

  • Gate cluster-name resolution behind the existing KubernetesMode() != "" check in the metrics, logs, and traces translators
  • Non-K8s hosts: GetClusterName/ec2:DescribeTags is no longer called during translation, so no hang.
  • Intentional change: on non-K8s hosts, an explicit cluster_name no longer emits k8s.cluster.name.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

  • Added TestClusterNameSkippedNonK8s which asserts set_cluster_name is absent when KubernetesMode="" and updated existing cluster-name test to run under EKS mode. Regenerated 2 goldens.
  • End to end test with defective agent
[ec2-user@ip-172-31-85-207 ~]$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/tmp/dbi.json
****** processing amazon-cloudwatch-agent ******
Starting config-downloader, this will map back to a call to amazon-cloudwatch-agent
Executing /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent with arguments: [config-downloader -output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d -config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml -multi-config default -mode ec2 -download-source file:/tmp/dbi.json]Got Home directory: /root
I! Set home dir Linux: /root
I! SDKRegionWithCredsMap region:
I! Trying to detect region from ec2
D! [EC2] Found active network interface
I! imds retry client will retry 1 times
Start configuration validation...
2026/08/26 20:17:53 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_dbi.json.tmp ...
2026/08/26 20:17:53 I! Valid Json input schema.
2026/08/26 20:19:54 Describe EC2 Tag Fail. Will retry the request: RequestError: send request failed
caused by: Post "https://ec2.us-east-1.amazonaws.com/": dial tcp 240.0.0.1:443: i/o timeout
2026/08/26 20:19:54 W! It is the 0 time, going to sleep 200ms before retrying.
  • End to end test with the fix
[ec2-user@ip-172-31-85-207 ~]$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/tmp/dbi.json -s
****** processing amazon-cloudwatch-agent ******
Starting config-downloader, this will map back to a call to amazon-cloudwatch-agent
Executing /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent with arguments: [config-downloader -config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml -multi-config default -mode ec2 -download-source file:/tmp/dbi.json -output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d]Got Home directory: /root
I! Set home dir Linux: /root
I! SDKRegionWithCredsMap region:
I! Trying to detect region from ec2
D! [EC2] Found active network interface
I! imds retry client will retry 1 times
Start configuration validation...
2026/08/26 21:00:14 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_dbi.json.tmp ...
2026/08/26 21:00:14 I! Valid Json input schema.
2026/08/26 21:00:14 Configuration validation first phase succeeded
Starting config-translator, this will map back to a call to amazon-cloudwatch-agent
Executing /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent with arguments: [config-translator -config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml -multi-config default -input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d -output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml -mode ec2]D! [EC2] Found active network interface
I! imds retry client will retry 1 times
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
[ec2-user@ip-172-31-85-207 ~]$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
{
  "status": "running",
  "starttime": "2026-08-26T21:00:15+00:00",
  "configstatus": "configured",
  "version": "1.300071.0b1720"
}
[ec2-user@ip-172-31-85-207 ~]$

Requirements

  1. Run make fmt and make fmt-sh
  2. Run make lint

@Paamicky
Paamicky marked this pull request as ready for review August 26, 2026 21:19
@Paamicky
Paamicky requested a review from a team as a code owner August 26, 2026 21:19
@Paamicky Paamicky added the ready for testing Indicates this PR is ready for integration tests to run label Aug 26, 2026
@Paamicky
Paamicky force-pushed the mcommey/ec2describe-fix branch from ec23921 to 8a4d05d Compare August 26, 2026 22:01
@github-actions

github-actions Bot commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Binary Size Report

linux/amd64

Binary PR vs main (a2fbeb6) vs v1.300072.0
amazon-cloudwatch-agent 194.8 MB +0 B ${\color{red}▲}$ +1.5 MB (+0.8%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B +0 B
config-downloader 2.0 MB +0 B +0 B
config-translator 2.0 MB +0 B +0 B
start-amazon-cloudwatch-agent 2.5 MB +0 B +0 B
workload-discovery 3.0 MB +0 B +0 B
Total 206.4 MB +0 B ${\color{red}▲}$ +1.5 MB (+0.7%)
linux/amd64 amazon-cloudwatch-agent (last 9 main commits + this PR)

 195 ┤                                        
     ┤            ▄▄▄ ▄▄▄ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
 193 ┤▄▄▄ ▄▄▄ ▄▄▄ ███ ███ ███ ███ ███ ███ ███ 
 MB  └────────────────────────────────────────
      9884add                             PR

Notable changes:

linux/arm64
Binary PR vs main (a2fbeb6) vs v1.300072.0
amazon-cloudwatch-agent 178.6 MB +0 B ${\color{red}▲}$ +1.5 MB (+0.9%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B +0 B
config-downloader 2.0 MB +0 B +0 B
config-translator 2.0 MB +0 B +0 B
start-amazon-cloudwatch-agent 2.4 MB +0 B +0 B
workload-discovery 2.9 MB +0 B +0 B
Total 189.9 MB +0 B ${\color{red}▲}$ +1.5 MB (+0.8%)
windows/amd64
Binary PR vs main (a2fbeb6) vs v1.300072.0
amazon-cloudwatch-agent.exe 197.6 MB ${\color{red}▲}$ +6.7 KB (+0.0%) ${\color{red}▲}$ +1.5 MB (+0.8%)
amazon-cloudwatch-agent-config-wizard.exe 2.1 MB +0 B +0 B
config-downloader.exe 2.1 MB +0 B +0 B
config-translator.exe 2.1 MB +0 B +0 B
start-amazon-cloudwatch-agent.exe 2.3 MB +0 B +0 B
workload-discovery.exe 2.9 MB +0 B ${\color{red}▲}$ +512 B (+0.0%)
Total 209.3 MB ${\color{red}▲}$ +6.7 KB (+0.0%) ${\color{red}▲}$ +1.5 MB (+0.7%)
Investigating size changes

Use go-size-analyzer to compare binaries:

GOEXPERIMENT=jsonv2 go install github.com/Zxilly/go-size-analyzer/cmd/gsa@latest
gsa diff --old <baseline-binary> --new <new-binary>

@Paamicky
Paamicky force-pushed the mcommey/ec2describe-fix branch from 9dd81fa to 12a977b Compare August 27, 2026 14:20
@Paamicky
Paamicky merged commit d2b097a into main Aug 28, 2026
458 of 472 checks passed
@Paamicky
Paamicky deleted the mcommey/ec2describe-fix branch August 28, 2026 17:03
sky333999 added a commit that referenced this pull request Sep 3, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 3, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 16, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 18, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 18, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 21, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants