Skip to content

Fix CWAgent container insights - #2249

Merged
sky333999 merged 7 commits into
mainfrom
mcommey/OtelContainerInsight
Aug 21, 2026
Merged

sky333999 merged 7 commits into
mainfrom
mcommey/OtelContainerInsight

Conversation

@Paamicky

Copy link
Copy Markdown
Contributor

Description of the issue

While bringing OTEL Container Insights (OCI) into the CloudWatch Agent, several pieces of functionality present in the Helm implementation were dropped or omitted, leaving the agent-generated config diverged from the chart. Most notably: OCI logs were never delivered to CloudWatch Logs because the log-group/log-stream metadata was missing; cluster-scoped metrics (apiserver, kube-state-metrics) carried the scraper node's identity. Attributes limits of <150 guaranteed to the Zeus endpoint was broken as well as Batching.

Description of changes

Realigns the agent's OTEL Container Insights pipelines with the Helm implementation and fixes the dropped/omitted functionality:

  • OCI logs delivery: route CI application/host logs through the shared logs pipeline, which supplies the log-group/log-stream metadata that was missing (logs now reach CloudWatch Logs), including host-log gating (host logs omit
    service.name/deployment.environment.name).
  • schema_url reordering: move the schema_url clear into the shared metrics + logs pipelines (after the re-stamping
    resourcedetection/opentelemetry) and remove the redundant per-CI-pipeline clears.
  • Cluster-metric identity suppression: for cluster-scoped metrics (apiserver, kube-state-metrics), suppress the scraper node's host.*, cloud.availability_zone, and ec2.tag.* so series stay stable across scraper reschedules.
  • Attribute limit: add awsattributelimit (150-cap) on the shared metrics pipeline to stay under the CloudWatch OTLP backend limit.
  • Batching: set the shared metrics batch to 1000/10s which affects all OTLP metrics
  • resourcedetection: unify the CI resourcedetection definition (detectors: [eks, env, ec2], timeout: 2s) across pipelines.

All changes are gated to EKS OTEL Container Insights; ECS, host metrics, and non-CI pipelines are unaffected. Goldens regenerated.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

  • Unit: go build ./... and go test ./translator/... pass (goldens updated); gofmt clean.
  • End-to-end on EKS (cluster mcommey-otelci-final, agent built from this branch, deployed via operator):
  • OCI logs now reach CloudWatch with the correct and verified attributes
  • Cluster metrics (apiserver/KSM via Query Studio): no host., no cloud.availability_zone, no ec2.tag.; node metrics retain node identity.
  • Metrics export cleanly

Requirements

  1. Run make fmt and make fmt-sh
  2. Run make lint

@Paamicky
Paamicky requested a review from movence August 13, 2026 18:04
@Paamicky Paamicky self-assigned this Aug 13, 2026
@Paamicky Paamicky added the ready for testing Indicates this PR is ready for integration tests to run label Aug 13, 2026
@Paamicky
Paamicky requested a review from sky333999 August 17, 2026 18:04
@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from aa5093b to 4be3af9 Compare August 17, 2026 19:08
@Paamicky
Paamicky marked this pull request as ready for review August 17, 2026 19:08
@Paamicky
Paamicky requested a review from a team as a code owner August 17, 2026 19:08
@Paamicky Paamicky changed the title Fix CWAgent container infights Fix CWAgent container insights Aug 17, 2026
@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from bae8c04 to 452caf0 Compare August 17, 2026 20:05
@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from 452caf0 to a0cb8bf Compare August 17, 2026 20:18
@github-actions

github-actions Bot commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Binary Size Report

linux/amd64

Binary PR vs main (87efdff) vs v1.300072.0
amazon-cloudwatch-agent 194.7 MB ${\color{red}▲}$ +1.4 MB (+0.7%) ${\color{red}▲}$ +1.4 MB (+0.7%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B +0 B
config-downloader 2.0 MB +0 B +0 B
config-translator 2.0 MB +0 B +0 B
start-amazon-cloudwatch-agent 2.5 MB +0 B +0 B
workload-discovery 3.0 MB +0 B +0 B
Total 206.3 MB ${\color{red}▲}$ +1.4 MB (+0.7%) ${\color{red}▲}$ +1.4 MB (+0.7%)
linux/amd64 amazon-cloudwatch-agent (last 9 main commits + this PR)

 195 ┤                                        
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
 193 ┤▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ███ 
 MB  └────────────────────────────────────────
      589a727             v1.300072.0     PR

Notable changes:

  • +1.4 MB at PR
linux/arm64
Binary PR vs main (87efdff) vs v1.300072.0
amazon-cloudwatch-agent 178.5 MB ${\color{red}▲}$ +1.4 MB (+0.8%) ${\color{red}▲}$ +1.4 MB (+0.8%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B +0 B
config-downloader 2.0 MB +0 B +0 B
config-translator 2.0 MB +0 B +0 B
start-amazon-cloudwatch-agent 2.4 MB +0 B +0 B
workload-discovery 2.9 MB +0 B +0 B
Total 189.8 MB ${\color{red}▲}$ +1.4 MB (+0.8%) ${\color{red}▲}$ +1.4 MB (+0.8%)
windows/amd64
Binary PR vs main (87efdff) vs v1.300072.0
amazon-cloudwatch-agent.exe 196.1 MB ${\color{red}▲}$ +8.7 KB (+0.0%) ${\color{red}▲}$ +8.7 KB (+0.0%)
amazon-cloudwatch-agent-config-wizard.exe 2.1 MB +0 B +0 B
config-downloader.exe 2.1 MB +0 B +0 B
config-translator.exe 2.1 MB +0 B +0 B
start-amazon-cloudwatch-agent.exe 2.3 MB +0 B +0 B
workload-discovery.exe 2.9 MB +0 B +0 B
Total 207.7 MB ${\color{red}▲}$ +8.7 KB (+0.0%) ${\color{red}▲}$ +8.7 KB (+0.0%)
Investigating size changes

Use go-size-analyzer to compare binaries:

GOEXPERIMENT=jsonv2 go install github.com/Zxilly/go-size-analyzer/cmd/gsa@latest
gsa diff --old <baseline-binary> --new <new-binary>

@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from 839d539 to 0fe7082 Compare August 18, 2026 15:38
@Paamicky
Paamicky requested a review from jefchien August 18, 2026 16:05
@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from 0fe7082 to 553469b Compare August 18, 2026 19:14
Comment thread .github/workflows/e2e-test.yml Resolved
@sky333999
sky333999 merged commit 51fb106 into main Aug 21, 2026
468 of 472 checks passed
@sky333999
sky333999 deleted the mcommey/OtelContainerInsight branch August 21, 2026 15:09
sky333999 added a commit that referenced this pull request Sep 3, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 3, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 16, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 18, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 18, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 21, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants