Skip to content

Add KEDA and Karpenter metrics scrapping to CWAgent - #2250

Merged
Paamicky merged 5 commits into
mainfrom
mcommey/CI-keda-kapernter
Aug 21, 2026
Merged

Paamicky merged 5 commits into
mainfrom
mcommey/CI-keda-kapernter

Conversation

@Paamicky

Copy link
Copy Markdown
Contributor

Description of the issue

OTEL Container Insights supports now supports scrapping of KEDA and Karpenter cluster metrics in Helm Chart.
Customer should be able to configure and scrape those metrics on the CloudWatch Agent.

Description of changes

  • Added agent-side KEDA and Karpenter cluster-scoped pipelines (keda.yaml, karpenter.yaml), gated on
    opentelemetry.collect.container_insights.solutions.{keda,karpenter}.
  • Added solutions field in the cwagent schema.json for customers
  • They forward into the shared metrics/opentelemetry pipeline and use mark_cluster, so they inherit the
    shared host/AZ/ec2.tag suppression, schema_url clear, awsattributelimit, and batching.
  • Aligned set_workload to the canonical full definition so all copies are identical (avoids an order-dependent merge
    collision).
  • Enabled solutions in the cluster test config and regenerated the golden.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this
contribution, under the terms of your choice.

Tests

  • go build ./... and go test ./translator/... pass; gofmt clean; golden regenerated.
  • E2E on EKS (operator + JSON config, solutions enabled): cluster-scraper discovered the KEDA/Karpenter
    endpoints and added the scrape jobs; confirmed keda_scaler_active and karpenter_nodes_total in CloudWatch.
Screenshot 2026-08-14 at 20 57 42 Screenshot 2026-08-14 at 20 58 45

Requirements

  1. Run make fmt and make fmt-sh
  2. Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

@Paamicky Paamicky added the ready for testing Indicates this PR is ready for integration tests to run label Aug 17, 2026
@Paamicky Paamicky changed the title Mcommey/ci keda kapernter mcommey/ci keda kapernter Aug 17, 2026
@Paamicky
Paamicky marked this pull request as ready for review August 17, 2026 19:47
@Paamicky
Paamicky requested a review from a team as a code owner August 17, 2026 19:47
@Paamicky Paamicky changed the title mcommey/ci keda kapernter Add KEDA and Karpenter metrics scrapping to CWAgent Aug 17, 2026
@Paamicky
Paamicky changed the base branch from main to mcommey/OtelContainerInsight August 19, 2026 14:03
@Paamicky
Paamicky force-pushed the mcommey/CI-keda-kapernter branch from 6b39d69 to 2599868 Compare August 19, 2026 14:11
Comment thread translator/config/schema.json Resolved
Comment thread translator/config/schema.json Resolved
@Paamicky
Paamicky force-pushed the mcommey/CI-keda-kapernter branch 2 times, most recently from d6967d2 to 786fc17 Compare August 20, 2026 18:45
sky333999
sky333999 previously approved these changes Aug 20, 2026
@github-actions

github-actions Bot commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Binary Size Report

linux/amd64

Binary PR vs main (87efdff) vs v1.300072.0
amazon-cloudwatch-agent 194.8 MB ${\color{red}▲}$ +1.5 MB (+0.8%) ${\color{red}▲}$ +1.5 MB (+0.8%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B +0 B
config-downloader 2.0 MB +0 B +0 B
config-translator 2.0 MB +0 B +0 B
start-amazon-cloudwatch-agent 2.5 MB +0 B +0 B
workload-discovery 3.0 MB +0 B +0 B
Total 206.3 MB ${\color{red}▲}$ +1.5 MB (+0.7%) ${\color{red}▲}$ +1.5 MB (+0.7%)
linux/amd64 amazon-cloudwatch-agent (last 9 main commits + this PR)

 195 ┤                                        
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                                    ███ 
 193 ┤▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ███ 
 MB  └────────────────────────────────────────
      589a727             v1.300072.0     PR

Notable changes:

  • +1.5 MB at PR
linux/arm64
Binary PR vs main (87efdff) vs v1.300072.0
amazon-cloudwatch-agent 178.5 MB ${\color{red}▲}$ +1.4 MB (+0.8%) ${\color{red}▲}$ +1.4 MB (+0.8%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B +0 B
config-downloader 2.0 MB +0 B +0 B
config-translator 2.0 MB +0 B +0 B
start-amazon-cloudwatch-agent 2.4 MB +0 B +0 B
workload-discovery 2.9 MB +0 B +0 B
Total 189.8 MB ${\color{red}▲}$ +1.4 MB (+0.8%) ${\color{red}▲}$ +1.4 MB (+0.8%)
windows/amd64
Binary PR vs main (87efdff) vs v1.300072.0
amazon-cloudwatch-agent.exe 196.1 MB ${\color{red}▲}$ +34.3 KB (+0.0%) ${\color{red}▲}$ +34.3 KB (+0.0%)
amazon-cloudwatch-agent-config-wizard.exe 2.1 MB +0 B +0 B
config-downloader.exe 2.1 MB +0 B +0 B
config-translator.exe 2.1 MB +0 B +0 B
start-amazon-cloudwatch-agent.exe 2.3 MB +0 B +0 B
workload-discovery.exe 2.9 MB +0 B +0 B
Total 207.8 MB ${\color{red}▲}$ +34.3 KB (+0.0%) ${\color{red}▲}$ +34.3 KB (+0.0%)
Investigating size changes

Use go-size-analyzer to compare binaries:

GOEXPERIMENT=jsonv2 go install github.com/Zxilly/go-size-analyzer/cmd/gsa@latest
gsa diff --old <baseline-binary> --new <new-binary>

jefchien
jefchien previously approved these changes Aug 20, 2026
Comment thread translator/config/schema.json Outdated
"type": "object",
"properties": {
"enabled": {
"description": "Default toggle(true) for solutions; when false, no solutions run even.",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Description seems incomplete. I think we're trying to say that this takes precedence over the individual solution "enabled" toggles.

@Paamicky
Paamicky dismissed stale reviews from jefchien and sky333999 via f338b13 August 21, 2026 00:34
@Paamicky
Paamicky force-pushed the mcommey/CI-keda-kapernter branch from 21283c2 to f338b13 Compare August 21, 2026 00:34
@Paamicky
Paamicky disabled auto-merge August 21, 2026 00:53
@Paamicky
Paamicky force-pushed the mcommey/CI-keda-kapernter branch from f338b13 to 1e5bc05 Compare August 21, 2026 02:12
TravisStark
TravisStark previously approved these changes Aug 21, 2026
JayPolanco
JayPolanco previously approved these changes Aug 21, 2026
@JayPolanco
JayPolanco disabled auto-merge August 21, 2026 14:36
@JayPolanco
JayPolanco disabled auto-merge August 21, 2026 14:37
@Paamicky
Paamicky disabled auto-merge August 21, 2026 14:40
Base automatically changed from mcommey/OtelContainerInsight to main August 21, 2026 15:09
@Paamicky
Paamicky dismissed stale reviews from JayPolanco and TravisStark August 21, 2026 15:09

The merge-base changed after approval.

@sky333999
sky333999 force-pushed the mcommey/CI-keda-kapernter branch from 1e5bc05 to f9d2e34 Compare August 21, 2026 15:09
@Paamicky
Paamicky merged commit 2b4d0a1 into main Aug 21, 2026
9 checks passed
@Paamicky
Paamicky deleted the mcommey/CI-keda-kapernter branch August 21, 2026 15:27
sky333999 added a commit that referenced this pull request Sep 3, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 3, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 16, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 18, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 18, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
sky333999 added a commit that referenced this pull request Sep 21, 2026
Adapts the self-telemetry (#2246) and GCE/GKE (#2255, #2258, #2259)
features to the v0.150 collector APIs and fork components:

- self-telemetry: v0.150 moves service telemetry config behind
  component.Config (concrete type in service/telemetry/otelconftelemetry),
  so tests assert the concrete config before reading its fields; the two
  self-telemetry goldens pick up the v0.150 telemetry shape.
- resourcedetection: set ignore_detector_errors on gcp_config.yaml so the
  GCP detector degrades instead of aborting startup, matching the other
  five embedded configs (TestAllEmbeddedConfigsIgnoreDetectorErrors
  enforces this) and update the GCP test expectations accordingly.
- migrate the EC2 tag test stubs to the aws-sdk-go-v2
  DescribeTagsAPIClient signature.
- extension/selftelemetry: otelprom WithoutCounterSuffixes/WithoutUnits
  are deprecated as of the exporter version in this cohort; keep them
  (the replacement WithTranslationStrategy also changes metric name
  escaping) with nolint until they are migrated deliberately.
- regenerate the Container Insights (#2249, #2250) and GCE (#2260)
  goldens for v0.150 defaults, and drop a duplicated
  skip_replicaset_watch key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants