⚽ Live Match: aws/amazon cloudwatch agent/pull/

EduVencovsky · 2026-08-04T14:12:19Z

Description of the issue

Database Insights collects per-resource (table/index) metrics and server metrics from a single PostgreSQL receiver at the default 10s interval. Per-resource metrics scale with the number of relations in the monitored database — a database with thousands of tables produces thousands of readings each scrape. Collecting them every 10s is unnecessary for the DBI experience and inflates ingestion volume.

Description of changes

Per-resource metrics split to a dedicated receiver at 60s (configurable):

The DBI metrics pipeline now runs two postgresql receiver instances per monitored database: one for server/per-database metrics at the default 10s, and one for per-resource (table/index) metrics at 60s.
A new per-instance config field per_resource_collection_interval (in seconds) lets customers override the 60s default. If omitted or zero, defaults to 60s.
Both scopes only ever disable metrics — the set of metrics emitted is identical to before, partitioned by cadence.
The per-resource instance disables events (query-sample / top-query) so DBLoad and TopSQL are not double-counted.
A reflection-based helper (forEachMetric) walks the receiver's generated metrics config so per-resource metrics are matched by mapstructure name rather than hand-maintained field references.
TestTranslator_MetricSets_Disjoint asserts the two scopes are disjoint and their union equals the default enabled set, guarding against double emission or metric loss.

Events receiver metrics disabled (WithMetricsDisabled / metricScopeNone):

The events receiver now explicitly disables all metrics, since it feeds a logs pipeline only and the metrics were never collected. The generated config now reflects what the instance actually emits.

Config shape (unchanged if field omitted):

{
  "opentelemetry": {
    "collect": {
      "database_insights": {
        "postgresql": [{
          "endpoint": "localhost:5432",
          "username": "cw_monitor",
          "password_file": "/opt/.pgpass",
          "per_resource_collection_interval": 120
        }]
      }
    }
  }
}

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

TestTranslator_MetricSets_Disjoint — proves server + per-resource union equals the default set (no drop, no double emission)
TestTranslator_ServerMetricsOnly / TestTranslator_PerResourceMetricsOnly — per-field assertions on each scope
TestTranslator_MetricsDisabled — events receiver emits zero metrics, events survive
TestParseDbiPostgresqlInstances_PerResourceCollectionInterval — default (60s), custom (120s), zero-falls-back-to-default
TestTranslator_Translate_Defaults_MetricScope — default scope unchanged
tocwconfig golden tests (TestDBIConfigLinux, TestCombinedV1V2EC2Config, TestCombinedV1V2EKSConfig) — regenerated and passing
Local end-to-end: built agent from branch, ran against real PostgreSQL 14 (Docker), confirmed server metrics at 10s (6 samples/min) and per-resource at 60s (1 sample/min) via CloudWatch SampleCount in account 923855483676
al2023:otel_collect_database_insights_test passed in CI (16m58s) on an earlier push of this branch

Requirements

Run make fmt and make fmt-sh
Run make lint

github-actions · 2026-08-05T11:29:42Z

Binary Size Report

linux/amd64

Binary	PR	vs `main` (`589a727`)	vs `v1.300070.0`
amazon-cloudwatch-agent	193.3 MB	${\color{red}▲}$ +4.1 KB (+0.0%)	${\color{green}▼}$ 38.0 MB (-16.4%)
amazon-cloudwatch-agent-config-wizard	2.0 MB	+0 B	${\color{red}▲}$ +86.0 KB (+4.5%)
config-downloader	2.0 MB	+0 B	${\color{red}▲}$ +86.0 KB (+4.5%)
config-translator	2.0 MB	+0 B	${\color{red}▲}$ +86.0 KB (+4.5%)
start-amazon-cloudwatch-agent	2.5 MB	+0 B	${\color{red}▲}$ +81.9 KB (+3.4%)
workload-discovery	3.0 MB	+0 B	${\color{red}▲}$ +94.2 KB (+3.2%)
Total	204.9 MB	${\color{red}▲}$ +4.1 KB (+0.0%)	${\color{green}▼}$ 37.6 MB (-15.5%)

linux/amd64 amazon-cloudwatch-agent (last 9 main commits + this PR)

 193 ┤                                        
     ┤                        ▄▄▄ ▄▄▄ ▄▄▄ ███ 
     ┤        ███ ███ ███ ███ ███ ███ ███ ███ 
     ┤        ███ ███ ███ ███ ███ ███ ███ ███ 
     ┤        ███ ███ ███ ███ ███ ███ ███ ███ 
     ┤        ███ ███ ███ ███ ███ ███ ███ ███ 
     ┤        ███ ███ ███ ███ ███ ███ ███ ███ 
     ┤        ███ ███ ███ ███ ███ ███ ███ ███ 
 193 ┤▄▄▄ ▄▄▄ ███ ███ ███ ███ ███ ███ ███ ███ 
 MB  └────────────────────────────────────────
      0c715d0                             PR

linux/arm64

Binary	PR	vs `main` (`589a727`)	vs `v1.300070.0`
amazon-cloudwatch-agent	177.1 MB	+0 B	${\color{green}▼}$ 34.4 MB (-16.3%)
amazon-cloudwatch-agent-config-wizard	2.0 MB	+0 B	${\color{green}▼}$ 22 B (-0.0%)
config-downloader	2.0 MB	+0 B	${\color{green}▼}$ 22 B (-0.0%)
config-translator	2.0 MB	+0 B	${\color{green}▼}$ 22 B (-0.0%)
start-amazon-cloudwatch-agent	2.4 MB	+0 B	${\color{red}▲}$ +65.5 KB (+2.8%)
workload-discovery	2.9 MB	+0 B	${\color{red}▲}$ +65.5 KB (+2.3%)
Total	188.4 MB	+0 B	${\color{green}▼}$ 34.3 MB (-15.4%)

windows/amd64

Binary	PR	vs `main` (`589a727`)	vs `v1.300070.0`
amazon-cloudwatch-agent.exe	196.1 MB	${\color{red}▲}$ +12.3 KB (+0.0%)	${\color{green}▼}$ 38.1 MB (-16.3%)
amazon-cloudwatch-agent-config-wizard.exe	2.1 MB	+0 B	${\color{red}▲}$ +130.6 KB (+6.5%)
config-downloader.exe	2.1 MB	+0 B	${\color{red}▲}$ +130.0 KB (+6.5%)
config-translator.exe	2.1 MB	+0 B	${\color{red}▲}$ +130.6 KB (+6.5%)
start-amazon-cloudwatch-agent.exe	2.3 MB	+0 B	${\color{red}▲}$ +112.1 KB (+5.2%)
workload-discovery.exe	2.9 MB	+0 B	${\color{red}▲}$ +97.3 KB (+3.5%)
Total	207.7 MB	${\color{red}▲}$ +12.3 KB (+0.0%)	${\color{green}▼}$ 37.5 MB (-15.3%)

Investigating size changes

Use go-size-analyzer to compare binaries:

GOEXPERIMENT=jsonv2 go install github.com/Zxilly/go-size-analyzer/cmd/gsa@latest
gsa diff --old <baseline-binary> --new <new-binary>

JayPolanco · 2026-08-05T15:26:27Z

How would i customer set this up? Will this be in the JSON, do they have to do it int he yaml?

JayPolanco · 2026-08-05T15:31:28Z

Should also have yaml translations for all 3 options probably so we can confirm that the behavior is what we expect per field enabled

JayPolanco · 2026-08-06T15:45:03Z

+	perResourceRx := t.pgReceiver("metrics_perresource",
+		postgresql.WithPerResourceMetricsOnly(),
+		postgresql.WithCollectionInterval(60*time.Second),
+		postgresql.WithEventsDisabled(),


Are these disabled by default?

zhihonl · 2026-08-06T15:48:10Z

+	serverRx := t.pgReceiver("metrics", postgresql.WithServerMetricsOnly())
+	perResourceRx := t.pgReceiver("metrics_perresource",
+		postgresql.WithPerResourceMetricsOnly(),
+		postgresql.WithCollectionInterval(60*time.Second),


Translator is updated when we need to translate a key in the agent JSON to expand into a template YAML config. The changes we've made in this PR doesn't give the customer a way to configure these values in the JSON.

Is this intended to be an opinionated YAML config that shouldn't let customers modify by JSON?

zhihonl · 2026-08-06T16:28:56Z

Can the PR description be updated with more details following the PR template

# Description of the issue
_Describe the problem or feature in addition to a link to the issues._

# Description of changes
_How does this change address the problem?_

# License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

# Tests
_Describe what tests you have done._

# Requirements
_Before commiting your code, please do the following steps._
1. Run `make fmt` and `make fmt-sh`
2. Run `make lint`

-------
### Integration Tests
To run integration tests against this PR, add the `ready for testing` label.

JayPolanco · 2026-08-06T18:23:09Z

+// with a postgresql.table.name / postgresql.index.name resource attribute.
+// postgresql.table.count is intentionally excluded: it is a per-database server metric
+// recorded alongside the other server metrics.
+var perResourceMetricNames = map[string]bool{


Since this is hardcoded, we'll have to keep in mind that upstream could always add more metrics, which would then start being published every 10s like the rest of the metrics. Might be worth having some sort of test in the future

Is this even possible? If so how?

Technically it is. I think it's less likely in the sense that I don't think it's very common for them to implement new metrics as defaults. likely they'd be optional. BUt just a heads up

JayPolanco · 2026-08-06T19:11:30Z

Did we test this on an actual instance? You can copy what our integ test for dbi does to setup and confirm that everything is working as expected

movence · 2026-08-07T14:05:40Z

 	s2mConn := signaltometrics.NewTranslator(common.DbiConnectorTopsql)

+	// Split postgresql metric collection into two receiver instances so per-resource
+	// (table/index) metrics are scraped at 60s while server metrics stay at the default


the comment could be more specific that it's defaulted at 60s unless configured otherwise

AshuSrv · 2026-08-07T14:23:10Z

        top_query_collection:
            collection_interval: 1m0s
-            max_explain_each_interval: 1000
+            max_explain_each_interval: 0


looks like this is reverting one of my changes.

movence · 2026-08-10T13:30:56Z

+                      "per_resource_collection_interval": {
+                        "description": "How often per-resource (table/index) metrics are collected, in seconds. Defaults to 60 if omitted.",
+                        "type": "integer",
+                        "minimum": 1


this would make configurations like per_resource_collection_interval: 0 to be invalid which doesn't seem correct

…eceiver

EduVencovsky force-pushed the dbi-per-resource-60s branch from f9c5e60 to ca93050 Compare August 4, 2026 14:40

EduVencovsky added the ready for testing Indicates this PR is ready for integration tests to run label Aug 4, 2026

EduVencovsky force-pushed the dbi-per-resource-60s branch 2 times, most recently from 2c01139 to aeb133d Compare August 5, 2026 10:41

JayPolanco reviewed Aug 6, 2026

View reviewed changes

zhihonl reviewed Aug 6, 2026

View reviewed changes

EduVencovsky force-pushed the dbi-per-resource-60s branch from aeb133d to 122e0c7 Compare August 6, 2026 17:26

zhihonl previously approved these changes Aug 6, 2026

View reviewed changes

JayPolanco reviewed Aug 6, 2026

View reviewed changes

EduVencovsky marked this pull request as ready for review August 6, 2026 19:03

EduVencovsky requested a review from a team as a code owner August 6, 2026 19:03

EduVencovsky force-pushed the dbi-per-resource-60s branch from 122e0c7 to 190e97e Compare August 6, 2026 19:52

EduVencovsky added ready for testing Indicates this PR is ready for integration tests to run and removed ready for testing Indicates this PR is ready for integration tests to run labels Aug 7, 2026

EduVencovsky force-pushed the dbi-per-resource-60s branch from 190e97e to 2fb7ddc Compare August 7, 2026 11:14

EduVencovsky dismissed zhihonl’s stale review via 3caf30b August 7, 2026 13:39

EduVencovsky force-pushed the dbi-per-resource-60s branch 2 times, most recently from 3caf30b to 619a30b Compare August 7, 2026 14:08

movence reviewed Aug 7, 2026

View reviewed changes

AshuSrv reviewed Aug 7, 2026

View reviewed changes

EduVencovsky force-pushed the dbi-per-resource-60s branch 2 times, most recently from 5d1074a to 82423ad Compare August 7, 2026 16:49

JayPolanco previously approved these changes Aug 7, 2026

View reviewed changes

EduVencovsky force-pushed the dbi-per-resource-60s branch from 82423ad to 1cadef7 Compare August 10, 2026 10:01

movence reviewed Aug 10, 2026

View reviewed changes

EduVencovsky dismissed JayPolanco’s stale review via 612f0fd August 10, 2026 14:25

EduVencovsky force-pushed the dbi-per-resource-60s branch from 1cadef7 to 612f0fd Compare August 10, 2026 14:25

movence approved these changes Aug 10, 2026

View reviewed changes

zhihonl approved these changes Aug 10, 2026

View reviewed changes

JayPolanco approved these changes Aug 10, 2026

View reviewed changes

feat(dbi): split postgresql per-resource metrics to a dedicated 60s r…

d72a4c2

…eceiver

EduVencovsky force-pushed the dbi-per-resource-60s branch from 612f0fd to d72a4c2 Compare August 10, 2026 16:12

movence merged commit 81a0387 into main Aug 10, 2026
856 of 869 checks passed

movence deleted the dbi-per-resource-60s branch August 10, 2026 18:21

Conversation

EduVencovsky commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the issue

Description of changes

License

Tests

Requirements

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Report

linux/amd64

Uh oh!

JayPolanco commented Aug 5, 2026

Uh oh!

JayPolanco commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhihonl commented Aug 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JayPolanco commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

EduVencovsky commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 5, 2026 •

edited

Loading

JayPolanco commented Aug 6, 2026 •

edited

Loading