Skip to content

Add file tailing support to opentelemetry section - #2206

Merged
jefchien merged 7 commits into
mainfrom
otel-file-log
Jul 17, 2026
Merged

jefchien merged 7 commits into
mainfrom
otel-file-log

Conversation

@jefchien

Copy link
Copy Markdown
Contributor

Description of changes

Add file tailing support to the opentelemetry config section via the filelogreceiver, at opentelemetry.collect.files.collect_list.

  • Extended the filelog receiver translator with options for encoding, multiline, timestamp parsing, resource attributes, and file_storage
  • Created per-entry pipeline in translator/translate/otel/pipeline/opentelemetry/files/ (receiver dedup by config hash)
    • Default log group: /aws/cwagent/files
    • Default log stream: {host.id}/{file.name}
  • Schema validation in translator/config/schema.json
  • Extracted strftime-to-regex and strftime-to-Go-layout maps into shared internal/util/timestamp/ package

Supported fields (matching v1 naming): file_path, log_group_name, log_stream_name, multi_line_start_pattern, timestamp_format, timezone, encoding.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Added and updated unit tests. Added sample config for the new configuration option. Tested E2E on EC2 to validate tailing, default log group/stream, custom log group/stream, multi-line, receiver dedup, timestamp parsing, and state persistence/resume.

Requirements

Before commiting your code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

@jefchien jefchien added the ready for testing Indicates this PR is ready for integration tests to run label Jul 16, 2026
@jefchien
jefchien requested a review from a team as a code owner July 16, 2026 15:56
JayPolanco
JayPolanco previously approved these changes Jul 16, 2026
JayPolanco
JayPolanco previously approved these changes Jul 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Binary Size Report

linux/amd64

Binary PR vs main (60b2321) vs v1.300070.0
amazon-cloudwatch-agent 233.3 MB ${\color{red}▲}$ +36.9 KB (+0.0%) ${\color{red}▲}$ +2.0 MB (+0.8%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B ${\color{red}▲}$ +86.0 KB (+4.5%)
config-downloader 2.0 MB +0 B ${\color{red}▲}$ +86.0 KB (+4.5%)
config-translator 2.0 MB +0 B ${\color{red}▲}$ +86.0 KB (+4.5%)
start-amazon-cloudwatch-agent 2.5 MB +0 B ${\color{red}▲}$ +81.9 KB (+3.4%)
workload-discovery 3.0 MB +0 B ${\color{red}▲}$ +86.0 KB (+2.9%)
Total 244.8 MB ${\color{red}▲}$ +36.9 KB (+0.0%) ${\color{red}▲}$ +2.4 MB (+1.0%)
linux/amd64 amazon-cloudwatch-agent (last 9 main commits + this PR)

 233 ┤                                        
     ┤            ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
 231 ┤▄▄▄ ▄▄▄ ▄▄▄ ███ ███ ███ ███ ███ ███ ███ 
 MB  └────────────────────────────────────────
      42a07a3                             PR

Notable changes:

linux/arm64
Binary PR vs main (60b2321) vs v1.300070.0
amazon-cloudwatch-agent 212.2 MB ${\color{red}▲}$ +65.5 KB (+0.0%) ${\color{red}▲}$ +720.9 KB (+0.3%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B ${\color{green}▼}$ 22 B (-0.0%)
config-downloader 2.0 MB +0 B ${\color{green}▼}$ 22 B (-0.0%)
config-translator 2.0 MB +0 B ${\color{green}▼}$ 22 B (-0.0%)
start-amazon-cloudwatch-agent 2.4 MB +0 B ${\color{red}▲}$ +65.5 KB (+2.8%)
workload-discovery 2.9 MB +0 B ${\color{red}▲}$ +65.5 KB (+2.3%)
Total 223.5 MB ${\color{red}▲}$ +65.5 KB (+0.0%) ${\color{red}▲}$ +851.8 KB (+0.4%)
windows/amd64
Binary PR vs main (60b2321) vs v1.300070.0
amazon-cloudwatch-agent.exe 236.2 MB ${\color{red}▲}$ +45.6 KB (+0.0%) ${\color{red}▲}$ +2.1 MB (+0.9%)
amazon-cloudwatch-agent-config-wizard.exe 2.1 MB +0 B ${\color{red}▲}$ +130.6 KB (+6.5%)
config-downloader.exe 2.1 MB +0 B ${\color{red}▲}$ +130.0 KB (+6.5%)
config-translator.exe 2.1 MB +0 B ${\color{red}▲}$ +130.6 KB (+6.5%)
start-amazon-cloudwatch-agent.exe 2.3 MB +0 B ${\color{red}▲}$ +112.1 KB (+5.2%)
workload-discovery.exe 2.9 MB +0 B ${\color{red}▲}$ +91.6 KB (+3.3%)
Total 247.9 MB ${\color{red}▲}$ +45.6 KB (+0.0%) ${\color{red}▲}$ +2.6 MB (+1.1%)
Investigating size changes

Use go-size-analyzer to compare binaries:

GOEXPERIMENT=jsonv2 go install github.com/Zxilly/go-size-analyzer/cmd/gsa@latest
gsa diff --old <baseline-binary> --new <new-binary>

okankoAMZ
okankoAMZ previously approved these changes Jul 17, 2026
@jefchien
jefchien dismissed stale reviews from JayPolanco and okankoAMZ via 01beeb2 July 17, 2026 18:47
// buildTimestampOperatorMap returns a regex_parser operator config as a raw map.
// Using a raw map avoids the operator.Config marshaling bug when round-tripping through confmap.
func buildTimestampOperatorMap(format, timezone string) map[string]any {
location := "Local"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a nit: The json schema has 3 enums it allows, but the if condition here only targets 2 of the 3

@jefchien
jefchien merged commit 5d99671 into main Jul 17, 2026
405 of 413 checks passed
@jefchien
jefchien deleted the otel-file-log branch July 17, 2026 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants