Skip to content

Fix loki shutdown log loss and surface dropped cloud logs - #6232

Merged
joanlopez merged 3 commits into
masterfrom
local-exec-log-drainfix
Aug 1, 2026
Merged

joanlopez merged 3 commits into
masterfrom
local-exec-log-drainfix

Conversation

@ankur22

@ankur22 ankur22 commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #6171 (now merged), rebased directly onto master. Addresses two review threads deferred from that PR. All commits signed.

What?

Two fixes deferred from #6171's review:

  1. Drain the loki hook on shutdown (internal/log/loki.go). lokiHook.Listen's ctx.Done() branch pushed only the entries it had already consumed into its buffer and never drained what was still queued in the channel, so a shutdown could lose a tail of logs. It now drains the channel before the final push. This is shared code β€” it also fixes --out loki's final-batch loss β€” and the loss was made observable by Stream --local-execution logs to Grafana CloudΒ #6171's cloud-log Drain() path. Split into a pure-refactor commit (extract the per-entry buffering into an appendEntry helper) and the fix itself.

  2. Report dropped cloud logs (internal/log/cloud/cloud.go). The cloud log Pusher.Fire counted entries it dropped when the front buffer was full, but nothing read that counter, so a full buffer lost logs silently. It now emits a synthetic k6 dropped N log messages… warning at shutdown, mirroring how --out loki reports its own limit drops. The notice is pushed via a new log.NoticePusher capability on the loki hook, which sends it as a standalone request exempt from the hook's per-batch limit (a full final batch would otherwise swallow it), and it is pushed after the final flush so it carries the newest timestamp in the warning stream it shares with the hook's own limit-drop message β€” avoiding an out-of-order write that Loki rejects with unordered_writes=false.

Three atomic commits, each individually buildable + lintable + green under make check.

Why?

Both were flagged in #6171's review as pre-existing gaps that its Drain() path exposed, but that belonged in a dedicated follow-up rather than widening that PR:

  • The shutdown tail-loss lives in the shared loki hook (also used by --out loki), so fixing it there benefits both paths and was kept out of the cloud-logs PR.
  • The write-only dropped counter only becomes reliably reportable once the drain is fixed β€” otherwise the "dropped N" notice could itself be lost on shutdown, which is why it is sequenced after the drain fix here.

Checklist

  • I have performed a self-review of my code.
  • I have commented on my code, particularly in hard-to-understand areas.
  • I have added tests for my changes.
  • I have run linter and tests locally (make check) and all pass.

Related PR(s)/Issue(s)

@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 30, 2026 22:11 — with GitHub Actions Inactive
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 30, 2026 22:13 — with GitHub Actions Inactive
@ankur22
ankur22 force-pushed the local-exec-log-drainfix branch from 40d86cb to 45fa421 Compare July 31, 2026 10:08
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 31, 2026 10:14 — with GitHub Actions Inactive
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 31, 2026 10:15 — with GitHub Actions Inactive
@ankur22
ankur22 marked this pull request as ready for review July 31, 2026 11:04
@ankur22
ankur22 requested a review from a team as a code owner July 31, 2026 11:04
@ankur22
ankur22 requested review from inancgumus and janHildebrandt98 and removed request for a team July 31, 2026 11:04
@cursor cursor Bot mentioned this pull request Jul 31, 2026
4 of 8 tasks
@ankur22
ankur22 force-pushed the local-exec-log-drainfix branch from 45fa421 to defd0b8 Compare July 31, 2026 12:34
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 31, 2026 12:40 — with GitHub Actions Inactive
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 31, 2026 12:42 — with GitHub Actions Inactive
@ankur22
ankur22 force-pushed the local-exec-log-drainfix branch from defd0b8 to c83c387 Compare July 31, 2026 12:50
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 31, 2026 12:56 — with GitHub Actions Inactive
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 31, 2026 12:58 — with GitHub Actions Inactive
inancgumus
inancgumus previously approved these changes Jul 31, 2026
Base automatically changed from local-exec-logs-take3 to master July 31, 2026 16:23
@ankur22
ankur22 dismissed stale reviews from janHildebrandt98 and inancgumus July 31, 2026 16:23

The base branch was changed.

ankur22 added 3 commits July 31, 2026 17:25
Extract the per-entry buffering in lokiHook.Listen (the limit check,
label building and appending to msgs) into a local appendEntry
closure, so the shutdown path can reuse the same logic.

Pure refactor with no behavior change: the existing loki tests still
pass unchanged.
lokiHook.Listen's shutdown branch pushed only the entries already
consumed into its buffer, never draining what was still queued in the
channel, so a ctx-cancel could lose a tail of logs. This is shared
with --out loki and was made observable by the cloud log pusher's
Drain path.

Drain the channel before the final push. The gocognit nolint is
extended because the drain adds one branch to an already-exempt
function.
Fire counted entries dropped when the buffer was full, but nothing ever
read that counter, so a full buffer lost logs silently. Surface the
count as a synthetic warning, mirroring how --out loki reports its own
limit drops.

The notice is pushed through the loki hook's NoticePusher capability,
which sends it immediately and exempt from the per-batch limit, so a
full final batch (many buffered entries at shutdown, exactly when drops
are likely) cannot swallow the notice itself. It is pushed after the
final flush so it is the newest entry in the warning stream it shares
with the hook's own limit-drop message, avoiding an out-of-order write
that Loki with unordered_writes=false would reject.
@ankur22
ankur22 force-pushed the local-exec-log-drainfix branch from c83c387 to 9cfc4e7 Compare July 31, 2026 16:27
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 31, 2026 16:34 — with GitHub Actions Inactive
@ankur22
ankur22 temporarily deployed to azure-trusted-signing July 31, 2026 16:35 — with GitHub Actions Inactive
@joanlopez joanlopez added this to the v2.2.0 milestone Aug 1, 2026
@joanlopez
joanlopez merged commit 36961ee into master Aug 1, 2026
58 checks passed
@joanlopez
joanlopez deleted the local-exec-log-drainfix branch August 1, 2026 06:13
social4hyq pushed a commit to social4hyq/homebrew-core that referenced this pull request Sep 20, 2026
k6 2.2.0

Created-by: HarmonybrewBot
Commit-by: HarmonybrewBot
Merged-by: HarmonybrewBot
Description: Created by `brew bump`

---

Created with `brew bump-formula-pr`.<details>
  <summary>release notes</summary>
  <pre>k6 `v2.2.0` is here πŸŽ‰! This release includes:

- `k6 cloud run --local-execution` now streams k6's logs to Grafana Cloud, so the test run's log view works for local execution too.
- `chromium.connectOverCDP()`, which connects browser tests to an already-running Chromium instance.
- `TextEncoder` and `TextDecoder` available as globals, and `WritableStream` support in `k6/experimental/streams`.
- A `k6 cloud load-zone list` command.
- Two new experimental feature flags: `merge-run-tags` and `freeze-env`.

## Breaking changes

There are no breaking changes in this release.

## New features

### `k6 cloud run --local-execution` streams logs to Grafana Cloud [#6171](grafana/k6#6171)

When a cloud test runs locally with `k6 cloud run --local-execution`, k6's logs now stream to the Grafana Cloud test run, so the run's log view is populated the same way it is for cloud execution. Previously, local-execution logs stayed on the machine running k6 and never reached the cloud. Work with Grafana's secrets management to safely work with secrets and redact them if they're accidentally leaked into logs are pushed. Use the new `--no-cloud-logs` flag opts out to opt out of streaming of logs when working with `--local-execution`:

```shell
k6 cloud run --local-execution script.js
k6 cloud run --local-execution --no-cloud-logs script.js
```

### Connect to a running browser with `chromium.connectOverCDP()` [#6165](grafana/k6#6165)

The browser module can now attach to an existing Chromium-based browser over the Chrome DevTools Protocol, mirroring Playwright's `browserType.connectOverCDP()`. Pass the browser's WebSocket endpoint and k6 manages the returned browser's connection β€” it's auto-closed at the end of the iteration, though you can call `close()` earlier to release the connection on demand.

```js
import { chromium } from 'k6/browser';

export default async function () {
  const browser = await chromium.connectOverCDP('ws://localhost:9222/devtools/browser/<id>');
  const page = await browser.newPage();

  try {
    await page.goto('https://quickpizza.grafana.com/');
  } finally {
    await page.close();
    await browser.close();
  }
}
```

Unlike the `K6_BROWSER_WS_URL` environment variable, the endpoint is a runtime value β€” you can, for example, request a fresh session URL from a browser provider's API in `setup()` and connect to it from the iterations.

### `TextEncoder` and `TextDecoder` globals [#6182](grafana/k6#6182)

`TextEncoder` and `TextDecoder` are now available as standard globals in both the init and VU contexts, no import required β€” matching how they are exposed in browsers and other JavaScript runtimes.

```js
const encoded = new TextEncoder().encode('Hello, world!');
const decoded = new TextDecoder().decode(encoded);
```

### `WritableStream` in `k6/experimental/streams` [#6132](grafana/k6#6132)

The experimental streams module now implements `WritableStream` and `WritableStreamDefaultWriter` following the WHATWG Streams specification, complementing the existing `ReadableStream` and paving the way for a future `TransformStream` implementation.

```js
import { WritableStream } from 'k6/experimental/streams';

export default async function () {
  const stream = new WritableStream({
    write(chunk) {
      console.log(`wrote ${chunk}`);
    },
  });

  const writer = stream.getWriter();
  await writer.write('hello');
  await writer.close();
}
```

### `k6 cloud load-zone list` command [#6142](grafana/k6#6142)

A new `k6 cloud load-zone list` subcommand lists the load zones β€” public and private β€” available in the configured Grafana Cloud k6 stack, mirroring the existing `k6 cloud project list` command. Output defaults to a human-readable table; pass `--json` to emit a JSON array instead.

```console
$ k6 cloud load-zone list
Load zones for https://example.grafana.net:

ID                     NAME                     TYPE      AVAILABLE
amazon:us:ashburn      Ashburn, US (Amazon)     public    yes
amazon:sa:cape town    Cape Town, SA (Amazon)   public    yes
```

### Configurable `handleSummary()` timeout [#5854](grafana/k6#5854)

The time budget for the `handleSummary()` callback β€” previously hardcoded to 120 seconds β€” is now configurable through the `handleSummaryTimeout` option or the `K6_HANDLE_SUMMARY_TIMEOUT` environment variable, so long-running tests with heavy summaries no longer fail with `handleSummary() execution timed out`. Thanks, @LBaronceli!

```js
export const options = {
  handleSummaryTimeout: '5m',
};
```

### New experimental feature flags: `merge-run-tags` and `freeze-env`

Two new experimental flags join the feature-flag system introduced in v2.1.0:

- [#5714](grafana/k6#5714) `merge-run-tags` merges run tags per key across config layers, so `options.tags` in a script is no longer silently discarded when `--tag` or `K6_TAGS` is also used β€” higher-priority layers win on conflicting keys instead of replacing the whole map. Thanks, @yordis!
- [#6032](grafana/k6#6032) `freeze-env` freezes the `__ENV` object, so modifications from script code throw a `TypeError` (in strict mode) instead of silently persisting across iterations and scenarios. Thanks, @lohitkolluri!

```shell
k6 run --features merge-run-tags,freeze-env script.js
```

## UX improvements and enhancements

- [#5631](grafana/k6#5631) Makes the browser module's header accessors β€” `response.allHeaders()`, `headerValue()`, `headerValues()`, and `headersArray()` β€” return the raw wire headers (including `Set-Cookie` and security-related headers), correctly paired with each hop of a redirect chain instead of Chrome's provisional headers. As part of this, `headerValues()` now matches header names case-insensitively and splits repeated values on newlines rather than commas, and the `browser_data_sent`/`browser_data_received` metrics now include the raw header bytes and no longer vary run-to-run with CDP event ordering.
- [#6208](grafana/k6#6208) Makes `k6 cloud` reject the run flags (for example, `--vus`) with an `unknown flag` error and a non-zero exit code. Previously `k6 cloud --vus 10 script.js` accepted the flags, printed the help text, and exited 0 β€” running tests with `k6 cloud` directly was deprecated in v2.0.0 in favor of `k6 cloud run`.
- [#6096](grafana/k6#6096) Points the cloud secrets error at `K6_CLOUD_SECRETS_TOKEN` and `K6_CLOUD_SECRETS_ENDPOINT` when a test run is reused via `K6_CLOUD_PUSH_REF_ID` under `--local-execution`, instead of suggesting the `--local-execution` flag the user is already using.
- [#6196](grafana/k6#6196) Adds `catch` blocks to the browser examples so a failing iteration reports the original error instead of a subsequent `page.close()` failure. Thanks, @locker95!

## Bug fixes

- [#6234](grafana/k6#6234) Classifies HTTP/2 errors by message so the `error_code` metric tag stays correct when k6 is built with Go 1.27 (whose `x/net/http2` delegates to the standard library), and explicitly enables HTTP/2 negotiation on VU transports.
- [#6232](grafana/k6#6232) Drains queued log entries in the Loki hook at shutdown so `--out loki` and cloud log streaming no longer lose the final batch, and emits a `k6 dropped N log messages` warning when the cloud log buffer overflows instead of dropping logs silently.
- [#6125](grafana/k6#6125) Serializes the first concurrent open of a file in the caching filesystem so parallel `fs.open()` calls on the same file no longer read zero or truncated bytes.
- [#6147](grafana/k6#6147) Fixes a data race and inconsistent request-interception state when browser routes are added or removed concurrently. Thanks, @somak2kai!
- [#6070](grafana/k6#6070) Flushes buffered file log output once per second so recent logs aren't lost when k6 is killed before shutdown. Thanks, @rohan-patnaik!
- [#6205](grafana/k6#6205) Stops sending an invalid `Sec-WebSocket-Protocol` header when tailing Grafana Cloud logs; spec-strict servers rejected the handshake with `websocket: bad handshake`.
- [#6200](grafana/k6#6200) Leaves a counter's `rate` unset when the observed duration is zero, instead of computing `+Inf` and spuriously failing `rate` thresholds. Thanks, @samarth70!
- [#6195](grafana/k6#6195) Initializes a gauge's maximum from the first sample so all-negative gauge series no longer report `max=0`. Thanks, @Solaris-star!
- [#6145](grafana/k6#6145) Prevents the OpenTelemetry output from panicking at startup when basic auth is configured without `K6_OTEL_HEADERS`. Thanks, @lukdz!
- [#6140](grafana/k6#6140) Stops `SharedArray` deep-freezing JS primitives, which needlessly wrapped large strings in `String` objects β€” cutting memory usage in the reported reproduction from roughly 1 GB to 100 MB.

## Maintenance and internal improvements

- [#6126](grafana/k6#6126), [#6224](grafana/k6#6224), [#6229](grafana/k6#6229) Adds anonymous extension usage to the k6 usage report: a run reports the Go module path, version, and type of registry-cataloged extensions it actually uses (imported `k6/x/` modules, output extensions selected with `--out`, and `k6 x` subcommands). Private and unlisted extensions are never reported, and the existing `--no-usage-report` opt-out covers it.
- [#6183](grafana/k6#6183), [#6218](grafana/k6#6218) Updates Sobek and regexp2, making `W

See merge request: Harmonybrew/homebrew-core!16299

This branch was previously deployed

1 inactive deployment
azure-trusted-signing β€” 9cfc4e7b Deployed Jul 31, 2026 by ankur22 via sign-packages #10224
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants