Skip to content

Drama 3: cache prompt prefixes and optimize prefill and GPU decoding - #2133

Merged
jordimas merged 15 commits into
OpenNMT:masterfrom
jordimas:drama3-prefix-cache
Oct 30, 2026
Merged

Drama 3: cache prompt prefixes and optimize prefill and GPU decoding#2133
jordimas merged 15 commits into
OpenNMT:masterfrom
jordimas:drama3-prefix-cache

Conversation

@jordimas

@jordimas jordimas commented Oct 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator
  • Cache static prompt prefixes, reusing backbone and audio-tower state across requests, with cache controls and hit/miss diagnostics.
  • Increase the default prefill chunk size from 16 to 128 in the native and Python APIs.
  • Fuse gated-delta A/B projections into one GPU kernel during single-token decoding, falling back when unsupported.

jordimas added 14 commits October 26, 2026 19:30
@jordimas jordimas changed the title Drama 3: prefix caching like in LLM models Drama 3: cache prompt prefixes and optimize prefill and GPU decoding Oct 30, 2026
@jordimas
jordimas merged commit 78b1e17 into OpenNMT:master Oct 30, 2026
14 checks passed
@jordimas
jordimas deleted the drama3-prefix-cache branch October 30, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jordimas