Skip to content

fix: eole relative position key scale - #2138

Merged
jordimas merged 1 commit into
OpenNMT:masterfrom
maksimcialehin:fix-eole-relative-key-scale
Oct 27, 2026
Merged

jordimas merged 1 commit into
OpenNMT:masterfrom
maksimcialehin:fix-eole-relative-key-scale

Conversation

@maksimcialehin

Copy link
Copy Markdown
Contributor

eole with position_encoding_type: Relative scores the relative-key term with an unscaled query ((Q Kᵀ) / sqrt(d) + Q R_kᵀ). CTranslate2 applies queries_scale = 1/sqrt(d) to both terms. The eole converter copied the same embedding table into relative_position_keys and relative_position_values, so the key term was sqrt(d) times too small.

This pre-scales relative_position_keys by sqrt(dim_per_head) and leaves relative_position_values unchanged.
Fixes #2134

@jordimas
jordimas requested a review from vince62s October 27, 2026 11:54
@vince62s

Copy link
Copy Markdown
Member

LGTM

@jordimas

Copy link
Copy Markdown
Collaborator

Thanks!

@jordimas
jordimas merged commit cdadde1 into OpenNMT:master Oct 27, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

eole converter: position_encoding_type: Relative models are converted with the wrong scale on relative keys

3 participants