[tokinasin]

Overview

Only CPU and CUDA are currently supported, changes to other backends are to mark as unsupported.

YaRN calculates correction intervals to switch frequency interpolation methods, but the setting to not round the boundaries of these intervals to integers (truncate=false) was not supported.

Models that use truncate=false include gpt-oss and some PLaMo-3 models; to support those models correctly, the GGML operations and CPU/CUDA backends are modified to handle calculations without rounding. The default behavior is truncate=true, and existing calls not explicitly configured will maintain their previous behavior.

Additional information

• models that use truncate=false
  • gpt-oss: https://huggingface.co/openai/gpt-oss-20b/blob/main/config.json#L66
  • plamo3: https://huggingface.co/pfnet/plamo-3-610m-fin-instruct/blob/main/modeling_plamo.py#L236

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES
  • I used Codex for research, code generation and reviews, etc.

[CISC]

Generally seems ok, @ggerganov would have to weight in on the implementation.

I assume there will be follow ups for at least one backend with test and conversion/loading?

[tokinasin]

Added CUDA backend support and tests.
I plan to handle the conversion/loading fixes in a separate PR.

[tokinasin]

Rebased since the Metal test-fusion CI failure appears to have been fixed in #30618.

[tokinasin]

@ggerganov

Thanks for the reviews so far!

Since I'm limited to having two PRs open at a time, I'd appreciate it if you could take another look at this PR when you have a chance.

Please let me know if there's anything else I need to address.

[CISC]

Bad merge. Nvm, rebase please. :)

[ggerganov]

One more rebase and we can merge.

[0cc4m]

Same as CUDA, Vulkan already supports this, so disabling it is not needed, it can just be forwarded. E.g. with this diff:

diff

diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
index cb891e354..75f8dfb69 100644
--- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp
+++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
@@ -10786,7 +10786,6 @@ static vk_op_rope_push_constants ggml_vk_make_rope_constants(const ggml_tensor *
     const int n_dims        = ((const int32_t *) dst->op_params)[1];
     const int mode          = ((const int32_t *) dst->op_params)[2];
     const int n_offs        = ((const int32_t *) dst->op_params)[15];
-    // const int n_ctx         = ((const int32_t *) dst->op_params)[3];
     const int n_ctx_orig    = ((const int32_t *) dst->op_params)[4];
     const float freq_base   = ((const float *)   dst->op_params)[5];
     const float freq_scale  = ((const float *)   dst->op_params)[6];
@@ -10802,7 +10801,7 @@ static vk_op_rope_push_constants ggml_vk_make_rope_constants(const ggml_tensor *
     const bool is_imrope = mode == GGML_ROPE_TYPE_IMROPE;

     float corr_dims[2];
-    ggml_rope_yarn_corr_dims(n_dims, n_ctx_orig, freq_base, beta_fast, beta_slow, true, corr_dims);
+    ggml_rope_yarn_corr_dims(n_dims, n_ctx_orig, freq_base, beta_fast, beta_slow, ggml_rope_get_truncate(dst), corr_dims);

     const float theta_scale = powf(freq_base, -2.0f/n_dims);

@@ -11286,20 +11285,6 @@ void ggml_vk_rope(ggml_backend_vk_context * ctx, vk_context& subctx, const ggml_
     const ggml_tensor * src1 = dst->src[1];
     const ggml_tensor * src2 = dst->src[2];
     const ggml_tensor * src3 = nullptr;
-    const int n_dims        = ((int32_t *) dst->op_params)[1];
-    const int mode          = ((int32_t *) dst->op_params)[2];
-    // const int n_ctx         = ((int32_t *) dst->op_params)[3];
-    const int n_ctx_orig    = ((int32_t *) dst->op_params)[4];
-    const float freq_base   = ((float *)   dst->op_params)[5];
-    const float beta_fast   = ((float *)   dst->op_params)[9];
-    const float beta_slow   = ((float *)   dst->op_params)[10];
-    int sections[4] {};
-    if (mode & GGML_ROPE_TYPE_MROPE) {
-        memcpy(sections, (int32_t *) dst->op_params + 11, sizeof(int)*4);
-    }
-
-    float corr_dims[2];
-    ggml_rope_yarn_corr_dims(n_dims, n_ctx_orig, freq_base, beta_fast, beta_slow, true, corr_dims);

     uint32_t set_rows_stride = 0;
     // Fused rope + view + set_rows passes the set_rows destination stride in set_rows_stride
@@ -14074,7 +14059,7 @@ bool ggml_vk_can_fuse_rope_set_rows(ggml_backend_vk_context * ctx, const struct
         return false;
     }

-    return ggml_rope_get_truncate(rope);
+    return true;
 }

 bool ggml_vk_can_fuse_rms_norm_set_rows(ggml_backend_vk_context * ctx, const struct ggml_cgraph * cgraph,
@@ -14212,7 +14197,7 @@ bool ggml_vk_can_fuse_rms_norm_mul_rope(ggml_backend_vk_context * ctx, const str
         return false;
     }

-    return ggml_rope_get_truncate(rope);
+    return true;
 }

 uint32_t ggml_vk_fuse_multi_add(ggml_backend_vk_context * ctx, const struct ggml_cgraph * cgraph, int node_idx) {
@@ -15762,12 +15747,8 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
         case GGML_OP_REPEAT_BACK:
             return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32;
         case GGML_OP_ROPE:
-            if (!ggml_rope_get_truncate(op)) {
-                return false;
-            }
             return ggml_is_contiguous_rows(op) && ggml_is_contiguous_rows(op->src[0]);
         case GGML_OP_ROPE_BACK:
-            return ggml_rope_get_truncate(op);
         case GGML_OP_NONE:
         case GGML_OP_RESHAPE:
         case GGML_OP_VIEW:

If you prefer I can also just do this as a follow-up PR.

[tokinasin]

Same as CUDA, Vulkan already supports this, so disabling it is not needed, it can just be forwarded. E.g. with this diff:
diff

If you prefer I can also just do this as a follow-up PR.

@0cc4m Thanks for the suggestion!

I've already opened a draft follow-up PR (#30802) to support truncation config on other backends I have access to, including Vulkan.

Supporting the remaining backends (metal, hexagon, etc.) should be fairly straightforward as well, but I've left them out for now since I can't verify the changes myself.

I can also incorporate those changes into this PR if that's preferred :)

[0cc4m]

No, that's fine as well, I hadn't seen that PR.