Repository navigation
ci : reduce (disable SYCL and CANN builds/releases) - #23705
Conversation
[no ci]
github-actions
Bot
added
the
devops
[no ci]
|
@arthw @hipudding PTAL - I am planning to disable the SYCL and CANN builds and releases until we provision more resources. @IMbackK In case you have some ideas about the ROCm/HIP builds. These are probably likely to stay for now, but if the CI continues to be slow, we'll likely have to remove those too. Btw, I see that for ROCm we only create a Linux release and for HIP we create only a Windows release. Why is that? I.e. why not create both Linux/Windows for both? |
ggerganov
changed the title
|
@ggerganov HIP is a programming language of which ROCm is an implementation. As for why the linux build is called rocm and the windows build is called HIP, i have no idea, we should probably just call both of them ROCm as we dont actually support running the HIP backend on platforms other than ROCm anymore (ie hip-cpu or hip-nvidia) I dont really have any ideas for reducing the CI impact of the HIP backend. Really we should be doing more builds of it, not less, as currently we dont build for all targets until release time, which has cause release time build failures before as we have plenty of ifdefe'd code paths that are only compiled on specific targets. I dont see any way here other than acquiring more resources. |
|
Ok, thanks. I'll fix the naming to use ROCm for both. Edit: will keep the names for now. |
CISC
left a comment
CISC
left a comment
There was a problem hiding this comment.
We could also disable caching of the oneAPI toolkit at the risk of download failing...
|
We can try this from |
|
@ggerganov SYCL CI has been separated from the build.yml, other backend PR won't trigger SYCL CI action. Lots of windows users use the binary package directly. Thank you! |
|
Maybe we could reduce the CI workload. |
|
How long do the SYCL jobs take without the cache? |
|
about 20 mins: main time is download oneAPI and install it locally. |
|
In a pure CUDA code changed PR: #23349 But only CUDA jobs is useful in fact. Note, there is no SYCL job running for this PR. |
|
In the case of that pr its actually only the hip jobs that are useful not the cuda ones altho separating if a change affects the hip backend the cuda backend or both is beyond a ci script. |
* origin/master: (59 commits) ggml-zendnn : fixed naming of matmul function (ggml-org#20964) ci : do not allocate ccache for 3rd-party hosted runners (ggml-org#23730) ci : move [no release] check to dedicated check_release job (ggml-org#23734) ci : add `[no release]` keyword + fix sanitizer builds (ggml-org#23728) ci : move macos jobs to the apple workflow + fix names (ggml-org#23721) vulkan: optimize conv2d and implement coopmat1 support (ggml-org#22620) ci : remove vulkan SDK dep from webgpu job (ggml-org#23718) hexagon: add support for CONCAT op (ggml-org#23648) ci : move more CPU jobs to self-hosted runners (ggml-org#23715) ci : move sanitizer jobs to self-hosted runners (ggml-org#23713) ci : reduce (disable SYCL and CANN builds/releases) (ggml-org#23705) convert : support Gemma4ForCausalLM architecture (ggml-org#23682) models : Attach Mistral3 NVFP4 weight scales (ggml-org#23629) SYCL: implement ggml_sycl_pool_vmm (ggml-org#22862) tests: test-backend-ops -j <N> to run tests in parallel (ggml-org#23637) model : add support for talkie-1930-13b (ggml-org#22596) ggml-webgpu: Add MMVQ path for Q4/Q8/Q2_K/Q4_K and clean up legacy MUL_MAT pipeline (ggml-org#23594) [WebGPU] Check batch_compute_passes before sending passes when not doing GPU profiling (ggml-org#23457) CUDA: missing PDL sync for FWHT, better fallback (ggml-org#23690) metal : add apple device id (ggml-org#23566) ...
* ci : reduce [no ci] * cont : disable sycl, cann + rename caches [no ci] * cont : cann [no ci]
|
At Qt Creator we save the This bypasses the 10 GB cache that GitHub has, since you can have unlimited space for build artifacts 😅 See https://github.com/qt-creator/qt-creator/blob/master/.github/workflows/build_cmake.yml#L556 for details. The build artifacts are short lived and I think it doesn't affect GitHub's disk space that much. I think it won't be too hard for an LLM to convert the CMake code to something else used by llama.cpp's CI build yaml files. |
|
Please return SYCL. |
|
@ggerganov @cics Thank you! |
|
@arthw Would need to see the implementation and the performance to decide. You can give it a try in a fork and when you have something working I'll take a look. |
|
Thanks a lot, having the Windows SYCL builds back again would be awesome. |
|
@ggerganov Thank you! |
Same pattern as PR #74 (build-sycl.yml): upstream PR ggml-org#23705 commented out every job in this workflow, but the push:master + pull_request auto-triggers stayed live. Any PR touching ggml/src/ggml-cann/** would create a zero-job "failure" run. Keep workflow_dispatch so the workflow stays available for manual runs once jobs are re-enabled (which requires provisioning dedicated runners per the upstream TODO). Co-authored-by: marksverdhei <mark.sverdhei@gmail.com>
Upstream disabled the CANN (Huawei Ascend NPU) build by commenting out its only job (ggml-org PR ggml-org#23705), leaving an empty `jobs:` map. GitHub Actions treats that as a startup_failure on any push touching the file, producing a spurious red run. CANN is irrelevant to this fork's targets (macOS/Metal, Linux Vulkan, Windows CPU/Vulkan/CUDA).
|
we need ascend 910B docker image, how to build it? |
Upstream disabled the CANN build by commenting out its entire jobs section (PR ggml-org#23705), leaving a workflow file with zero jobs. GitHub preflight rejects it, so every push (here and upstream) creates a 0s failure run. Verified: upstream ggml-org/llama.cpp fails identically since 2026-05-26. Remove from fleet; we inherit upstreams fixed version whenever they repair or drop it.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
FYI: currently the last CANN bulid is b9334 |
Upstream disabled the CANN build by commenting out its entire jobs section (PR ggml-org#23705), leaving a workflow file with zero jobs. GitHub preflight rejects it, so every push (here and upstream) creates a 0s failure run. Verified: upstream ggml-org/llama.cpp fails identically since 2026-05-26. Remove from fleet; we inherit upstreams fixed version whenever they repair or drop it.
* ci : reduce [no ci] * cont : disable sycl, cann + rename caches [no ci] * cont : cann [no ci]
* ci : reduce [no ci] * cont : disable sycl, cann + rename caches [no ci] * cont : cann [no ci]
Upstream disabled the CANN (Huawei Ascend NPU) build by commenting out its only job (ggml-org PR ggml-org#23705), leaving an empty `jobs:` map. GitHub Actions treats that as a startup_failure on any push touching the file, producing a spurious red run. CANN is irrelevant to this fork's targets (macOS/Metal, Linux Vulkan, Windows CPU/Vulkan/CUDA). (cherry picked from commit d86eb0b)
build-cann.yml has every job commented out - upstream disabled the build in ggml-org/llama.cpp#23705 - yet it still asked to run on push and pull_request. A workflow with no jobs cannot be loaded, so GitHub recorded a failed run with zero jobs and the file path as its name on every push: a red check that says nothing about the commit, and that has been red for the last six of them. Its push filter also named `master`, which this fork does not have (the branch is `main`), so restoring it verbatim from upstream would still never fire. Both facts are in the file for whoever re-enables the build once dedicated CANN runners exist. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
Removing the push trigger was not enough: build-cann.yml has every real job commented out (upstream disabled the build in ggml-org/llama.cpp#23705), and a workflow file with no jobs cannot be parsed. GitHub then ignores the file's triggers - which is why the `master` push filter never mattered on this fork's `main` - and records a failed run named after the file path with zero jobs on every push. One real job keeps the file valid and loadable, the build stays disabled, and with only `workflow_dispatch` left there is nothing to run on push. Dispatching it by hand now prints why the build is off and which block to restore. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
第 245 轮：上一轮定的待追项是
「45 个跨类型 fattn-vec 配对在 FA_ALL_QUANTS=ON 时的编译时间代价」。
那个数语料里没有。
故改去查 release.yml（57,190 字节，1,541 行），
结果发现它根本没有 linux-cuda 这个作业。
一、发布流水线里没有 linux-cuda 这个作业
按 2 空格缩进逐一枚举 release.yml 的 15 个顶层作业:
check-release(41) macos-cpu(62) ubuntu-cpu(143)
ubuntu-vulkan(229) android-arm64(314) ubuntu-24-openvino(406)
windows-cpu(503) windows(571) windows-cuda(670)
ubuntu-22-rocm(962) windows-hip(1082) ios-xcode(1205)
ui(1355) release(1360) ui-publish(1531)
15 个名字里，带 cuda 的只有 windows-cuda 一个。
判据（本轮最重要）：
当一个发布流水线有 15 个作业、
而 Linux 与 CUDA 这两件事从未在同一行出现时，
这不是「漏了」，
而是「那两件事被分配给两个平台」。
而对 RTX PRO 6000 的意义是直接的：
那张卡跑在 Linux 上时，
没有一份官方 CUDA 二进制可供对比 --
每一个 Linux CUDA 数字都是某个人自己编的。
二、发布用的那个 CUDA 作业也不设架构参数
release.yml 第 714-724 行:
cmake -S . -B build -G "Ninja Multi-Config" ^
-DGGML_BACKEND_DL=ON ^
-DGGML_NATIVE=OFF ^
-DGGML_CPU=OFF ^
-DGGML_CUDA=ON ^
-DLLAMA_BUILD_BORINGSSL=ON ^
-DGGML_CUDA_CUB_3DOT2=ON
六个 -D 参数，一个 CMAKE_CUDA_ARCHITECTURES 都没有。
判据（本轮最重要）：
当发布流水线与 CI 都不写架构时，
「这个二进制含哪些架构」这个问题
就只能去 CMakeLists 的默认值里回答 --
而那是一个会随 CUDA 版本变的表达式。
故「CUDA >= 12.8 追加 120a-real；>= 12.9 追加 121a-real」
在这条流水线上是唯一的架构来源。
三、CUDA 那份发布物是一个 dll
第 724 行: cmake --build ... --target ggml-cuda
第 734 行: 7z a -snl ... .\build\bin\Release\ggml-cuda.dll
判据（本轮最重要）：
当发布物是一个后端 dll 而不是一个可执行文件时，
「这个包能用」这句话就多了一个前提：
运行它的那个 llama-server 必须与它 ABI 兼容。
故 GGML_BACKEND_DL=ON 不是「更模块化」，
而是「必须再配一个主程序」。
而这条与 10.36 那个 wheel shadowing 是同一件事的两面：
动态后端既能让你换掉它，也能让别人的那份顶替你。
四、三个 CUDA 版本号：12.4 / 13.0.2 / 13.3
release.yml 第 679-681 行:
strategy:
matrix:
cuda: ['12.4', '13.3']
而 BTL-3 自己的打包脚本用的是 13.0.2。
判据（本轮最重要）：
当同一个仓库里出现三个不同的 CUDA 版本号时，
它们不是「三个版本」，而是「三个不同的承诺」--
CI 承诺 Ada 能编，
发布承诺 Windows 能用，
打包承诺这张卡能跑。
故「用 CUDA 13 就行」这句话，
在这份仓库里至少对应三个不同的具体版本。
五、被注释掉的是 sycl 与 cann，不是 cuda
release.yml 里被 # 注释掉的顶层作业名只有三个:
windows-sycl (759 行)
ubuntu-24-sycl (872 行)
openEuler-cann (1265 行)
第 757-758 行的注释:
# TODO: this build is disabled to save Github Actions resources
# (ggml-org/llama.cpp#23705)
# in order to enable it again, we have to provision dedicated runners
判据（本轮最重要）：
当一个「被禁用」的作业旁边写着「为了省资源」时，
不能用它反推「它曾经能跑」--
而更不能用它反推「另一个作业曾经存在过」。
故那个「没有 linux-cuda」不能被解释成「linux-cuda 被禁用了」，
它从来没有被写过。
而这条正是本报告反复遇到的那类错：
看到一段被注释的 YAML 就以为「这里本来有一个」，
而那个「本来」是我补上去的。
六、四条 CUDA 下载链接全部带 win
第 1493-1494 行:
- [Windows x64 (CUDA 12)](.../llama-...-bin-win-cuda-12.4-x64.zip)
- [CUDA 12.4 DLLs](.../cudart-llama-bin-win-cuda-12.4-x64.zip)
- [Windows x64 (CUDA 13)](.../llama-...-bin-win-cuda-13.3-x64.zip)
- [CUDA 13.3 DLLs](.../cudart-llama-bin-win-cuda-13.3-x64.zip)
四条链接的档名里全部含 win。
判据（本轮最重要）：
当一份发布说明里同一个后端的四条链接都带同一个平台词时，
那个平台词不是修饰，而是范围。
故「llama.cpp 有 CUDA 发布」这句话在 Linux 上是假的 --
它有 Windows 的 CUDA 发布。
故任何声称「用官方 llama.cpp CUDA 二进制测得」的 Linux 数字，
都不可能是官方二进制。
七、故语料里每一个 Linux + CUDA 数字，都是自己编的
三条串起来:
1. 发布流水线没有 linux-cuda 作业
2. 发布说明里四条 CUDA 链接全部是 win
3. 而 RTX PRO 6000 上的实测全部是 Linux
故那些数字背后的二进制，没有一份能追溯到官方发布物。
判据（本轮最重要）：
当「官方没有这个平台的二进制」时，
这个平台上的每一个数字都自带一个未记录的变量：
编译器版本、架构列表、CMake 开关。
故那些 PRO 6000 数字的「可比性」，
不能只问「同一个模型、同一个量化」，
还必须问「同一份自编方式」。
八、这条链有四个环节，没有一个检查 Blackwell
CI（89-real，Ada） 不检查
发布（Windows，不设架构） 不检查
打包（120a-real） 编，但不验证
实测（用户自己编） 这一步才是唯一的验证
四个环节里只有第三与第四会碰到 120a，
而它们都不产出「被验证过」的证据。
判据（本轮最重要）：
当一条链上的每一环都只保证「我这一环没报错」时，
整条链不保证任何端到端的事 --
而「Blackwell 能跑」恰好是一件端到端的事。
而这条也解释了 10.568 那个反直觉结论：
唯一会编到 Blackwell 的自动化是手动触发的 Windows 作业，
而 Windows 上并没有人测 RTX PRO 6000。
方法学补充：
这一轮是「要查的那个数不存在，故改查另一个文件，
结果撞见了更大的缺口」。
上一轮定的待追项是
「45 个跨类型配对在 FA_ALL_QUANTS=ON 时的编译时间代价」。
那个数在语料里没有、也查不到。
于是改去查 release.yml，
结果发现它根本没有 linux-cuda 这个作业。
故「一个查不到的数」可以是一个转向信号：
它说明我原先假设的那条路并不存在。
判据：
当一个数查不到时，先别记为缺口 --
先去查「产生那个数的东西」是否存在。
规模变化：
来源 466 -> 467（release.yml 57,190 字节）
证据 2913 -> 2919
主张 2119 -> 2127（高置信 1769 -> 1777）
轨迹 1003 -> 1005 事件
新建 10.576 到 10.583（8 节）
参考文献覆盖 467/467。孤儿源 2 个（长期存在）。
26 项测试全部通过。
Overview
I believe we are trashing the Github Actions cache too much lately which is causing slow CI overall. This PR aims to disable some of the builds with the goal to lift some of the cache and runners pressure.
SYCLbuilds alone consume more than 1/3 of the total 10GB cache that we have. I don't think it's reasonable, so disabling them for now. In order to re-enable, we have to provision dedicated runners.openEulerbuilds are not consuming cache which is good. However, they allocate slots from the GH hosted runners. I'd like to move these builds to dedicated runners too.TODO
Additional information
Also prefixed the caches with
cache-gha-to be able to search and match easily.Requirements