# llama.cpp for OpenCL
-- [Background](#background)
-- [OS](#os)
-- [Hardware](#hardware)
-- [DataType Supports](#datatype-supports)
-- [Model Preparation](#model-preparation)
-- [CMake Options](#cmake-options)
-- [Android](#android)
-- [Windows 11 Arm64](#windows-11-arm64)
-- [Linux](#Linux)
-- [Known Issue](#known-issues)
-- [TODO](#todo)
+- [llama.cpp for OpenCL](#llamacpp-for-opencl)
+ - [Background](#background)
+ - [Llama.cpp + OpenCL](#llamacpp--opencl)
+ - [OS](#os)
+ - [Hardware](#hardware)
+ - [Adreno GPU](#adreno-gpu)
+ - [DataType Supports](#datatype-supports)
+ - [Model Preparation](#model-preparation)
+ - [Binary Kernel Library](#binary-kernel-library)
+ - [CMake Options](#cmake-options)
+ - [Android](#android)
+ - [I. Setup Environment](#i-setup-environment)
+ - [II. Build llama.cpp](#ii-build-llamacpp)
+ - [Windows 11 Arm64](#windows-11-arm64)
+ - [I. Setup Environment](#i-setup-environment-1)
+ - [II. Build llama.cpp](#ii-build-llamacpp-1)
+ - [Linux](#linux)
+ - [I. Setup Environment](#i-setup-environment-2)
+ - [II. Build llama.cpp](#ii-build-llamacpp-2)
+ - [Known Issues](#known-issues)
+ - [TODO](#todo)
## Background
**Verified devices**
-| Adreno GPU | Status |
-|:------------------------------------:|:-------:|
-| Adreno 750 (Snapdragon 8 Gen 3) | Support |
-| Adreno 830 (Snapdragon 8 Elite) | Support |
-| Adreno X85 (Snapdragon X Elite) | Support |
+| Adreno GPU | Status |
+|:-------------------------------------:|:-------:|
+| Adreno 750 (Snapdragon 8 Gen 3) | Support |
+| Adreno 830 (Snapdragon 8 Elite) | Support |
+| Adreno 840 (Snapdragon 8 Elite Gen 5) | Support |
+| Adreno X1-85 (Snapdragon X Elite) | Support |
+| Adreno X2-90 (Snapdragon X2 Elite) | Support |
> A6x GPUs with a recent driver and compiler are supported; they are usually found in IoT platforms.
However, A6x GPUs in phones are likely not supported due to the outdated driver and compiler.
| DataType | Status |
|:----------------------:|:--------------------------:|
+| Q1_0 | Support |
| Q4_0 | Support |
-| Q6_K | Support, but not optimized |
+| Q4_1 | Support |
+| Q5_0 | Support |
+| Q5_1 | Support |
| Q8_0 | Support |
+| Q4_K | Support |
+| Q5_K | Support |
+| Q6_K | Support |
| MXFP4 | Support |
+| IQ4_NL | Support |
## Model Preparation
-You can refer to the general [llama-quantize tool](/tools/quantize/README.md) for steps to convert a model in Hugging Face safetensor format to GGUF with quantization.
+Since common quantizations are supported now, it is recommanded to download GGUF models directly from Huggingface.
-Currently we support `Q4_0` quantization and have optimized for it. To achieve best performance on Adreno GPU, add `--pure` to `llama-quantize` (i.e., make all weights in `Q4_0`). For example,
+## Binary Kernel Library
-```sh
-./llama-quantize --pure ggml-model-qwen2.5-3b-f16.gguf ggml-model-qwen-3b-Q4_0.gguf Q4_0
-```
-
-Since `Q6_K` is also supported, `Q4_0` quantization without `--pure` will also work. However, the performance will be worse compared to pure `Q4_0` quantization.
-
-### `MXFP4` MoE Models
-
-OpenAI gpt-oss models are MoE models in `MXFP4`. The quantized model will be in `MXFP4_MOE`, a mixture of `MXFP4` and `Q8_0`.
-For this quantization, there is no need to specify `--pure`.
-For gpt-oss-20b model, you can directly [download](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) the quantized GGUF file in `MXFP4_MOE` from Hugging Face.
+A prebuilt binary kernel library has been introduced for Adreno GPUs.
+It currently targets X2 GPUs (X2-90, X2-85 and X2-45) found in Snapdragon X2 SoC.
+The library currently contains kernels for MUL_MAT_ID with Q4_0, Q4_1, Q4_K, MXFP4.
+The library must be manually downloaded from https://softwarecenter.qualcomm.com/catalog/item/Adreno_Kernel_Library_GGML.
-Although it is possible to quantize gpt-oss-20b model in pure `Q4_0` (all weights in `Q4_0`), it is not recommended since `MXFP4` has been optimized for MoE while `Q4_0` is not. In addition, accuracy should degrade with such pure `Q4_0` quantization.
-Hence, using the default `MXFP4_MOE` quantization (see the link above) is recommended for this model.
+To allow using the kernel library, add `-DGGML_OPENCL_USE_ADRENO_BIN_KERNELS=ON` when configuring with CMake.
+Then, extract `adreno-opencl-kernels.dll` from the zip file downloaded from the above URL and put it alongside the executables.
+If kernels compatible with the current GPU are found in the library, they will be loaded and used.
-> Note that the `Q4_0` model found [here](https://huggingface.co/unsloth/gpt-oss-20b-GGUF/blob/main/gpt-oss-20b-Q4_0.gguf) is a mixture of `Q4_0`, `Q8_0` and `MXFP4` and gives better performance than `MXFP4_MOE` quantization.
## CMake Options
The OpenCL backend has the following CMake options that control the behavior of the backend.
-| CMake options | Default value | Description |
-|:---------------------------------:|:--------------:|:------------------------------------------|
-| `GGML_OPENCL_EMBED_KERNELS` | `ON` | Embed OpenCL kernels into the executable. |
-| `GGML_OPENCL_USE_ADRENO_KERNELS` | `ON` | Use kernels optimized for Adreno. |
+| CMake options | Default value | Description |
+|:------------------------------------:|:--------------:|:------------------------------------------|
+| `GGML_OPENCL_EMBED_KERNELS` | `ON` | Embed OpenCL kernels into the executable. |
+| `GGML_OPENCL_USE_ADRENO_KERNELS` | `ON` | Use kernels optimized for Adreno. |
+| `GGML_OPENCL_USE_ADRENO_BIN_KERNELS` | `OFF` | Allow using binary kernel lib for Adreno. |
## Android
## TODO
-- Optimization for Q6_K
-- Support and optimization for Q4_K
- Improve flash attention
+- Improve OpenCL C kernels performance
#include "ggml-backend-impl.h"
#include "ggml.h"
+#ifdef GGML_OPENCL_USE_ADRENO_BIN_KERNELS
+#include "libdl.h"
+#ifdef _WIN32
+#define KERNEL_LIB_NAME "adreno-opencl-kernels.dll"
+#else
+#define KERNEL_LIB_NAME "libadreno-opencl-kernels.so"
+#endif // _WIN32
+#endif // GGML_OPENCL_USE_ADRENO_BIN_KERNELS
+
+typedef const void * (*get_adreno_bin_kernel_func_t)(
+ const char * name,
+ const char * gpu_name,
+ const char * compiler_ver,
+ size_t * out_size
+);
+
#include <CL/cl.h>
#include <inttypes.h>
bool adreno_has_large_buffer;
bool adreno_use_large_buffer;
+ bool adreno_use_bin_kernels;
+ get_adreno_bin_kernel_func_t get_adreno_bin_kernel_func = nullptr;
ggml_cl_compiler_version adreno_cl_compiler_version;
std::string kernel_compile_opts; // cached for lazy-compiled kernels.
cl_kernel kernel_gated_delta_net_f32[4][2][2] = {};
cl_kernel kernel_timestep_embedding;
- cl_kernel kernel_gemv_moe_q4_0_f32_ns, kernel_gemm_moe_q4_0_f32_ns;
- cl_kernel kernel_gemv_moe_q4_1_f32_ns, kernel_gemm_moe_q4_1_f32_ns;
+ cl_kernel kernel_gemv_moe_q4_0_f32_ns, kernel_gemm_moe_q4_0_f32_ns, kernel_gemm_moe_q4_0_f32_ns_bin;
+ cl_kernel kernel_gemv_moe_q4_1_f32_ns, kernel_gemm_moe_q4_1_f32_ns, kernel_gemm_moe_q4_1_f32_ns_bin;
cl_kernel kernel_gemv_moe_q5_0_f32_ns, kernel_gemm_moe_q5_0_f32_ns;
cl_kernel kernel_gemv_moe_q5_1_f32_ns, kernel_gemm_moe_q5_1_f32_ns;
- cl_kernel kernel_gemv_moe_q4_k_f32_ns, kernel_gemm_moe_q4_k_f32_ns;
+ cl_kernel kernel_gemv_moe_q4_k_f32_ns, kernel_gemm_moe_q4_k_f32_ns, kernel_gemm_moe_q4_k_f32_ns_bin;
cl_kernel kernel_gemv_moe_q5_k_f32_ns, kernel_gemm_moe_q5_k_f32_ns;
cl_kernel kernel_gemv_moe_q6_k_f32_ns, kernel_gemm_moe_q6_k_f32_ns;
cl_kernel kernel_gemv_moe_mxfp4_f32, kernel_gemm_moe_mxfp4_f32;
- cl_kernel kernel_gemv_moe_mxfp4_f32_ns, kernel_gemm_moe_mxfp4_f32_ns;
+ cl_kernel kernel_gemv_moe_mxfp4_f32_ns, kernel_gemm_moe_mxfp4_f32_ns, kernel_gemm_moe_mxfp4_f32_ns_bin;
cl_kernel kernel_moe_reorder_b;
cl_kernel kernel_moe_histogram, kernel_moe_scan, kernel_moe_fill, kernel_moe_scatter;
cl_kernel kernel_mul_mv_id_q4_0_f32_8x_flat;
#endif
}
+ const void * get_adreno_bin_kernel(const std::string &kernel_name, size_t *bin_size) const {
+ if (!get_adreno_bin_kernel_func) {
+ return nullptr;
+ }
+
+ size_t sz;
+ const void * kernel_bin = get_adreno_bin_kernel_func(
+ kernel_name.c_str(), device_name.c_str(), driver_version.c_str(), &sz);
+ if (bin_size) {
+ *bin_size = sz;
+ }
+ return kernel_bin;
+ }
+
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
// Transpose kernels
cl_program program_transpose;
cl_kernel kernel_gemv_noshuffle_q4_0_f32_32000_1_4096;
cl_kernel kernel_gemv_noshuffle_q4_1_f32;
cl_kernel kernel_gemm_noshuffle_q4_1_f32;
- cl_kernel kernel_gemm_noshuffle_q8_0_f32;
+ cl_kernel kernel_gemm_noshuffle_q8_0_f32, kernel_gemm_noshuffle_q8_0_f32_bin;
cl_kernel kernel_gemv_noshuffle_q8_0_f32;
cl_kernel kernel_gemm_noshuffle_q1_0_f32;
cl_kernel kernel_gemv_noshuffle_q1_0_f32;
return build_program_from_source_ex(ctx, dev, program_buffer, compile_opts, /*fatal=*/true);
}
+static cl_program build_program_from_binary(cl_context ctx, cl_device_id dev, const char* program_buffer, const std::string &compile_opts, size_t bin_size = 0) {
+ cl_program p;
+ char *program_log;
+ size_t log_size;
+ int err;
+
+ p = clCreateProgramWithBinary(ctx, 1, &dev, &bin_size, (const unsigned char**)&program_buffer, NULL, &err);
+ if(err < 0) {
+ GGML_LOG_ERROR("OpenCL error creating program from binary");
+ exit(1);
+ }
+
+ err = clBuildProgram(p, 0, NULL, compile_opts.c_str(), NULL, NULL);
+ if(err < 0) {
+ clGetProgramBuildInfo(p, dev, CL_PROGRAM_BUILD_LOG, 0, NULL, &log_size);
+ program_log = (char*) malloc(log_size + 1);
+ program_log[log_size] = '\0';
+ clGetProgramBuildInfo(p, dev, CL_PROGRAM_BUILD_LOG, log_size + 1, program_log, NULL);
+ GGML_LOG_ERROR("ggml_opencl: kernel compile error:\n\n%s\n", program_log);
+ free(program_log);
+ exit(1);
+ }
+
+ return p;
+}
+
static void load_cl_kernels_argsort(ggml_backend_opencl_context *backend_ctx) {
// compiler options for general kernels
auto opencl_c_std =
}
}
+static bool use_adreno_bin_kernels(ggml_backend_opencl_context * backend_ctx) {
+#ifndef GGML_OPENCL_USE_ADRENO_BIN_KERNELS
+ return false;
+#else
+ if (backend_ctx->gpu_family != GPU_FAMILY::ADRENO) {
+ return false;
+ }
+ return backend_ctx->adreno_use_bin_kernels;
+#endif // GGML_OPENCL_USE_ADRENO_BIN_KERNELS
+}
+
static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
if (backend_ctx->kernels_loaded) {
return;
GGML_LOG_CONT(".");
}
+ // gemm_noshuffle_q8_0_f32_bin
+ {
+ size_t bin_size = 0;
+ backend_ctx->kernel_gemm_noshuffle_q8_0_f32_bin = nullptr;
+
+ if (use_adreno_bin_kernels(backend_ctx)) {
+ const char * kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_noshuffle_q8_0_f32_ila", &bin_size);
+ if (kernel_bin && bin_size > 0) {
+ cl_program prog =
+ build_program_from_binary(backend_ctx->context, backend_ctx->device, kernel_bin, compile_opts, bin_size);
+
+ CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q8_0_f32_bin = clCreateKernel(prog, "kernel_gemm_noshuffle_q8_0_f32_ila", &err), err));
+ CL_CHECK(clReleaseProgram(prog));
+ GGML_LOG_CONT(".");
+ }
+ }
+ }
+
// gemv_noshuffle_general_q8_0_f32
{
std::string CL_gemv_compile_opts = std::string("-cl-std=") + opencl_c_std +
GGML_LOG_CONT(".");
}
+ // gemm_moe_q4_1_f32_ns_bin
+ {
+ size_t bin_size = 0;
+ backend_ctx->kernel_gemm_moe_q4_1_f32_ns_bin = nullptr;
+
+ if (use_adreno_bin_kernels(backend_ctx)) {
+ const char * kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_moe_q4_1_f32_ns_ila", &bin_size);
+ if (kernel_bin && bin_size > 0) {
+ cl_program prog =
+ build_program_from_binary(backend_ctx->context, backend_ctx->device, kernel_bin, CL_moe_compile_opts, bin_size);
+
+ CL_CHECK((backend_ctx->kernel_gemm_moe_q4_1_f32_ns_bin = clCreateKernel(prog, "kernel_gemm_moe_q4_1_f32_ns_ila", &err), err));
+ CL_CHECK(clReleaseProgram(prog));
+ GGML_LOG_CONT(".");
+ }
+ }
+ }
+
// gemv_moe_mxfp4_f32
{
#ifdef GGML_OPENCL_EMBED_KERNELS
GGML_LOG_CONT(".");
}
+ // gemm_moe_q4_0_f32_ns_bin
+ {
+ size_t bin_size = 0;
+ backend_ctx->kernel_gemm_moe_q4_0_f32_ns_bin = nullptr;
+
+ if (use_adreno_bin_kernels(backend_ctx)) {
+ const char * kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_moe_q4_0_f32_ns_ila", &bin_size);
+ if (kernel_bin && bin_size > 0) {
+ cl_program prog =
+ build_program_from_binary(backend_ctx->context, backend_ctx->device, kernel_bin, CL_moe_compile_opts, bin_size);
+
+ CL_CHECK((backend_ctx->kernel_gemm_moe_q4_0_f32_ns_bin = clCreateKernel(prog, "kernel_gemm_moe_q4_0_f32_ns_ila", &err), err));
+ CL_CHECK(clReleaseProgram(prog));
+ GGML_LOG_CONT(".");
+ }
+ }
+ }
+
// gemv_moe_q5_0_f32_ns
{
#ifdef GGML_OPENCL_EMBED_KERNELS
GGML_LOG_CONT(".");
}
+ // gemm_moe_q4_k_f32_ns_bin
+ {
+ size_t bin_size = 0;
+ backend_ctx->kernel_gemm_moe_q4_k_f32_ns_bin = nullptr;
+
+ if (use_adreno_bin_kernels(backend_ctx)) {
+ const char * kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_moe_q4_k_f32_ns_ila", &bin_size);
+ if (kernel_bin && bin_size > 0) {
+ cl_program prog =
+ build_program_from_binary(backend_ctx->context, backend_ctx->device, kernel_bin, CL_moe_compile_opts, bin_size);
+
+ CL_CHECK((backend_ctx->kernel_gemm_moe_q4_k_f32_ns_bin = clCreateKernel(prog, "kernel_gemm_moe_q4_k_f32_ns_ila", &err), err));
+ CL_CHECK(clReleaseProgram(prog));
+ GGML_LOG_CONT(".");
+ }
+ }
+ }
+
// gemv_moe_q5_k_f32_ns
{
#ifdef GGML_OPENCL_EMBED_KERNELS
cl_program prog =
build_program_from_source(backend_ctx->context, backend_ctx->device, kernel_src.c_str(), CL_moe_compile_opts);
- CL_CHECK((backend_ctx->kernel_gemm_moe_mxfp4_f32_ns = clCreateKernel(prog, "kernel_gemm_moe_mxfp4_f32_ns", &err), err));
- CL_CHECK(clReleaseProgram(prog));
- GGML_LOG_CONT(".");
+ CL_CHECK((backend_ctx->kernel_gemm_moe_mxfp4_f32_ns = clCreateKernel(prog, "kernel_gemm_moe_mxfp4_f32_ns", &err), err));
+ CL_CHECK(clReleaseProgram(prog));
+ GGML_LOG_CONT(".");
+ }
+
+ // gemm_moe_mxfp4_f32_ns_bin
+ {
+ size_t bin_size = 0;
+ backend_ctx->kernel_gemm_moe_mxfp4_f32_ns_bin = nullptr;
+
+ if (use_adreno_bin_kernels(backend_ctx)) {
+ const char * kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_moe_mxfp4_f32_ns_ila", &bin_size);
+ if (kernel_bin && bin_size > 0) {
+ cl_program prog =
+ build_program_from_binary(backend_ctx->context, backend_ctx->device, kernel_bin, CL_moe_compile_opts, bin_size);
+
+ CL_CHECK((backend_ctx->kernel_gemm_moe_mxfp4_f32_ns_bin = clCreateKernel(prog, "kernel_gemm_moe_mxfp4_f32_ns_ila", &err), err));
+ CL_CHECK(clReleaseProgram(prog));
+ GGML_LOG_CONT(".");
+ }
+ }
}
// moe_reorder_b
backend_ctx->adreno_use_large_buffer = getenv("GGML_OPENCL_ADRENO_USE_LARGE_BUFFER") != nullptr &&
backend_ctx->gpu_family == GPU_FAMILY::ADRENO;
+#ifdef GGML_OPENCL_USE_ADRENO_BIN_KERNELS
+ // try loading adreno binary kernels if enabled
+ // if fails to load, builtin kernels will be used
+ {
+ dl_handle * kernel_lib_handle = dl_load_library(KERNEL_LIB_NAME);
+ backend_ctx->adreno_use_bin_kernels = false;
+
+ if (kernel_lib_handle) {
+ backend_ctx->get_adreno_bin_kernel_func = (get_adreno_bin_kernel_func_t)dl_get_sym(kernel_lib_handle, "get_adreno_kernels");
+ if (backend_ctx->get_adreno_bin_kernel_func) {
+ GGML_LOG_INFO("ggml_opencl: loaded bin kernel library %s\n", KERNEL_LIB_NAME);
+ backend_ctx->adreno_use_bin_kernels = true;
+ } else {
+ GGML_LOG_INFO("ggml_opencl: bin kernel library %s is invalid, will use builtin kernels\n", KERNEL_LIB_NAME);
+ }
+ } else {
+ GGML_LOG_INFO("ggml_opencl: failed to load %s, will use builtin kernels\n", KERNEL_LIB_NAME);
+ }
+ }
+#endif // GGML_OPENCL_USE_ADRENO_BIN_KERNELS
+
cl_int err;
// A local ref of cl_context for convenience
CL_CHECK(clReleaseMemObject(b_img));
CL_CHECK(clReleaseMemObject(b_sub_buf));
} else {
+ // use bin kernel if available
+ if (backend_ctx->kernel_gemm_noshuffle_q8_0_f32_bin) {
+ int K_pad = K;
+
+ cl_mem b_sub_buf = nullptr;
+ cl_mem d_sub_buf = nullptr;
+
+ cl_mem a_img = nullptr;
+ cl_mem s_img = nullptr;
+ cl_mem b_img = nullptr;
+ cl_mem d_img = nullptr;
+
+ // subbuffer for activations
+ region.origin = offset1;
+ region.size = K_pad * N * sizeof(float);
+ CL_CHECK((b_sub_buf = clCreateSubBuffer(extra1->data_device, 0, CL_BUFFER_CREATE_TYPE_REGION, ®ion, &err), err));
+
+ // Create subbuffer and image1d_buffer for dst
+ region.origin = (extrad->offset); // + dst->view_offs;
+ region.size = M * N * sizeof(float);
+ CL_CHECK((d_sub_buf = clCreateSubBuffer((extrad->data_device), 0, CL_BUFFER_CREATE_TYPE_REGION, ®ion, &err), err));
+
+ // create an image for A
+ img_fmt = { CL_R, CL_FLOAT};
+ memset(&img_desc, 0, sizeof(img_desc));
+ img_desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
+ img_desc.image_width = M * K / 4; // Divide by 4 for char -> float
+ img_desc.buffer = extra0_q8_0->q;
+ CL_CHECK((a_img = clCreateImage(context, CL_MEM_READ_ONLY, &img_fmt, &img_desc, NULL, &err), err));
+
+ // create an image for Scale
+ img_fmt = { CL_R, CL_HALF_FLOAT};
+ memset(&img_desc, 0, sizeof(img_desc));
+ img_desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
+ img_desc.image_width = M * K / 32; // Block size is 32
+ img_desc.buffer = extra0_q8_0->d;
+ CL_CHECK((s_img = clCreateImage(context, CL_MEM_READ_ONLY, &img_fmt, &img_desc, NULL, &err), err));
+
+ // create an image for B from sub_buffer
+ img_fmt = {CL_R, CL_FLOAT};
+ memset(&img_desc, 0, sizeof(img_desc));
+ img_desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
+ img_desc.image_width = K_pad * N;
+ img_desc.buffer = b_sub_buf;
+ CL_CHECK((b_img = clCreateImage(context, CL_MEM_READ_ONLY, &img_fmt, &img_desc, NULL, &err), err));
+
+ // img for d
+ img_fmt = {CL_R, CL_FLOAT};
+ memset(&img_desc, 0, sizeof(img_desc));
+ img_desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
+ img_desc.image_width = M * N;
+ img_desc.buffer = d_sub_buf;
+ CL_CHECK((d_img = clCreateImage(context, CL_MEM_WRITE_ONLY, &img_fmt, &img_desc, NULL, &err), err));
+
+ // gemm
+ kernel = backend_ctx->kernel_gemm_noshuffle_q8_0_f32_bin;
+
+ bool layoutA_Mfirst = true;
+ bool layoutS_Mfirst = true;
+ bool layoutB_Nfirst = false;
+ bool layoutC_Mfirst = true;
+
+ cl_uint lineStrideMatrixAinBytes = layoutA_Mfirst ? M * 4 : K; // int8
+ cl_uint lineStrideMatrixSinBytes = layoutS_Mfirst ? M * 2 : (K / 32) * 2; // fp16
+ cl_uint lineStrideMatrixBinBytes = layoutB_Nfirst ? N * 4 : K_pad * 4; // fp32
+ cl_uint lineStrideMatrixCinBytes = layoutC_Mfirst ? M * 4 : N * 4; // fp32
+
+ CL_CHECK(clSetKernelArg(kernel, 0, sizeof(cl_mem), &a_img));
+ CL_CHECK(clSetKernelArg(kernel, 1, sizeof(cl_mem), &s_img));
+ CL_CHECK(clSetKernelArg(kernel, 2, sizeof(cl_mem), &b_img));
+ CL_CHECK(clSetKernelArg(kernel, 3, sizeof(int), &extra1->offset));
+ CL_CHECK(clSetKernelArg(kernel, 4, sizeof(cl_mem), &d_img));
+ CL_CHECK(clSetKernelArg(kernel, 5, sizeof(int), &extrad->offset));
+ CL_CHECK(clSetKernelArg(kernel, 6, sizeof(int), &K));
+ CL_CHECK(clSetKernelArg(kernel, 7, sizeof(int), &lineStrideMatrixAinBytes));
+ CL_CHECK(clSetKernelArg(kernel, 8, sizeof(int), &lineStrideMatrixSinBytes));
+ CL_CHECK(clSetKernelArg(kernel, 9, sizeof(int), &lineStrideMatrixBinBytes));
+ CL_CHECK(clSetKernelArg(kernel, 10, sizeof(int), &lineStrideMatrixCinBytes));
+
+ size_t global_work_size[] = { 64, (size_t)CEIL_DIV(M, 64), (size_t)CEIL_DIV(N, 64)};
+ size_t local_work_size[] = { 64, 2, 2 };
+
+ backend_ctx->enqueue_ndrange_kernel(kernel, 3, global_work_size, local_work_size, dst);
+
+ CL_CHECK(clReleaseMemObject(b_sub_buf));
+ CL_CHECK(clReleaseMemObject(d_sub_buf));
+ CL_CHECK(clReleaseMemObject(a_img));
+ CL_CHECK(clReleaseMemObject(s_img));
+ CL_CHECK(clReleaseMemObject(b_img));
+ CL_CHECK(clReleaseMemObject(d_img));
+ return;
+ }
+
cl_mem b_sub_buf = nullptr;
cl_mem b_sub_buf_trans = nullptr;
cl_mem b_img = nullptr;
} else { // for gemm
kernel = backend_ctx->kernel_gemm_moe_q4_0_f32_ns;
+ if (backend_ctx->kernel_gemm_moe_q4_0_f32_ns_bin) {
+ kernel = backend_ctx->kernel_gemm_moe_q4_0_f32_ns_bin;
+ }
// Reorder router if called from test-backend-ops or when new router is generated.
// Otherwise reuse the reordered result from previous mul_mat_id call.
cl_image_desc image_desc_buf_src1;
image_format_buf_src1 = {CL_RGBA, CL_FLOAT};
image_desc_buf_src1 = {CL_MEM_OBJECT_IMAGE1D_BUFFER, static_cast<size_t>(ne00 * max_post_router_tile * n_tile_size / 4), 0,0,0,0,0,0,0, {buf_src1_reordered}};
+ if (backend_ctx->kernel_gemm_moe_q4_0_f32_ns_bin) {
+ // bin kernel uses slightly different image format
+ image_format_buf_src1 = {CL_R, CL_FLOAT};
+ image_desc_buf_src1.image_width = static_cast<size_t>(ne00 * max_post_router_tile * n_tile_size);
+ }
image_src1_reordered = clCreateImage(backend_ctx->context, CL_MEM_READ_ONLY, &image_format_buf_src1, &image_desc_buf_src1, NULL, &status);
CL_CHECK(status);
} else { // for gemm
kernel = backend_ctx->kernel_gemm_moe_q4_1_f32_ns;
+ if (backend_ctx->kernel_gemm_moe_q4_1_f32_ns_bin) {
+ kernel = backend_ctx->kernel_gemm_moe_q4_1_f32_ns_bin;
+ }
// Reorder router if called from test-backend-ops or when new router is generated.
// Otherwise reuse the reordered result from previous mul_mat_id call.
cl_image_desc image_desc_buf_src1;
image_format_buf_src1 = {CL_RGBA, CL_FLOAT};
image_desc_buf_src1 = {CL_MEM_OBJECT_IMAGE1D_BUFFER, static_cast<size_t>(ne00 * max_post_router_tile * n_tile_size / 4), 0,0,0,0,0,0,0, {buf_src1_reordered}};
+ if (backend_ctx->kernel_gemm_moe_q4_1_f32_ns_bin) {
+ // bin kernel uses slightly different image format
+ image_format_buf_src1 = {CL_R, CL_FLOAT};
+ image_desc_buf_src1.image_width = static_cast<size_t>(ne00 * max_post_router_tile * n_tile_size);
+ }
image_src1_reordered = clCreateImage(backend_ctx->context, CL_MEM_READ_ONLY, &image_format_buf_src1, &image_desc_buf_src1, NULL, &status);
CL_CHECK(status);
} else { // for gemm
kernel = backend_ctx->kernel_gemm_moe_q4_k_f32_ns;
+ if (backend_ctx->kernel_gemm_moe_q4_k_f32_ns_bin) {
+ kernel = backend_ctx->kernel_gemm_moe_q4_k_f32_ns_bin;
+ }
// Reorder router if called from test-backend-ops or when new router is generated.
// Otherwise reuse the reordered result from previous mul_mat_id call.
CL_CHECK(status);
cl_image_format image_format_buf_src1 = {CL_RGBA, CL_FLOAT};
cl_image_desc image_desc_buf_src1 = {CL_MEM_OBJECT_IMAGE1D_BUFFER, static_cast<size_t>(ne00 * max_post_router_tile * n_tile_size / 4), 0,0,0,0,0,0,0, {buf_src1_reordered}};
+ if (backend_ctx->kernel_gemm_moe_q4_k_f32_ns_bin) {
+ // bin kernel uses slightly different image format
+ image_format_buf_src1 = {CL_R, CL_FLOAT};
+ image_desc_buf_src1.image_width = static_cast<size_t>(ne00 * max_post_router_tile * n_tile_size);
+ }
image_src1_reordered = clCreateImage(backend_ctx->context, CL_MEM_READ_ONLY, &image_format_buf_src1, &image_desc_buf_src1, NULL, &status);
CL_CHECK(status);
} else { // for gemm
kernel = backend_ctx->kernel_gemm_moe_mxfp4_f32_ns;
+ if (backend_ctx->kernel_gemm_moe_mxfp4_f32_ns_bin) {
+ kernel = backend_ctx->kernel_gemm_moe_mxfp4_f32_ns_bin;
+ }
// Reorder router if called from test-backend-ops or when new router is generated.
// Otherwise reuse the reordered result from previous mul_mat_id call.
cl_image_desc image_desc_buf_src1;
image_format_buf_src1 = {CL_RGBA, CL_FLOAT};
image_desc_buf_src1 = {CL_MEM_OBJECT_IMAGE1D_BUFFER, static_cast<size_t>(ne00 * max_post_router_tile * n_tile_size / 4), 0,0,0,0,0,0,0, {buf_src1_reordered}};
+ if (backend_ctx->kernel_gemm_moe_mxfp4_f32_ns_bin) {
+ // bin kernel uses slightly different image format
+ image_format_buf_src1 = {CL_R, CL_FLOAT};
+ image_desc_buf_src1.image_width = static_cast<size_t>(ne00 * max_post_router_tile * n_tile_size);
+ }
image_src1_reordered = clCreateImage(backend_ctx->context, CL_MEM_READ_ONLY, &image_format_buf_src1, &image_desc_buf_src1, NULL, &status);
CL_CHECK(status);
overview / pkg / ggml / sources / llama.cpp / commitdiff