]> git.djapps.eu Git - pkg/ggml/sources/ggml/commit
CANN: implement quantized MUL_MAT_ID for MoE models (llama/19228)
authorhipudding <redacted>
Tue, 10 Feb 2026 06:18:59 +0000 (14:18 +0800)
committerGeorgi Gerganov <redacted>
Sat, 14 Feb 2026 22:20:18 +0000 (00:20 +0200)
commit184b112a9a105156e63062a34a71f1ac040df344
tree59937ac8f3d891ec2a1ce47abcf13dff9311d97b
parentb07ff832320fe71c504ad056c53690fc4f25a837
CANN: implement quantized MUL_MAT_ID for MoE models (llama/19228)

Implement ggml_cann_mul_mat_id_quant function to support quantized matrix
multiplication for Mixture of Experts (MoE) architectures on CANN backend.

Key features:
- Support Q4_0 and Q8_0 quantized weight formats
- Use IndexSelect to dynamically route expert-specific weights based on indices
- Leverage WeightQuantBatchMatmulV2 for efficient quantized computation
- Handle automatic F16 type conversion for hardware compatibility
- Support both per-expert and broadcast input modes

Implementation details:
- Extract expert weights and scales using CANN IndexSelect operation
- Process each batch and expert combination independently
- Create proper tensor views with correct stride for matmul operations
- Automatic input/output type casting to/from F16 as needed

Testing: All test cases passed for supported types (F32, F16, Q4_0, Q8_0).
src/ggml-cann/aclnn_ops.cpp