-# whisper.cpp for SYCL\r
-\r
-[Background](#background)\r
-\r
-[OS](#os)\r
-\r
-[Intel GPU](#intel-gpu)\r
-\r
-[Linux](#linux)\r
-\r
-[Environment Variable](#environment-variable)\r
-\r
-[Known Issue](#known-issue)\r
-\r
-[Todo](#todo)\r
-\r
-## Background\r
-\r
-SYCL is a higher-level programming model to improve programming productivity on various hardware accelerators\97such as CPUs, GPUs, and FPGAs. It is a single-source embedded domain-specific language based on pure C++17.\r
-\r
-oneAPI is a specification that is open and standards-based, supporting multiple architecture types including but not limited to GPU, CPU, and FPGA. The spec has both direct programming and API-based programming paradigms.\r
-\r
-Intel uses the SYCL as direct programming language to support CPU, GPUs and FPGAs.\r
-\r
-To avoid re-inventing the wheel, this code refers other code paths in llama.cpp (like OpenBLAS, cuBLAS, CLBlast). We use a open-source tool [SYCLomatic](https://github.com/oneapi-src/SYCLomatic) (Commercial release [Intel® DPC++ Compatibility Tool](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compatibility-tool.html)) migrate to SYCL.\r
-\r
-The whisper.cpp for SYCL is used to support Intel GPUs.\r
-\r
-For Intel CPU, recommend to use whisper.cpp for X86 (Intel MKL build).\r
-\r
-## OS\r
-\r
-|OS|Status|Verified|\r
-|-|-|-|\r
-|Linux|Support|Ubuntu 22.04|\r
-|Windows|Ongoing| |\r
-\r
-\r
-## Intel GPU\r
-\r
-|Intel GPU| Status | Verified Model|\r
-|-|-|-|\r
-|Intel Data Center Max Series| Support| Max 1550|\r
-|Intel Data Center Flex Series| Support| Flex 170|\r
-|Intel Arc Series| Support| Arc 770|\r
-|Intel built-in Arc GPU| Support| built-in Arc GPU in Meteor Lake|\r
-|Intel iGPU| Support| iGPU in i5-1250P, i7-1165G7|\r
-\r
-\r
-## Linux\r
-\r
-### Setup Environment\r
-\r
-1. Install Intel GPU driver.\r
-\r
-a. Please install Intel GPU driver by official guide: [Install GPU Drivers](https://dgpu-docs.intel.com/driver/installation.html).\r
-\r
-Note: for iGPU, please install the client GPU driver.\r
-\r
-b. Add user to group: video, render.\r
-\r
-```\r
-sudo usermod -aG render username\r
-sudo usermod -aG video username\r
-```\r
-\r
-Note: re-login to enable it.\r
-\r
-c. Check\r
-\r
-```\r
-sudo apt install clinfo\r
-sudo clinfo -l\r
-```\r
-\r
-Output (example):\r
-\r
-```\r
-Platform #0: Intel(R) OpenCL Graphics\r
- `-- Device #0: Intel(R) Arc(TM) A770 Graphics\r
-\r
-\r
-Platform #0: Intel(R) OpenCL HD Graphics\r
- `-- Device #0: Intel(R) Iris(R) Xe Graphics [0x9a49]\r
-```\r
-\r
-2. Install Intel® oneAPI Base toolkit.\r
-\r
-\r
-a. Please follow the procedure in [Get the Intel® oneAPI Base Toolkit ](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html).\r
-\r
-Recommend to install to default folder: **/opt/intel/oneapi**.\r
-\r
-Following guide use the default folder as example. If you use other folder, please modify the following guide info with your folder.\r
-\r
-b. Check\r
-\r
-```\r
-source /opt/intel/oneapi/setvars.sh\r
-\r
-sycl-ls\r
-```\r
-\r
-There should be one or more level-zero devices. Like **[ext_oneapi_level_zero:gpu:0]**.\r
-\r
-Output (example):\r
-```\r
-[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2 [2023.16.10.0.17_160000]\r
-[opencl:cpu:1] Intel(R) OpenCL, 13th Gen Intel(R) Core(TM) i7-13700K OpenCL 3.0 (Build 0) [2023.16.10.0.17_160000]\r
-[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics OpenCL 3.0 NEO [23.30.26918.50]\r
-[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Arc(TM) A770 Graphics 1.3 [1.3.26918]\r
-\r
-```\r
-\r
-2. Build locally:\r
-\r
-```\r
-mkdir -p build\r
-cd build\r
-source /opt/intel/oneapi/setvars.sh\r
-\r
-#for FP16\r
-#cmake .. -DWHISPER_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DWHISPER_SYCL_F16=ON \r
-\r
-#for FP32\r
-cmake .. -DWHISPER_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx\r
-\r
-#build example/main only\r
-#cmake --build . --config Release --target main\r
-\r
-#build all binary\r
-cmake --build . --config Release -v\r
-\r
-```\r
-\r
-or\r
-\r
-```\r
-./examples/sycl/build.sh\r
-```\r
-\r
-Note:\r
-\r
-- By default, it will build for all binary files. It will take more time. To reduce the time, we recommend to build for **example/main** only.\r
-\r
-### Run\r
-\r
-1. Put model file to folder **models**\r
-\r
-2. Enable oneAPI running environment\r
-\r
-```\r
-source /opt/intel/oneapi/setvars.sh\r
-```\r
-\r
-3. List device ID\r
-\r
-Run without parameter:\r
-\r
-```\r
-./build/bin/ls-sycl-device\r
-\r
-or\r
-\r
-./build/bin/main\r
-```\r
-\r
-Check the ID in startup log, like:\r
-\r
-```\r
-found 4 SYCL devices:\r
- Device 0: Intel(R) Arc(TM) A770 Graphics, compute capability 1.3,\r
- max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136\r
- Device 1: Intel(R) FPGA Emulation Device, compute capability 1.2,\r
- max compute_units 24, max work group size 67108864, max sub group size 64, global mem size 67065057280\r
- Device 2: 13th Gen Intel(R) Core(TM) i7-13700K, compute capability 3.0,\r
- max compute_units 24, max work group size 8192, max sub group size 64, global mem size 67065057280\r
- Device 3: Intel(R) Arc(TM) A770 Graphics, compute capability 3.0,\r
- max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136\r
-\r
-```\r
-\r
-|Attribute|Note|\r
-|-|-|\r
-|compute capability 1.3|Level-zero running time, recommended |\r
-|compute capability 3.0|OpenCL running time, slower than level-zero in most cases|\r
-\r
-4. Set device ID and execute whisper.cpp\r
-\r
-Set device ID = 0 by **GGML_SYCL_DEVICE=0**\r
-\r
-```\r
-GGML_SYCL_DEVICE=0 ./build/bin/main -m models/ggml-base.en.bin -f samples/jfk.wav\r
-```\r
-or run by script:\r
-\r
-```\r
-./examples/sycl/run_whisper.sh\r
-```\r
-\r
-\r
-\r
-5. Check the device ID in output\r
-\r
-Like:\r
-```\r
-Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device\r
-```\r
-\r
-\r
-## Environment Variable\r
-\r
-#### Build\r
-\r
-|Name|Value|Function|\r
-|-|-|-|\r
-|WHISPER_SYCL|ON (mandatory)|Enable build with SYCL code path. <br>For FP32/FP16, WHISPER_SYCL=ON is mandatory.|\r
-|WHISPER_SYCL_F16|ON (optional)|Enable FP16 build with SYCL code path.For FP32, do not set it.|\r
-|CMAKE_C_COMPILER|icx|Use icx compiler for SYCL code path|\r
-|CMAKE_CXX_COMPILER|icpx|use icpx for SYCL code path|\r
-\r
-#### Running\r
-\r
-\r
-|Name|Value|Function|\r
-|-|-|-|\r
-|GGML_SYCL_DEVICE|0 (default) or 1|Set the device id used. Check the device ids by default running output|\r
-|GGML_SYCL_DEBUG|0 (default) or 1|Enable log function by macro: GGML_SYCL_DEBUG|\r
-\r
-## Known Issue\r
-\r
-- Error: `error while loading shared libraries: libsycl.so.7: cannot open shared object file: No such file or directory`.\r
-\r
- Miss to enable oneAPI running environment.\r
-\r
- Install oneAPI base toolkit and enable it by: `source /opt/intel/oneapi/setvars.sh`.\r
-\r
-\r
-- Hang during startup\r
-\r
- llama.cpp use mmap as default way to read model file and copy to GPU. In some system, memcpy will be abnormal and block.\r
-\r
- Solution: add **--no-mmap**.\r
-\r
-## Todo\r
-\r
-- Support to build in Windows.\r
-\r
-- Support multiple cards.
\ No newline at end of file
+# whisper.cpp for SYCL
+
+[Background](#background)
+
+[OS](#os)
+
+[Intel GPU](#intel-gpu)
+
+[Linux](#linux)
+
+[Environment Variable](#environment-variable)
+
+[Known Issue](#known-issue)
+
+[Todo](#todo)
+
+## Background
+
+SYCL is a higher-level programming model to improve programming productivity on various hardware accelerators—such as CPUs, GPUs, and FPGAs. It is a single-source embedded domain-specific language based on pure C++17.
+
+oneAPI is a specification that is open and standards-based, supporting multiple architecture types including but not limited to GPU, CPU, and FPGA. The spec has both direct programming and API-based programming paradigms.
+
+Intel uses the SYCL as direct programming language to support CPU, GPUs and FPGAs.
+
+To avoid re-inventing the wheel, this code refers other code paths in llama.cpp (like OpenBLAS, cuBLAS, CLBlast). We use a open-source tool [SYCLomatic](https://github.com/oneapi-src/SYCLomatic) (Commercial release [Intel® DPC++ Compatibility Tool](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compatibility-tool.html)) migrate to SYCL.
+
+The whisper.cpp for SYCL is used to support Intel GPUs.
+
+For Intel CPU, recommend to use whisper.cpp for X86 (Intel MKL build).
+
+## OS
+
+|OS|Status|Verified|
+|-|-|-|
+|Linux|Support|Ubuntu 22.04|
+|Windows|Ongoing| |
+
+
+## Intel GPU
+
+|Intel GPU| Status | Verified Model|
+|-|-|-|
+|Intel Data Center Max Series| Support| Max 1550|
+|Intel Data Center Flex Series| Support| Flex 170|
+|Intel Arc Series| Support| Arc 770|
+|Intel built-in Arc GPU| Support| built-in Arc GPU in Meteor Lake|
+|Intel iGPU| Support| iGPU in i5-1250P, i7-1165G7|
+
+
+## Linux
+
+### Setup Environment
+
+1. Install Intel GPU driver.
+
+a. Please install Intel GPU driver by official guide: [Install GPU Drivers](https://dgpu-docs.intel.com/driver/installation.html).
+
+Note: for iGPU, please install the client GPU driver.
+
+b. Add user to group: video, render.
+
+```
+sudo usermod -aG render username
+sudo usermod -aG video username
+```
+
+Note: re-login to enable it.
+
+c. Check
+
+```
+sudo apt install clinfo
+sudo clinfo -l
+```
+
+Output (example):
+
+```
+Platform #0: Intel(R) OpenCL Graphics
+ `-- Device #0: Intel(R) Arc(TM) A770 Graphics
+
+
+Platform #0: Intel(R) OpenCL HD Graphics
+ `-- Device #0: Intel(R) Iris(R) Xe Graphics [0x9a49]
+```
+
+2. Install Intel® oneAPI Base toolkit.
+
+
+a. Please follow the procedure in [Get the Intel® oneAPI Base Toolkit ](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html).
+
+Recommend to install to default folder: **/opt/intel/oneapi**.
+
+Following guide use the default folder as example. If you use other folder, please modify the following guide info with your folder.
+
+b. Check
+
+```
+source /opt/intel/oneapi/setvars.sh
+
+sycl-ls
+```
+
+There should be one or more level-zero devices. Like **[ext_oneapi_level_zero:gpu:0]**.
+
+Output (example):
+```
+[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2 [2023.16.10.0.17_160000]
+[opencl:cpu:1] Intel(R) OpenCL, 13th Gen Intel(R) Core(TM) i7-13700K OpenCL 3.0 (Build 0) [2023.16.10.0.17_160000]
+[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics OpenCL 3.0 NEO [23.30.26918.50]
+[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Arc(TM) A770 Graphics 1.3 [1.3.26918]
+
+```
+
+2. Build locally:
+
+```
+mkdir -p build
+cd build
+source /opt/intel/oneapi/setvars.sh
+
+#for FP16
+#cmake .. -DWHISPER_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DWHISPER_SYCL_F16=ON
+
+#for FP32
+cmake .. -DWHISPER_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx
+
+#build example/main only
+#cmake --build . --config Release --target main
+
+#build all binary
+cmake --build . --config Release -v
+
+```
+
+or
+
+```
+./examples/sycl/build.sh
+```
+
+Note:
+
+- By default, it will build for all binary files. It will take more time. To reduce the time, we recommend to build for **example/main** only.
+
+### Run
+
+1. Put model file to folder **models**
+
+2. Enable oneAPI running environment
+
+```
+source /opt/intel/oneapi/setvars.sh
+```
+
+3. List device ID
+
+Run without parameter:
+
+```
+./build/bin/ls-sycl-device
+
+or
+
+./build/bin/main
+```
+
+Check the ID in startup log, like:
+
+```
+found 4 SYCL devices:
+ Device 0: Intel(R) Arc(TM) A770 Graphics, compute capability 1.3,
+ max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136
+ Device 1: Intel(R) FPGA Emulation Device, compute capability 1.2,
+ max compute_units 24, max work group size 67108864, max sub group size 64, global mem size 67065057280
+ Device 2: 13th Gen Intel(R) Core(TM) i7-13700K, compute capability 3.0,
+ max compute_units 24, max work group size 8192, max sub group size 64, global mem size 67065057280
+ Device 3: Intel(R) Arc(TM) A770 Graphics, compute capability 3.0,
+ max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136
+
+```
+
+|Attribute|Note|
+|-|-|
+|compute capability 1.3|Level-zero running time, recommended |
+|compute capability 3.0|OpenCL running time, slower than level-zero in most cases|
+
+4. Set device ID and execute whisper.cpp
+
+Set device ID = 0 by **GGML_SYCL_DEVICE=0**
+
+```
+GGML_SYCL_DEVICE=0 ./build/bin/main -m models/ggml-base.en.bin -f samples/jfk.wav
+```
+or run by script:
+
+```
+./examples/sycl/run_whisper.sh
+```
+
+
+
+5. Check the device ID in output
+
+Like:
+```
+Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device
+```
+
+
+## Environment Variable
+
+#### Build
+
+|Name|Value|Function|
+|-|-|-|
+|WHISPER_SYCL|ON (mandatory)|Enable build with SYCL code path. <br>For FP32/FP16, WHISPER_SYCL=ON is mandatory.|
+|WHISPER_SYCL_F16|ON (optional)|Enable FP16 build with SYCL code path.For FP32, do not set it.|
+|CMAKE_C_COMPILER|icx|Use icx compiler for SYCL code path|
+|CMAKE_CXX_COMPILER|icpx|use icpx for SYCL code path|
+
+#### Running
+
+
+|Name|Value|Function|
+|-|-|-|
+|GGML_SYCL_DEVICE|0 (default) or 1|Set the device id used. Check the device ids by default running output|
+|GGML_SYCL_DEBUG|0 (default) or 1|Enable log function by macro: GGML_SYCL_DEBUG|
+
+## Known Issue
+
+- Error: `error while loading shared libraries: libsycl.so.7: cannot open shared object file: No such file or directory`.
+
+ Miss to enable oneAPI running environment.
+
+ Install oneAPI base toolkit and enable it by: `source /opt/intel/oneapi/setvars.sh`.
+
+
+- Hang during startup
+
+ llama.cpp use mmap as default way to read model file and copy to GPU. In some system, memcpy will be abnormal and block.
+
+ Solution: add **--no-mmap**.
+
+## Todo
+
+- Support to build in Windows.
+
+- Support multiple cards.
overview / pkg / ggml / sources / whisper.cpp / commitdiff