Performance issues with Intel Core Ultra 9 285H

[andrey-savov]

Running llama-b5350-bin-ubuntu-x64.zip stock builds on Intel Core Ultra 9 285H laptop. Basically, runs at 0.5tps, while same runs at 20tps on Core i5-14600KF.

Is there anything I'm missing here?

Core i5 is older and supposedly slower. GPU runs fine, but that's beyond the point.

System info Core i5 (20tps):

system_info: n_threads = 10 (n_threads_batch = 10) / 20 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 
| CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

( I know it has CUDA, but isn't used in CPU only)

System info Core Ultra 9 (0.5tps):

system_info: n_threads = 16 (n_threads_batch = 16) / 16 
| CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

[andrey-savov]

Answering my own question, the reason is the number of threads is higher the number of P-cores + E-cores.

In 285H number of P-cores is 6, E-cores are 8 and LP-E cores are 2.

Judge for yourself:

❯ ./build/bin/llama-bench -t 14 -n 1 -m ~/Qwen2.5-7B-Instruct-Q6_K.gguf
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen2 7B Q6_K                  |   5.82 GiB |     7.62 B | CPU        |      14 |           pp512 |         30.84 ± 0.72 |
| qwen2 7B Q6_K                  |   5.82 GiB |     7.62 B | CPU        |      14 |             tg1 |         10.25 ± 1.76 |

build: e5c834f7 (5371)
❯ ./build/bin/llama-bench -t 6 -n 1 -m ~/Qwen2.5-7B-Instruct-Q6_K.gguf
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen2 7B Q6_K                  |   5.82 GiB |     7.62 B | CPU        |       6 |           pp512 |         17.78 ± 0.56 |
| qwen2 7B Q6_K                  |   5.82 GiB |     7.62 B | CPU        |       6 |             tg1 |          8.55 ± 0.27 |

build: e5c834f7 (5371)

The moment you cross the threshold (14):

❯ ./build/bin/llama-bench -t 15 -n 1 -m ~/Qwen2.5-7B-Instruct-Q6_K.gguf
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen2 7B Q6_K                  |   5.82 GiB |     7.62 B | CPU        |      15 |           pp512 |         21.54 ± 0.78 |
| qwen2 7B Q6_K                  |   5.82 GiB |     7.62 B | CPU        |      15 |             tg1 |          0.59 ± 0.01 |

build: e5c834f7 (5371)

Related discussion: #572, I did not restrict the cores manually, just set the number of threads.