Is there a way to cache multiple prompt prefixes?

[ghnp5]

Hey!

Is it possible to have multiple caches of prompt prefixes?

In my case, I will have about 8 prompt prefixes that will be rotating all the time. This makes cache_prompt mostly useless.

Is there a way to cache 8 variations of the prompt prefixes? (while still allowing me to inject suffixes that will always be different, and not expected to be cached)

Many thanks!

[ggerganov]

You can use -np 8. Don't forget to scale the context size. For example:

# before
llama-server -c 1024 ...

# after
llama-server -c 8192 -np 8 ...

This will automatically assign requests to the slot that matches the prefix the most. However the generation speed will be currently impacted negatively due to some implementation limitations.

[ghnp5]

Hey! Thanks for the reply.

Apologies - I'm having some trouble to understand how -np can help me here.

Please note I'm not running multiple inferences simultaneously. As I'm running this on a CPU, I'm running only 1 completion at a time. The problem is that once I rotate to the next prompt prefix, the previously cached prompt won't be used.

Does that make sense? Do you think -np will help?
Thanks again.