How to properly use llama-server bench

[Allan-Luu]

Hello, I wanted to know if someone recently has gotten llama-server bench working. I'm running into a lot of issues when attempting to follow the example.

When the server is running, to properly get a response I need to post to url <address>:<port>/completion, but in the llama.cpp/examples/server/bench/script.js line 104, it adds a suffixs of /chat/completions, which I'm assuming may be outdated?

const res = sse.open(`${server_url}/chat/completions`, params, function (client) {

So when updating that to remove the suffix, and changing the post json body removing the message key, and using a prompt key, I run into chunk handling errors from line 111. From the example's script.js:

let chunk = JSON.parse(event.data)
let choice = chunk.choices[0]
if (choice.finish_reason) {
    finish_reason = choice.finish_reason
}

if (chunk.usage) {
    prompt_tokens = chunk.usage.prompt_tokens
    llamacpp_prompt_tokens.add(prompt_tokens)
    llamacpp_prompt_tokens_total_counter.add(prompt_tokens)

    completions_tokens = chunk.usage.completion_tokens
    llamacpp_completion_tokens.add(completions_tokens)
    llamacpp_completion_tokens_total_counter.add(completions_tokens)
}

The response from my server is in format when posting a prompt to {server_url}/completion :

{
    "content": < string > ,
    "stop": < bool >,
    "id_slot": < int >,
    "multimodal": < bool > ,
    "index": < int >
}

and this format doesn't have any .usage .choices and so on.

I'd really like to get this to work, so any advice would be greatly appreciated! Thank you in advance.

[Allan-Luu]

my bad, figured out that I needed to add the /v1 suffix to my url for the script.js to properly formulate the full url to post to.