Chat Completions with Streaming

1 curl -X POST https://api.predictionguard.com/chat/completions with Streaming \ 2 -H "Authorization: Bearer <token>" \ 3 -H "Content-Type: application/json" \ 4 -d '{ 5 "model": "Hermes-2-Pro-Llama-3-8B", 6 "messages": [ 7 { 8 "role": "user", 9 "content": "How do you feel about the world in general?" 10 } 11 ], 12 "max_completion_tokens": 1000, 13 "frequency_penalty": 0.1, 14 "logit_bias": { 15 "128000": 10 16 }, 17 "presence_penalty": 0.1, 18 "stop": "hello", 19 "temperature": 1, 20 "top_p": 1, 21 "top_k": 50, 22 "stream": true, 23 "input": { 24 "pii": "replace", 25 "pii_replace_method": "random" 26 } 27 }'

1 [ 2 { 3 "data": { 4 "id": "chat-6c8c291f-5a06-46e5-94fa-2b0b1e6444c7", 5 "object": "chat.completion.chunk", 6 "created": 1727796455, 7 "model": "Hermes-2-Pro-Llama-3-8B", 8 "choices": [ 9 { 10 "index": 0, 11 "delta": { 12 "content": " I" 13 }, 14 "generated_text": "null", 15 "finish_reason": "null" 16 } 17 ] 18 } 19 }, 20 { 21 "data": { 22 "id": "chat-6c8c291f-5a06-46e5-94fa-2b0b1e6444c7", 23 "object": "chat.completion.chunk", 24 "created": 1727796455, 25 "model": "Hermes-2-Pro-Llama-3-8B", 26 "choices": [ 27 { 28 "index": 0, 29 "delta": { 30 "content": " feel" 31 }, 32 "generated_text": "null", 33 "finish_reason": "length" 34 } 35 ] 36 } 37 } 38 ]

Generate chat completions based on a conversation history with streaming support.

Request

This endpoint expects an object.

modelstringRequired

The chat model to use for generating completions.

messagesstring or list of objectsRequired

frequency_penaltydoubleOptional

A value between -2.0 and 2.0, with positive values increasingly penalizing new tokens based on their frequency so far in order to decrease further occurrences.

logit_biasanyOptional

Modifies the likelihood of specified tokens appearing in a response.

max_completion_tokensintegerOptional

The maximum number of tokens in the generated completion.

presence_penaltydoubleOptional

A value between -2.0 and 2.0, with positive values causing a flat reduction of new tokens based on their existing presence so far in order to decrease further occurrences.

stopstring or list of stringsOptional

temperaturedoubleOptional

The temperature parameter for controlling randomness in completions.

top_pdoubleOptional

The diversity of the generated text based on nucleus sampling.

top_kintegerOptional

The diversity of the generated text based on top-k sampling.

streambooleanOptional

Turn streaming on.

inputobjectOptional

Options to affect the input of the request.

max_tokensintegerOptional

Deprecated. Please use max_completion_tokens.

Response

Successful response.

dataobjectOptional

Headers

Request

Response

Errors