Chat Completions Streaming — Prediction Guard

Generate chat completions based on a conversation history with streaming support.

Request

This endpoint expects an object.

modelstringRequired

The chat model to use for generating completions.

messagesstring or list of objectsRequired

max_completion_tokensintegerOptional

The maximum number of tokens in the generated completion.

temperaturedoubleOptional

The temperature parameter for controlling randomness in completions.

top_pdoubleOptional

The diversity of the generated text based on nucleus sampling.

top_kintegerOptional

The diversity of the generated text based on top-k sampling.

streambooleanOptionalDefaults to false

Turn streaming on.

inputobjectOptional

Options to affect the input of the request.

max_tokensintegerOptional

Deprecated. Please use max_completion_tokens.

Response

Successful response.

dataobjectOptional

Errors

1 curl -X POST https://api.predictionguard.com/chat/completions - Streaming Version \ 2 -H "Authorization: Bearer <token>" \ 3 -H "Content-Type: application/json" \ 4 -d '{ 5 "model": "Hermes-2-Pro-Llama-3-8B", 6 "messages": [ 7 { 8 "role": "user", 9 "content": "How do you feel about the world in general?" 10 } 11 ], 12 "max_completion_tokens": 1000, 13 "temperature": 1, 14 "top_p": 1, 15 "top_k": 50, 16 "stream": true, 17 "input": { 18 "pii": "replace", 19 "pii_replace_method": "random" 20 } 21 }'

1 [ 2 { 3 "data": { 4 "id": "chat-6c8c291f-5a06-46e5-94fa-2b0b1e6444c7", 5 "object": "chat.completion.chunk", 6 "created": 1727796455, 7 "model": "Hermes-2-Pro-Llama-3-8B", 8 "choices": [ 9 { 10 "index": 0, 11 "generated_text": "null", 12 "logprobs": 0, 13 "finish_reason": "null", 14 "delta": { 15 "content": " I" 16 } 17 } 18 ] 19 } 20 }, 21 { 22 "data": { 23 "id": "chat-6c8c291f-5a06-46e5-94fa-2b0b1e6444c7", 24 "object": "chat.completion.chunk", 25 "created": 1727796455, 26 "model": "Hermes-2-Pro-Llama-3-8B", 27 "choices": [ 28 { 29 "index": 0, 30 "generated_text": "I feel that the world is a complex and dynamic place. It has its fair share of challenges and difficulties, but also offers numerous opportunities for growth, learning, and connection. The world is constantly evolving, and as individuals, we must adapt and find ways to contribute positively to society. It's essential to maintain a balance between acknowledging the problems and celebrating the progress and achievements, while also remembering the importance of compassion, empathy, and understanding towards one another. Overall, I believe it's essential to stay optimistic and work together towards a better future for all.", 31 "logprobs": 0, 32 "finish_reason": "stop" 33 } 34 ] 35 } 36 } 37 ]

1	curl -X POST https://api.predictionguard.com/chat/completions - Streaming Version \
2	-H "Authorization: Bearer <token>" \
3	-H "Content-Type: application/json" \
4	-d '{
5	"model": "Hermes-2-Pro-Llama-3-8B",
6	"messages": [
7	{
8	"role": "user",
9	"content": "How do you feel about the world in general?"
10	}
11	],
12	"max_completion_tokens": 1000,
13	"temperature": 1,
14	"top_p": 1,
15	"top_k": 50,
16	"stream": true,
17	"input": {
18	"pii": "replace",
19	"pii_replace_method": "random"
20	}
21	}'

1	[
2	{
3	"data": {
4	"id": "chat-6c8c291f-5a06-46e5-94fa-2b0b1e6444c7",
5	"object": "chat.completion.chunk",
6	"created": 1727796455,
7	"model": "Hermes-2-Pro-Llama-3-8B",
8	"choices": [
9	{
10	"index": 0,
11	"generated_text": "null",
12	"logprobs": 0,
13	"finish_reason": "null",
14	"delta": {
15	"content": " I"
16	}
17	}
18	]
19	}
20	},
21	{
22	"data": {
23	"id": "chat-6c8c291f-5a06-46e5-94fa-2b0b1e6444c7",
24	"object": "chat.completion.chunk",
25	"created": 1727796455,
26	"model": "Hermes-2-Pro-Llama-3-8B",
27	"choices": [
28	{
29	"index": 0,
30	"generated_text": "I feel that the world is a complex and dynamic place. It has its fair share of challenges and difficulties, but also offers numerous opportunities for growth, learning, and connection. The world is constantly evolving, and as individuals, we must adapt and find ways to contribute positively to society. It's essential to maintain a balance between acknowledging the problems and celebrating the progress and achievements, while also remembering the importance of compassion, empathy, and understanding towards one another. Overall, I believe it's essential to stay optimistic and work together towards a better future for all.",
31	"logprobs": 0,
32	"finish_reason": "stop"
33	}
34	]
35	}
36	}
37	]