Tokenize | Prediction Guard

curl -X POST https://{your-pg.api-domain}.com/tokenize \
     -H "Authorization: Bearer <token>" \
     -H "Content-Type: application/json" \
     -d '{
  "model": "gpt-oss-120b",
  "input": "Tell me a joke."
}'

{
  "id": "token-ab046fcf-945f-421c-b9f0-1c75ff355203",
  "object": "tokens",
  "created": 1729871708,
  "model": "gpt-oss-120b",
  "tokens": [
    {
      "id": 1,
      "start": 0,
      "stop": 0,
      "text": ""
    },
    {
      "id": 851,
      "start": 0,
      "stop": 4,
      "text": "This"
    },
    {
      "id": 349,
      "start": 4,
      "stop": 7,
      "text": " is"
    },
    {
      "id": 264,
      "start": 7,
      "stop": 9,
      "text": " a"
    },
    {
      "id": 6029,
      "start": 9,
      "stop": 15,
      "text": " token"
    },
    {
      "id": 653,
      "start": 15,
      "stop": 18,
      "text": "ize"
    },
    {
      "id": 2757,
      "start": 18,
      "stop": 26,
      "text": " example"
    },
    {
      "id": 28723,
      "start": 26,
      "stop": 27,
      "text": "."
    }
  ]
}

Generates tokens for a string using a model's tokenizer.

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Request

This endpoint expects an object.

modelstringRequired

The name of the model to use for tokenization.

inputstringRequired

The text to tokenize.

Response

Successful response.

idstring

Unique ID for the token.

objectstring

Type of object (tokens).

createdinteger

Timestamp of when the tokens were generated.

modelstring

The name of the model used.

tokenslist of objects

An array of token data.

Errors

400

Bad Request Error

403

Forbidden Error

$	curl -X POST https://{your-pg.api-domain}.com/tokenize \
>	-H "Authorization: Bearer <token>" \
>	-H "Content-Type: application/json" \
>	-d '{
>	"model": "gpt-oss-120b",
>	"input": "Tell me a joke."
>	}'

1	{
2	"id": "token-ab046fcf-945f-421c-b9f0-1c75ff355203",
3	"object": "tokens",
4	"created": 1729871708,
5	"model": "gpt-oss-120b",
6	"tokens": [
7	{
8	"id": 1,
9	"start": 0,
10	"stop": 0,
11	"text": ""
12	},
13	{
14	"id": 851,
15	"start": 0,
16	"stop": 4,
17	"text": "This"
18	},
19	{
20	"id": 349,
21	"start": 4,
22	"stop": 7,
23	"text": " is"
24	},
25	{
26	"id": 264,
27	"start": 7,
28	"stop": 9,
29	"text": " a"
30	},
31	{
32	"id": 6029,
33	"start": 9,
34	"stop": 15,
35	"text": " token"
36	},
37	{
38	"id": 653,
39	"start": 15,
40	"stop": 18,
41	"text": "ize"
42	},
43	{
44	"id": 2757,
45	"start": 18,
46	"stop": 26,
47	"text": " example"
48	},
49	{
50	"id": 28723,
51	"start": 26,
52	"stop": 27,
53	"text": "."
54	}
55	]
56	}