Tokenize

POST

Generates tokens for a string using a model’s tokenizer.

Request

This endpoint expects an object.

modelstringRequired

The name of the model to use for tokenization.

inputstringRequired

The text to tokenize.

Response

Successful response.

idstringOptional

Unique ID for the token.

objectstringOptional

Type of object (tokens).

createdintegerOptional

Timestamp of when the tokens were generated.

modelstringOptional

The name of the model used.

tokenslist of objectsOptional

An array of token data.

Errors

POST

1 curl -X POST https://api.predictionguard.com/tokenize \
2      -H "Authorization: Bearer <token>" \
3      -H "Content-Type: application/json" \
4      -d '{
5   "model": "Hermes-3-Llama-3.1-8B",
6   "input": "Tell me a joke."
7 }'

1 {
2   "id": "token-ab046fcf-945f-421c-b9f0-1c75ff355203",
3   "object": "tokens",
4   "created": 1729871708,
5   "model": "Hermes-2-Pro-Mistral-7B",
6   "tokens": [
7     {
8       "id": 1,
9       "start": 0,
10       "stop": 0,
11       "text": ""
12     },
13     {
14       "id": 851,
15       "start": 0,
16       "stop": 4,
17       "text": "This"
18     },
19     {
20       "id": 349,
21       "start": 4,
22       "stop": 7,
23       "text": " is"
24     },
25     {
26       "id": 264,
27       "start": 7,
28       "stop": 9,
29       "text": " a"
30     },
31     {
32       "id": 6029,
33       "start": 9,
34       "stop": 15,
35       "text": " token"
36     },
37     {
38       "id": 653,
39       "start": 15,
40       "stop": 18,
41       "text": "ize"
42     },
43     {
44       "id": 2757,
45       "start": 18,
46       "stop": 26,
47       "text": " example"
48     },
49     {
50       "id": 28723,
51       "start": 26,
52       "stop": 27,
53       "text": "."
54     }
55   ]
56 }

Built with

1	curl -X POST https://api.predictionguard.com/tokenize \
2	-H "Authorization: Bearer <token>" \
3	-H "Content-Type: application/json" \
4	-d '{
5	"model": "Hermes-3-Llama-3.1-8B",
6	"input": "Tell me a joke."
7	}'

1	{
2	"id": "token-ab046fcf-945f-421c-b9f0-1c75ff355203",
3	"object": "tokens",
4	"created": 1729871708,
5	"model": "Hermes-2-Pro-Mistral-7B",
6	"tokens": [
7	{
8	"id": 1,
9	"start": 0,
10	"stop": 0,
11	"text": ""
12	},
13	{
14	"id": 851,
15	"start": 0,
16	"stop": 4,
17	"text": "This"
18	},
19	{
20	"id": 349,
21	"start": 4,
22	"stop": 7,
23	"text": " is"
24	},
25	{
26	"id": 264,
27	"start": 7,
28	"stop": 9,
29	"text": " a"
30	},
31	{
32	"id": 6029,
33	"start": 9,
34	"stop": 15,
35	"text": " token"
36	},
37	{
38	"id": 653,
39	"start": 15,
40	"stop": 18,
41	"text": "ize"
42	},
43	{
44	"id": 2757,
45	"start": 18,
46	"stop": 26,
47	"text": " example"
48	},
49	{
50	"id": 28723,
51	"start": 26,
52	"stop": 27,
53	"text": "."
54	}
55	]
56	}