Tokenize | Prediction Guard

curl -X POST https://{your-pg.api-domain}.com/tokenize \
     -H "Authorization: Bearer <token>" \
     -H "Content-Type: application/json" \
     -d '{
  "model": "{{TEXT_MODEL}}",
  "input": "Tell me a joke."
}'

{
  "id": "token-ab046fcf-945f-421c-b9f0-1c75ff355203",
  "object": "tokens",
  "created": 1729871708,
  "model": "{{TEXT_MODEL}}",
  "tokens": [
    {
      "id": 1,
      "start": 0,
      "stop": 0,
      "text": ""
    },
    {
      "id": 851,
      "start": 0,
      "stop": 4,
      "text": "This"
    },
    {
      "id": 349,
      "start": 4,
      "stop": 7,
      "text": " is"
    },
    {
      "id": 264,
      "start": 7,
      "stop": 9,
      "text": " a"
    },
    {
      "id": 6029,
      "start": 9,
      "stop": 15,
      "text": " token"
    },
    {
      "id": 653,
      "start": 15,
      "stop": 18,
      "text": "ize"
    },
    {
      "id": 2757,
      "start": 18,
      "stop": 26,
      "text": " example"
    },
    {
      "id": 28723,
      "start": 26,
      "stop": 27,
      "text": "."
    }
  ]
}

Generates tokens for a string using a model's tokenizer.

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Request

This endpoint expects an object.

modelstringRequired

The name of the model to use for tokenization.

inputstringRequired

The text to tokenize.

Response

Successful response.

idstringOptional

Unique ID for the token.

objectstringOptional

Type of object (tokens).

createdintegerOptional

Timestamp of when the tokens were generated.

modelstringOptional

The name of the model used.

tokenslist of objectsOptional

An array of token data.

Errors

400

Bad Request Error

403

Forbidden Error