Using LLMs

Tool Calling

Invoke external functions from PredictionGuard’s /chat/completions endpoint

Please use the /models endpoint to see which models support tool calling.

Overview

Tool calling extends your model’s capabilities beyond static text generation. With tool calling, models can:

  • Request real-time information (weather, stock prices)
  • Execute business workflows (create tickets, send emails)
  • Chain multiple operations together
  • Integrate with your existing APIs and services

Core Concepts

Tools

Functions you expose to the model, defined using JSON Schema. Each tool includes:

  • Name: Unique identifier for the function
  • Description: Clear explanation of what the tool does
  • Parameters: Expected inputs with types and constraints

Tool Calls

When the model determines a tool would help answer a query, it generates a tool call with:

  • The tool name to invoke
  • Arguments formatted as JSON
  • A unique call ID for tracking

Tool Choice Strategy

Controls how the model decides whether to use tools:

StrategyTypeBehaviorUse Case
"auto"stringModel decides if tools would helpGeneral assistants
"none"stringTools disabled for this requestPure text generation
"required"stringMust use at least one toolData-dependent queries

Quick Start

1. Define Your Tool

Create a JSON schema describing your function:

1{
2  "type": "function",
3  "function": {
4    "name": "get_current_weather",
5    "description": "Get current weather conditions for a specific location",
6    "parameters": {
7      "type": "object",
8      "properties": {
9        "location": {
10          "type": "string",
11          "description": "City and state, e.g., 'San Francisco, CA'"
12        },
13        "unit": {
14          "type": "string",
15          "enum": ["celsius", "fahrenheit"],
16          "description": "Temperature unit preference"
17        }
18      },
19      "required": ["location"],
20      "additionalProperties": false
21    },
22    "strict": false
23  }
24}

Write clear, specific descriptions for both the tool and its parameters. This helps the model understand when and how to use your tools correctly. Set strict: true to enforce exact schema adherence.

2. Make a Request with Tools

Include your tool definitions in the API request:

$curl -X POST https://api.predictionguard.com/chat/completions \
>  -H "Authorization: Bearer $PREDICTIONGUARD_API_KEY" \
>  -H "Content-Type: application/json" \
>  -d '{
>    "model": "Hermes-3-Llama-3.1-70B",
>    "messages": [
>      {
>        "role": "user",
>        "content": "What'\''s the weather like in San Francisco?"
>      }
>    ],
>    "tools": [
>      {
>        "type": "function",
>        "function": {
>          "name": "get_current_weather",
>          "description": "Get current weather conditions for a specific location",
>          "parameters": {
>            "type": "object",
>            "properties": {
>              "location": {
>                "type": "string",
>                "description": "City and state, e.g., '\''San Francisco, CA'\''"
>              },
>              "unit": {
>                "type": "string",
>                "enum": ["celsius", "fahrenheit"]
>              }
>            },
>            "required": ["location"]
>          },
>          "strict": false
>        }
>      }
>    ],
>    "tool_choice": "auto"
>  }'

3. Handle the Tool Call

When the model requests a tool, you’ll receive:

1{
2  "id": "chatcmpl-123",
3  "choices": [{
4    "index": 0,
5    "message": {
6      "role": "assistant",
7      "content": null,
8      "tool_calls": [{
9        "id": "call_abc123",
10        "type": "function",
11        "function": {
12          "name": "get_current_weather",
13          "arguments": "{\"location\":\"San Francisco, CA\",\"unit\":\"fahrenheit\"}"
14        }
15      }]
16    },
17    "finish_reason": "tool_calls"
18  }]
19}

4. Execute and Return Results

Parse the arguments, execute your function, and send back the result:

1# Parse the tool call
2import json
3tool_call = response["choices"][0]["message"]["tool_calls"][0]
4args = json.loads(tool_call["function"]["arguments"])
5
6# Execute your function
7weather_data = get_weather(args["location"], args.get("unit", "fahrenheit"))
8
9# Continue the conversation with the result
10messages.append({
11    "role": "assistant",
12    "content": None,
13    "tool_calls": [tool_call]
14})
15messages.append({
16    "role": "tool",
17    "tool_call_id": tool_call["id"],
18    "name": tool_call["function"]["name"],
19    "content": json.dumps(weather_data)
20})

5. Get the Final Response

The model incorporates the tool result and provides a natural language response:

1{
2  "role": "assistant",
3  "content": "The current weather in San Francisco is 68°F with partly cloudy skies. It's a pleasant day with light winds from the west at 10 mph."
4}

Best Practices

Tool Design

Keep tools focused: Each tool should have a single, clear purpose

1✅ get_user_profile(user_id)
2❌ get_user_data_and_posts_and_friends(user_id)

Use explicit parameters: Avoid ambiguous or overly flexible inputs

1✅ "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
2❌ "unit": {"type": "string", "description": "any temperature unit"}

Provide clear descriptions: Help the model understand when to use each tool

1✅ "description": "Retrieves real-time stock price for a given ticker symbol"
2❌ "description": "Gets data"

Error Handling

Build robust error handling for production use:

1try:
2    # Parse arguments safely
3    args = json.loads(tool_call["function"]["arguments"])
4    
5    # Validate required parameters
6    if "location" not in args:
7        raise ValueError("Missing required parameter: location")
8    
9    # Execute with timeout
10    result = await asyncio.wait_for(
11        get_weather(args["location"]), 
12        timeout=5.0
13    )
14    
15except json.JSONDecodeError:
16    result = {"error": "Invalid arguments provided"}
17except asyncio.TimeoutError:
18    result = {"error": "Weather service timeout"}
19except Exception as e:
20    result = {"error": f"Service error: {str(e)}"}

Security Considerations

Validate all inputs: Never trust model-generated arguments

1# Sanitize and validate
2location = args.get("location", "").strip()
3if not location or len(location) > 100:
4    raise ValueError("Invalid location")

Check for prompt injections: Use PredictionGuard’s injection detection API

1# Check user input for potential prompt injection
2injection_check = client.injection.check(
3    prompt=user_message,
4    detect=True
5)
6
7if injection_check["checks"][0]["probability"] > 0.5:
8    # High probability of injection attack
9    return {"error": "Invalid input detected"}

PredictionGuard provides a dedicated Injection API to detect prompt injection attacks. Consider checking user inputs before processing them with tool-enabled models to prevent malicious attempts to manipulate tool behavior.

Implement access controls: Verify permissions before execution

1if not user_has_permission(user_id, "weather_api"):
2    return {"error": "Access denied"}

Never expose sensitive data: Keep API keys and secrets server-side

1# ❌ Don't include in tool results
2{"api_key": "sk-123", "temperature": 72}
3
4# ✅ Return only necessary data
5{"temperature": 72, "condition": "sunny"}

Troubleshooting

Model Skips Required Tools

If tool_choice: "auto" isn’t working:

  • Add explicit instructions in the user message
  • Use named tool forcing
  • Ensure tool descriptions are clear

Invalid Arguments

Common causes and solutions:

  • Missing parameters: Add them to required array
  • Wrong types: Use enum for constrained values
  • Complex objects: Flatten nested structures

Performance Issues

Optimize tool calling performance:

  • Cache frequently requested data
  • Set reasonable timeouts

Examples

Weather Assistant

1tools = [{
2    "type": "function",
3    "function": {
4        "name": "get_weather",
5        "description": "Get current weather for a location",
6        "parameters": {
7            "type": "object",
8            "properties": {
9                "location": {"type": "string"},
10                "unit": {"type": "string", "enum": ["C", "F"]}
11            },
12            "required": ["location"]
13        }
14    }
15}]
16
17response = client.chat.completions.create(
18    model="Hermes-3-Llama-3.1-70B",
19    messages=[{"role": "user", "content": "Is it raining in Seattle?"}],
20    tools=tools
21)

Customer Support Bot

1tools = [
2    {
3        "type": "function",
4        "function": {
5            "name": "create_ticket",
6            "description": "Create a support ticket",
7            "parameters": {
8                "type": "object",
9                "properties": {
10                    "title": {"type": "string"},
11                    "priority": {"type": "string", "enum": ["low", "medium", "high"]},
12                    "category": {"type": "string"}
13                },
14                "required": ["title", "category"]
15            }
16        }
17    },
18    {
19        "type": "function",
20        "function": {
21            "name": "search_knowledge_base",
22            "description": "Search help articles",
23            "parameters": {
24                "type": "object",
25                "properties": {
26                    "query": {"type": "string"}
27                },
28                "required": ["query"]
29            }
30        }
31    }
32]

Next Steps

  • Explore our SDK examples for language-specific implementations

Support

Need help with tool calling?