Using LLMs

Tool Calling

Invoke external functions from PredictionGuard’s /chat/completions endpoint

Please use the /models endpoint to see which models support tool calling.

Overview

Tool calling extends your model’s capabilities beyond static text generation. With tool calling, models can:

  • Request real-time information (weather, stock prices)
  • Execute business workflows (create tickets, send emails)
  • Chain multiple operations together
  • Integrate with your existing APIs and services

Core Concepts

Tools

Functions you expose to the model, defined using JSON Schema. Each tool includes:

  • Name: Unique identifier for the function
  • Description: Clear explanation of what the tool does
  • Parameters: Expected inputs with types and constraints

Tool Calls

When the model determines a tool would help answer a query, it generates a tool call with:

  • The tool name to invoke
  • Arguments formatted as JSON
  • A unique call ID for tracking

Tool Choice Strategy

Controls how the model decides whether to use tools:

StrategyTypeBehaviorUse Case
"auto"stringModel decides if tools would helpGeneral assistants
"none"stringTools disabled for this requestPure text generation
"required"stringMust use at least one toolData-dependent queries

Quick Start

1. Define Your Tool

Create a JSON schema describing your function:

1tool = {
2  "type": "function",
3  "function": {
4    "name": "get_current_weather",
5    "description": "Get current weather conditions for a specific location",
6    "parameters": {
7      "type": "object",
8      "properties": {
9        "location": {
10          "type": "string",
11          "description": "City and state, e.g., 'San Francisco, CA'"
12        },
13        "unit": {
14          "type": "string",
15          "enum": ["celsius", "fahrenheit"],
16          "description": "Temperature unit preference"
17        }
18      },
19      "required": ["location"],
20      "additional_properties": false
21    },
22    "strict": false
23  }
24}

Write clear, specific descriptions for both the tool and its parameters. This helps the model understand when and how to use your tools correctly. Set strict: true to enforce exact schema adherence.

2. Make a Request with Tools

Include your tool definitions in the API request:

1from predictionguard import PredictionGuard
2
3client = PredictionGuard(api_key="<your pg api_key>")
4
5response = client.chat.completions.create(
6    model="gpt-oss-120b",
7    messages=[
8      {
9        "role": "user",
10        "content": "What'\''s the weather like in San Francisco?"
11      }
12    ],
13    tools=tool,
14    tool_choice="auto"
15)

3. Handle the Tool Call

When the model requests a tool, you’ll receive:

1{
2  "id": "chatcmpl-123",
3  "choices": [{
4    "index": 0,
5    "message": {
6      "role": "assistant",
7      "content": null,
8      "tool_calls": [{
9        "id": "call_abc123",
10        "type": "function",
11        "function": {
12          "name": "get_current_weather",
13          "arguments": "{\"location\":\"San Francisco, CA\",\"unit\":\"fahrenheit\"}"
14        }
15      }]
16    },
17    "finish_reason": "tool_calls"
18  }]
19}

4. Execute and Return Results

Parse the arguments, execute your function, and send back the result:

1# Parse the tool call
2import json
3
4tool_call = response["choices"][0]["message"]["tool_calls"][0]
5args = json.loads(tool_call["function"]["arguments"])
6
7# Execute your function
8weather_data = get_weather(args["location"], args.get("unit", "fahrenheit"))
9
10# Continue the conversation with the result
11messages.append({
12    "role": "assistant",
13    "content": None,
14    "tool_calls": [tool_call]
15})
16messages.append({
17    "role": "tool",
18    "tool_call_id": tool_call["id"],
19    "name": tool_call["function"]["name"],
20    "content": json.dumps(weather_data)
21})

5. Get the Final Response

The model incorporates the tool result and provides a natural language response:

1{
2  "role": "assistant",
3  "content": "The current weather in San Francisco is 68°F with partly cloudy skies. It's a pleasant day with light winds from the west at 10 mph."
4}

Best Practices

Tool Design

Keep tools focused: Each tool should have a single, clear purpose

1✅ get_user_profile(user_id)
2❌ get_user_data_and_posts_and_friends(user_id)

Use explicit parameters: Avoid ambiguous or overly flexible inputs

1✅ "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
2❌ "unit": {"type": "string", "description": "any temperature unit"}

Provide clear descriptions: Help the model understand when to use each tool

1✅ "description": "Retrieves real-time stock price for a given ticker symbol"
2❌ "description": "Gets data"

Error Handling

Build robust error handling for production use:

1import asyncio
2
3try:
4    # Parse arguments safely
5    args = json.loads(tool_call["function"]["arguments"])
6    
7    # Validate required parameters
8    if "location" not in args:
9        raise ValueError("Missing required parameter: location")
10    
11    # Execute with timeout
12    result = await asyncio.wait_for(
13        get_weather(args["location"]), 
14        timeout=5.0
15    )
16    
17except json.JSONDecodeError:
18    result = {"error": "Invalid arguments provided"}
19except asyncio.TimeoutError:
20    result = {"error": "Weather service timeout"}
21except Exception as e:
22    result = {"error": f"Service error: {str(e)}"}

Security Considerations

Validate all inputs: Never trust model-generated arguments

1# Sanitize and validate
2location = args.get("location", "").strip()
3if not location or len(location) > 100:
4    raise ValueError("Invalid location")

Check for prompt injections: Use PredictionGuard’s injection detection API

1# Check user input for potential prompt injection
2injection_check = client.injection.check(
3    prompt=user_message,
4    detect=True
5)
6
7if injection_check["checks"][0]["probability"] > 0.5:
8    # High probability of injection attack
9    print("error: Invalid input detected")

PredictionGuard provides a dedicated Injection API to detect prompt injection attacks. Consider checking user inputs before processing them with tool-enabled models to prevent malicious attempts to manipulate tool behavior.

Implement access controls: Verify permissions before execution

1if not user_has_permission(user_id, "weather_api"):
2    print("error: Access denied")

Never expose sensitive data: Keep API keys and secrets server-side

1# ❌ Don't include in tool results
2{"api_key": "sk-123", "temperature": 72}
3
4# ✅ Return only necessary data
5{"temperature": 72, "condition": "sunny"}

Troubleshooting

Model Skips Required Tools

If tool_choice: "auto" isn’t working:

  • Add explicit instructions in the user message
  • Use named tool forcing
  • Ensure tool descriptions are clear

Invalid Arguments

Common causes and solutions:

  • Missing parameters: Add them to required array
  • Wrong types: Use enum for constrained values
  • Complex objects: Flatten nested structures

Performance Issues

Optimize tool calling performance:

  • Cache frequently requested data
  • Set reasonable timeouts

Examples

Weather Assistant

1tools = [{
2    "type": "function",
3    "function": {
4        "name": "get_weather",
5        "description": "Get current weather for a location",
6        "parameters": {
7            "type": "object",
8            "properties": {
9                "location": {"type": "string"},
10                "unit": {"type": "string", "enum": ["C", "F"]}
11            },
12            "required": ["location"]
13        }
14    }
15}]
16
17response = client.chat.completions.create(
18    model="gpt-oss-120b",
19    messages=[{"role": "user", "content": "Is it raining in Seattle?"}],
20    tools=tools
21)

Customer Support Bot

1tools = [
2    {
3        "type": "function",
4        "function": {
5            "name": "create_ticket",
6            "description": "Create a support ticket",
7            "parameters": {
8                "type": "object",
9                "properties": {
10                    "title": {"type": "string"},
11                    "priority": {"type": "string", "enum": ["low", "medium", "high"]},
12                    "category": {"type": "string"}
13                },
14                "required": ["title", "category"]
15            }
16        }
17    },
18    {
19        "type": "function",
20        "function": {
21            "name": "search_knowledge_base",
22            "description": "Search help articles",
23            "parameters": {
24                "type": "object",
25                "properties": {
26                    "query": {"type": "string"}
27                },
28                "required": ["query"]
29            }
30        }
31    }
32]

Next Steps

  • Explore our SDK examples for language-specific implementations

Support

Need help with tool calling?