Validating LLM Output

Consistency

You might be struggling with inconsistent output from LLMs. Even if you set parameters like temperature low, you could get non-deterministic output from your models. Prediction Guard has built in consistency checks.

Rather than looping over multiple API calls, you can make one API call that will concurrently prompt an LLM multiple times, check for consistent output, and return an error if there is inconsistent output.

Let’s use the following example prompt template to illustrate the feature.

1import os
2import json
3
4import predictionguard as pg
5from langchain.prompts import PromptTemplate
6
7os.environ["PREDICTIONGUARD_TOKEN"] = "<your access token>"
8
9template = """Respond to the following query based on the context.
10
11Context: EVERY comment, DM + email suggestion has led us to this EXCITING announcement! 🎉 We have officially added TWO new candle subscription box options! 📦
12Exclusive Candle Box - $80 
13Monthly Candle Box - $45 (NEW!)
14Scent of The Month Box - $28 (NEW!)
15Head to stories to get ALLL the deets on each box! 👆 BONUS: Save 50% on your first box with code 50OFF! 🎉
16
17Query: {query}
18
19Result: """
20prompt = PromptTemplate(template=template, input_variables=["query"])

To enforce consistency on any output, it’s as simple as setting consistency equal to a boolean True in the output field/argument to Prediction Guard:

1result = pg.Completion.create(
2    model="Nous-Hermes-Llama2-13B",
3    prompt=prompt.format(query="What kind of post is this?"),
4    output={
5        "consistency": True
6    }
7)
8
9print(json.dumps(
10    result,
11    sort_keys=True,
12    indent=4,
13    separators=(',', ': ')
14))

If there is consistency in the calls to the LLM, you get standard output similar to:

{
    "choices": [
        {
            "index": 0,
            "output": "product announcement",
            "status": "success",
            "text": "product announcement"
        }
    ],
    "created": 1685539044,
    "id": "cmpl-4oFZt63m9EkydxJaDG1Fg2RwAebrK",
    "model": "Nous-Hermes-Llama2-13B",
    "object": "text_completion"
}

But if the LLM isn’t consistent in the output (in this case the classification of the text), you get an error:

{
    "choices": [
        {
            "index": 0,
            "status": "error: inconsistent results",
            "text": ""
        }
    ],
    "created": 1685538920,
    "id": "cmpl-dom2Lnj7vJZcqGzjHKTihBI7zNqr4",
    "model": "Nous-Hermes-Llama2-13B",
    "object": "text_completion"
}
Built with