Using Prediction Guard proxies in LangChain | Prediction Guard

LangChain is one of the most popular AI projects, and for good reason! LangChain helps you “Build applications with LLMs through composability.” LangChain doesn’t host LLMs or provide a standardized API for controlling LLMs, which is addressed by Prediction Guard. Therefore, combining the two (Prediction Guard + LangChain) gives you a framework for developing controlled and compliant applications powered by language models.

Installation and Setup

Install the Prediction Guard LangChain partner package with pip install langchain-predictionguard
Get a Prediction Guard api key (as described here) and set it as the environment variable PREDICTIONGUARD_API_KEY.

LLM Wrapper

There exists a Prediction Guard partner package, which you can access with

1 from langchain_predictionguard import PredictionGuard

You can provide the name of the Prediction Guard model as an argument when initializing the LLM:

1 pgllm = PredictionGuard(model="gpt-oss-120b")

You can also provide your api key and url directly as an argument, or set them as environment variables:

1 os.environ["PREDICTIONGUARD_API_KEY"] = "<api key>"
2 os.environ["PREDICTIONGUARD_URL"] = "<pg url>"
3 
4 pgllm = PredictionGuard(
5     model="gpt-oss-120b", 
6     predictionguard_api_key="<Your PG API Key>",
7     predictionguard_url="<Your PG API URL>")

Finally, you can provide an “output” argument that is used to validate the output of the LLM:

1 pgllm = PredictionGuard(model="gpt-oss-120b", output={"toxicity": True})

Example usage

Basic usage of the controlled or guarded LLM wrapper:

1 import os
2 
3 from langchain_predictionguard import PredictionGuard
4 from langchain import PromptTemplate, LLMChain
5 
6 # Define a prompt template
7 template = """Respond to the following query based on the context.
8 
9 Context: EVERY comment, DM + email suggestion has led us to this EXCITING announcement! 🎉 We have officially added TWO new candle subscription box options! 📦
10 Exclusive Candle Box - $80 
11 Monthly Candle Box - $45 (NEW!)
12 Scent of The Month Box - $28 (NEW!)
13 Head to stories to get ALLL the deets on each box! 👆 BONUS: Save 50% on your first box with code 50OFF! 🎉
14 
15 Query: {query}
16 
17 Result: """
18 prompt = PromptTemplate(template=template, input_variables=["query"])
19 
20 # With "guarding" or controlling the output of the LLM. See the 
21 # Prediction Guard docs (https://docs.predictionguard.com) to learn how to 
22 # control the output with integer, float, boolean, JSON, and other types and
23 # structures.
24 pgllm = PredictionGuard(model="gpt-oss-120b")
25 pgllm(prompt.format(query="What kind of post is this?"))

Basic LLM Chaining with the Prediction Guard wrapper:

1 import os
2 
3 from langchain import PromptTemplate, LLMChain
4 from langchain_predictionguard import PredictionGuard
5 
6 # Your Prediction Guard API key. Get one at predictionguard.com
7 os.environ["PREDICTIONGUARD_API_KEY"] = "<api key>"
8 
9 pgllm = PredictionGuard(model="gpt-oss-120b")
10 
11 template = """Question: {question}
12 
13 Answer: Let's think step by step."""
14 prompt = PromptTemplate(template=template, input_variables=["question"])
15 llm_chain = LLMChain(prompt=prompt, llm=pgllm, verbose=True)
16 
17 question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"
18 
19 llm_chain.predict(question=question)