Getting Started

Quick Start

Technical teams need to figure out how to integrate the latest Large Language Models (LLMs), but:

  • You can’t build robust systems with inconsistent, unvalidated outputs; and
  • LLM integrations scare corporate lawyers, finance departments, and security professionals due to hallucinations, cost, lack of compliance (e.g., HIPAA), leaked IP/PII, and “injection” vulnerabilities.

Some companies are moving forward anyway by investing tons of engineering time/money in their own wrappers around LLMs and expensive hosting with OpenAI/Azure. Others are ignoring these issues and pressing forward with fragile and risky LLM integrations.

At Prediction Guard, we think that you should get useful output from compliant AI systems (without crazy implementation/ hosting costs), so our solution lets you:

  1. De-risk LLM inputs to remove PII and prompt injections;
  2. Validate and check LLM outputs to guard against hallucination, toxicity and inconsistencies; and
  3. Implement private and compliant LLM systems (HIPAA and self-hosted) that give your legal counsel warm fuzzy feeling while still delighting your customers with AI features.

Sounds pretty great right? Follow the steps below to starting leveraging trustworthy LLMs:

1. Get access to Prediction Guard Enterprise

We host and control the latest LLMs for you in our secure and privacy-conserving enterprise platform, so you can focus on your prompts and chains. To access the hosted LLMs, contact us here to get an enterprise access token. You will need this access token to continue.

2. Start using one of our LLMs!

Suppose you want to prompt an LLM to answer a user query from a chat application. You can setup a message thread, which includes a system prompt (that instructs the LLM how to behave in responding) as follows:

"role": "system",
"content": "You are a helpful assistant. Your model is hosted by Prediction Guard, a leading AI company."
"role": "user",
"content": "Where can I access the LLMs in a safe and secure environment?"

3. Download the SDK for your favorite language

You can then use any of our official SDKs or REST API to prompt one of our LLMs!

1import json
2import os
4from predictionguard import PredictionGuard
6# You can set you Prediction Guard API Key as an env variable,
7# or when creating the client object
10client = PredictionGuard(
11 api_key="<your Prediction Guard API Key"
14messages = [
15 {
16 "role": "system",
17 "content": "You are a helpful chatbot that helps people learn."
18 },
19 {
20 "role": "user",
21 "content": "What is a good way to learn to code?"
22 }
25result =
26 "model": "Hermes-2-Pro-Llama-3-8B",
27 "messages": messages,
28 "max_tokens": 100
32 result,
33 sort_keys=True,
34 indent=4,
35 separators=(',', ': ')

Note, you will need to replace <your api key> in the above examples with your actual access token.