Skip to main content

​
Prompt Configs

A prompt config is a container for all versions of a specific prompt. It has:
  • Key: A unique identifier used in your code (e.g., welcome-message)
  • Name: A human-readable name for the dashboard
  • Description: Optional documentation about the prompt’s purpose
  • Variables: Placeholders that get filled in at execution time
Example: 
# In your code, you reference the prompt by its key
response = client.prompts.execute(
    prompt_key="welcome-message",
    variables={"name": "Alice"}
)

​
Prompt Versions

Each prompt config can have multiple versions. A version includes:
  • Model: The LLM to use (e.g., gpt-5, claude-3-5-sonnet-20241022)
  • Prompt Template: The actual prompt text with variable placeholders
  • Version Name: A label for this version (e.g., v1, formal-tone)
  • Status: Published or draft
Example version: 
Model: gpt-5
Prompt: Write a friendly welcome message for {{name}}.

​
Subject Identification

Subject identification is data you provide to ensure consistent version assignment. It’s used for:
  1. Deterministic Routing: Same subject always gets the same version
  2. Metrics Aggregation: Links metrics to the correct prompt version
Subject identification can be:
  • A simple string: "user_123"
  • A dictionary: {"user_id": "user_123", "tier": "premium"}
Always use the same subject when executing prompts and recording metrics for the same user or session. This ensures metrics are correctly attributed to prompt versions.
Example: 
# Execute with subject identification
response = client.prompts.execute(
    prompt_key="welcome-message",
    subject={"user_id": "user_123"},
    variables={"name": "Alice"}
)

# Record metric with same subject
client.metrics.record(
    metric_key="satisfaction",
    value=4.5,
    subject={"user_id": "user_123"}  # Same subject!
)

​
A/B Testing

A/B testing (or multivariate testing) lets you compare multiple prompt versions:
  1. Create multiple versions of a prompt
  2. Set up a traffic config that defines the distribution (e.g., 50% v1, 50% v2)
  3. Execute the prompt with subject identification
  4. Users are deterministically assigned to versions based on subject hash
The same subject always gets the same version. This ensures users have a consistent experience across sessions.
Example flow: 
# User 1 might get version A
response_1 = client.prompts.execute(
    prompt_key="product-pitch",
    subject={"user_id": "user_001"}
)

# User 2 might get version B
response_2 = client.prompts.execute(
    prompt_key="product-pitch",
    subject={"user_id": "user_002"}
)

# User 1 will ALWAYS get version A on subsequent calls
response_1_again = client.prompts.execute(
    prompt_key="product-pitch",
    subject={"user_id": "user_001"}
)
# response_1.version_id == response_1_again.version_id

​
Metrics

Metrics are measurements you track to evaluate prompt performance. They can be:
  • Float: Numerical values (e.g., satisfaction score, response time)
  • Boolean: True/false values (e.g., conversion, email opened)
  • Enum: Categorical values (e.g., sentiment: positive/negative/neutral)
Metrics are automatically aggregated per prompt version when you use the same subject. Example: 
# Record different metric types
client.metrics.record(
    metric_key="satisfaction",
    value=4.5,  # Float
    subject={"user_id": "user_123"}
)

client.metrics.record(
    metric_key="converted",
    value=True,  # Boolean
    subject={"user_id": "user_123"}
)

client.metrics.record(
    metric_key="sentiment",
    value="positive",  # Enum
    subject={"user_id": "user_123"}
)

​
Variables

Variables are dynamic values you pass to prompts at execution time. They’re defined using double curly braces in your prompt template. Dashboard: 
Write a {{tone}} welcome message for {{customer_name}} who just purchased {{product}}.
Code: 
response = client.prompts.execute(
    prompt_key="welcome-message",
    variables={
        "tone": "professional",
        "customer_name": "Alice",
        "product": "Pro Plan"
    }
)

​
Traffic Configs

A traffic config defines how users are distributed across prompt versions: 
{
  "v1": 50,  // 50% of users get v1
  "v2": 30,  // 30% get v2
  "v3": 20   // 20% get v3
}
Traffic configs enable:
  • Gradual rollouts: Start with 10% on new version, increase if metrics look good
  • A/B tests: Equal split (50/50) between two versions
  • Multivariate tests: Test 3+ versions simultaneously

​
Request IDs

Every prompt execution returns a request ID. This unique identifier:
  • Links prompts to metrics in the dashboard
  • Helps with debugging and support
  • Enables detailed audit trails
response = client.prompts.execute(prompt_key="welcome")
print(response.request_id)  # "req_abc123..."

​
Next Steps

Python Quickstart

Get started with the Python SDK

A/B Testing Guide

Learn how to run A/B tests