A/B Testing

Overview

A/B testing (also called split testing) lets you compare multiple prompt versions to see which performs better. Switchport provides deterministic routing so users get a consistent experience.

How It Works

Create multiple versions of a prompt in the dashboard
Set up a traffic config to distribute users across versions
Execute prompts with user identification to get deterministic version assignment
Record metrics to measure performance
Analyze results in the dashboard to identify winners

Creating Multiple Versions

In the Switchport dashboard:

Navigate to your prompt

Go to Prompts and select the prompt you want to test

Create first version

Click Add Version
Name: v1 or formal-tone
Set model and prompt template
Click Save and Publish

Create second version

Click Add Version again
Name: v2 or casual-tone
Use different wording or approach
Click Save and Publish

Create traffic config

Click Traffic Config
Set distribution (e.g., 50% v1, 50% v2)
Click Activate

Executing Prompts with A/B Testing

The key to A/B testing is using context for deterministic routing:

package main

import (
	"fmt"
	"log"

	"github.com/switchport-ai/switchport-go/switchport"
)

func main() {
	client, err := switchport.NewClient("")
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	// User 1 gets assigned a version
	response1, err := client.Prompts.Execute(
		"product-pitch",
		map[string]interface{}{"user_id": "user_001"},
		map[string]interface{}{"product": "Pro Plan"},
	)
	if err != nil {
		log.Fatalf("Failed to execute prompt: %v", err)
	}
	fmt.Printf("User 1 got: %s\n", response1.VersionName)

	// User 2 might get a different version
	response2, err := client.Prompts.Execute(
		"product-pitch",
		map[string]interface{}{"user_id": "user_002"},
		map[string]interface{}{"product": "Pro Plan"},
	)
	if err != nil {
		log.Fatalf("Failed to execute prompt: %v", err)
	}
	fmt.Printf("User 2 got: %s\n", response2.VersionName)

	// User 1 will ALWAYS get the same version
	response1Again, err := client.Prompts.Execute(
		"product-pitch",
		map[string]interface{}{"user_id": "user_001"},
		map[string]interface{}{"product": "Pro Plan"},
	)
	if err != nil {
		log.Fatalf("Failed to execute prompt: %v", err)
	}
	fmt.Printf("User 1 again: %s\n", response1Again.VersionName)

	// These will be equal:
	if response1.VersionID == response1Again.VersionID {
		fmt.Println("✓ Same user got same version (deterministic)")
	}
}

The same user always returns the same version. This ensures users have a consistent experience across sessions.

Recording Metrics for A/B Tests

To compare versions, record metrics with the same subject:

func sendWelcomeEmail(client *switchport.Client, user User) error {
	subject := map[string]interface{}{"user_id": user.ID}

	// Execute prompt (subject gets assigned to a version)
	response, err := client.Prompts.Execute(
		"welcome-email", subject,
		map[string]interface{}{
			"name":        user.Name,
			"signup_date": user.CreatedAt,
		},
	)
	if err != nil {
		return err
	}

	// Send the email
	if err := sendEmail(user.Email, response.Text); err != nil {
		return err
	}

	// Later, track if they converted
	_, err = client.Metrics.Record(
		"conversion",
		true,
		subject, // Same subject!
		nil,
	)
	return err
}

Switchport automatically:

Links the metric to the version that user saw
Aggregates metrics per version
Calculates averages, success rates, and distributions

Traffic Distribution

You can distribute traffic in various ways:

50/50 Split (Classic A/B)

{
  "v1": 50,
  "v2": 50
}

Multivariate Testing (A/B/C)

{
  "v1": 33,
  "v2": 33,
  "v3": 34
}

Gradual Rollout

Start with a small percentage on the new version:

{
  "v1": 90,
  "v2": 10
}

If metrics look good, increase the new version:

{
  "v1": 50,
  "v2": 50
}

Finally, roll out completely:

{
  "v2": 100
}

Complete Example: Email A/B Test

package main

import (
	"log"

	"github.com/switchport-ai/switchport-go/switchport"
)

type User struct {
	ID        string
	Name      string
	Email     string
	CreatedAt string
}

func sendWelcomeEmail(client *switchport.Client, user User) error {
	// Execute prompt (user gets assigned to a version)
	response, err := client.Prompts.Execute(
		"welcome-email",
		map[string]interface{}{"user_id": user.ID},
		map[string]interface{}{
			"name":        user.Name,
			"signup_date": user.CreatedAt,
		},
	)
	if err != nil {
		return err
	}

	// Send the email
	if err := sendEmail(user.Email, response.Text); err != nil {
		return err
	}

	// Track which version they saw (for debugging)
	logEvent("email_sent", map[string]interface{}{
		"user_id": user.ID,
		"version": response.VersionName,
	})

	return nil
}

func trackEmailOpened(client *switchport.Client, userID string) error {
	_, err := client.Metrics.Record(
		"email_opened",
		true,
		map[string]interface{}{"user_id": userID},
		nil,
	)
	return err
}

func trackEmailClicked(client *switchport.Client, userID string) error {
	_, err := client.Metrics.Record(
		"email_clicked",
		true,
		map[string]interface{}{"user_id": userID},
		nil,
	)
	return err
}

func trackConversion(client *switchport.Client, userID string) error {
	_, err := client.Metrics.Record(
		"conversion",
		true,
		map[string]interface{}{"user_id": userID},
		nil,
	)
	return err
}

func sendEmail(email, content string) error {
	// Implementation...
	return nil
}

func logEvent(event string, data map[string]interface{}) {
	log.Printf("Event: %s, Data: %v", event, data)
}

Analyzing Results

In the Switchport dashboard, you can view:

Metric averages per version: See which version has higher satisfaction scores
Conversion rates: Compare success rates for boolean metrics
Sample sizes: Ensure statistical significance
Confidence intervals: Understand the reliability of results

Best Practices

Use consistent context

Always use the same subject (e.g., user ID) for a given subject across all prompt executions and metric recordings.

Start with 50/50 splits

For initial A/B tests, use even splits to gather data faster.

Run tests long enough

Ensure you have enough data for statistical significance before declaring a winner.

Test one thing at a time

Change only one variable between versions to understand what drives performance differences.

Use gradual rollouts

For new versions, start with a small percentage to minimize risk.

Define success metrics upfront

Decide what metrics matter before running the test to avoid cherry-picking results.

Monitor for regressions

Watch for unexpected drops in other metrics when optimizing for one specific metric.

Common Use Cases

Email Marketing

Test subject lines, tone, call-to-action wording:

// Different email styles for different user segments
response, err := client.Prompts.Execute(
	"marketing-email",
	map[string]interface{}{
		"user_id": user.ID,
		"segment": user.Segment,
	},
	map[string]interface{}{"product": "Summer Sale"},
)

Metrics: open rate, click rate, conversion rate

Customer Support Chatbot

Test different conversation styles:

response, err := client.Prompts.Execute(
	"support-bot",
	map[string]interface{}{"user_id": user.ID},
	map[string]interface{}{"issue": userMessage},
)

Metrics: resolution rate, satisfaction score, escalation rate

Product Descriptions

Test different description styles:

response, err := client.Prompts.Execute(
	"product-description",
	map[string]interface{}{
		"product_id":   product.ID,
		"user_segment": segment,
	},
	map[string]interface{}{"product_name": product.Name},
)

Metrics: conversion rate, time on page, add-to-cart rate

Next Steps

Examples

See complete A/B testing examples

Metrics Reference

Learn more about recording metrics

Getting Started

Core Features

Examples

Overview

How It Works

Creating Multiple Versions

Executing Prompts with A/B Testing

Recording Metrics for A/B Tests

Traffic Distribution

50/50 Split (Classic A/B)

Multivariate Testing (A/B/C)

Gradual Rollout

Complete Example: Email A/B Test

Analyzing Results

Best Practices

Common Use Cases

Email Marketing

Customer Support Chatbot

Product Descriptions

Next Steps

Examples

Metrics Reference

​Overview

​How It Works

​Creating Multiple Versions

​Executing Prompts with A/B Testing

​Recording Metrics for A/B Tests

​Traffic Distribution

​50/50 Split (Classic A/B)

​Multivariate Testing (A/B/C)

​Gradual Rollout

​Complete Example: Email A/B Test

​Analyzing Results

​Best Practices

​Common Use Cases

​Email Marketing

​Customer Support Chatbot

​Product Descriptions

​Next Steps

Examples

Metrics Reference

Overview

How It Works

Creating Multiple Versions

Executing Prompts with A/B Testing

Recording Metrics for A/B Tests

Traffic Distribution

50/50 Split (Classic A/B)

Multivariate Testing (A/B/C)

Gradual Rollout

Complete Example: Email A/B Test

Analyzing Results

Best Practices

Common Use Cases

Email Marketing

Customer Support Chatbot

Product Descriptions

Next Steps