Skip to main content

​
Overview

A/B testing (also called split testing) lets you compare multiple prompt versions to see which performs better. Switchport provides deterministic routing so users get a consistent experience.

​
How It Works

  1. Create multiple versions of a prompt in the dashboard
  2. Set up a traffic config to distribute users across versions
  3. Execute prompts with user identification to get deterministic version assignment
  4. Record metrics to measure performance
  5. Analyze results in the dashboard to identify winners

​
Creating Multiple Versions

In the Switchport dashboard:
1

Navigate to your prompt

Go to Prompts and select the prompt you want to test
2

Create first version

  • Click Add Version
  • Name: v1 or formal-tone
  • Set model and prompt template
  • Click Save and Publish
3

Create second version

  • Click Add Version again
  • Name: v2 or casual-tone
  • Use different wording or approach
  • Click Save and Publish
4

Create traffic config

  • Click Traffic Config
  • Set distribution (e.g., 50% v1, 50% v2)
  • Click Activate

​
Executing Prompts with A/B Testing

The key to A/B testing is using context for deterministic routing: 
package main

import (
	"fmt"
	"log"

	"github.com/switchport-ai/switchport-go/switchport"
)

func main() {
	client, err := switchport.NewClient("")
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	// User 1 gets assigned a version
	response1, err := client.Prompts.Execute(
		"product-pitch",
		map[string]interface{}{"user_id": "user_001"},
		map[string]interface{}{"product": "Pro Plan"},
	)
	if err != nil {
		log.Fatalf("Failed to execute prompt: %v", err)
	}
	fmt.Printf("User 1 got: %s\n", response1.VersionName)

	// User 2 might get a different version
	response2, err := client.Prompts.Execute(
		"product-pitch",
		map[string]interface{}{"user_id": "user_002"},
		map[string]interface{}{"product": "Pro Plan"},
	)
	if err != nil {
		log.Fatalf("Failed to execute prompt: %v", err)
	}
	fmt.Printf("User 2 got: %s\n", response2.VersionName)

	// User 1 will ALWAYS get the same version
	response1Again, err := client.Prompts.Execute(
		"product-pitch",
		map[string]interface{}{"user_id": "user_001"},
		map[string]interface{}{"product": "Pro Plan"},
	)
	if err != nil {
		log.Fatalf("Failed to execute prompt: %v", err)
	}
	fmt.Printf("User 1 again: %s\n", response1Again.VersionName)

	// These will be equal:
	if response1.VersionID == response1Again.VersionID {
		fmt.Println("✓ Same user got same version (deterministic)")
	}
}
The same user always returns the same version. This ensures users have a consistent experience across sessions.

​
Recording Metrics for A/B Tests

To compare versions, record metrics with the same subject: 
func sendWelcomeEmail(client *switchport.Client, user User) error {
	subject := map[string]interface{}{"user_id": user.ID}

	// Execute prompt (subject gets assigned to a version)
	response, err := client.Prompts.Execute(
		"welcome-email", subject,
		map[string]interface{}{
			"name":        user.Name,
			"signup_date": user.CreatedAt,
		},
	)
	if err != nil {
		return err
	}

	// Send the email
	if err := sendEmail(user.Email, response.Text); err != nil {
		return err
	}

	// Later, track if they converted
	_, err = client.Metrics.Record(
		"conversion",
		true,
		subject, // Same subject!
		nil,
	)
	return err
}
Switchport automatically:
  • Links the metric to the version that user saw
  • Aggregates metrics per version
  • Calculates averages, success rates, and distributions

​
Traffic Distribution

You can distribute traffic in various ways:

​
50/50 Split (Classic A/B)

{
  "v1": 50,
  "v2": 50
}

​
Multivariate Testing (A/B/C)

{
  "v1": 33,
  "v2": 33,
  "v3": 34
}

​
Gradual Rollout

Start with a small percentage on the new version: 
{
  "v1": 90,
  "v2": 10
}
If metrics look good, increase the new version: 
{
  "v1": 50,
  "v2": 50
}
Finally, roll out completely: 
{
  "v2": 100
}

​
Complete Example: Email A/B Test

package main

import (
	"log"

	"github.com/switchport-ai/switchport-go/switchport"
)

type User struct {
	ID        string
	Name      string
	Email     string
	CreatedAt string
}

func sendWelcomeEmail(client *switchport.Client, user User) error {
	// Execute prompt (user gets assigned to a version)
	response, err := client.Prompts.Execute(
		"welcome-email",
		map[string]interface{}{"user_id": user.ID},
		map[string]interface{}{
			"name":        user.Name,
			"signup_date": user.CreatedAt,
		},
	)
	if err != nil {
		return err
	}

	// Send the email
	if err := sendEmail(user.Email, response.Text); err != nil {
		return err
	}

	// Track which version they saw (for debugging)
	logEvent("email_sent", map[string]interface{}{
		"user_id": user.ID,
		"version": response.VersionName,
	})

	return nil
}

func trackEmailOpened(client *switchport.Client, userID string) error {
	_, err := client.Metrics.Record(
		"email_opened",
		true,
		map[string]interface{}{"user_id": userID},
		nil,
	)
	return err
}

func trackEmailClicked(client *switchport.Client, userID string) error {
	_, err := client.Metrics.Record(
		"email_clicked",
		true,
		map[string]interface{}{"user_id": userID},
		nil,
	)
	return err
}

func trackConversion(client *switchport.Client, userID string) error {
	_, err := client.Metrics.Record(
		"conversion",
		true,
		map[string]interface{}{"user_id": userID},
		nil,
	)
	return err
}

func sendEmail(email, content string) error {
	// Implementation...
	return nil
}

func logEvent(event string, data map[string]interface{}) {
	log.Printf("Event: %s, Data: %v", event, data)
}

​
Analyzing Results

In the Switchport dashboard, you can view:
  • Metric averages per version: See which version has higher satisfaction scores
  • Conversion rates: Compare success rates for boolean metrics
  • Sample sizes: Ensure statistical significance
  • Confidence intervals: Understand the reliability of results

​
Best Practices

Always use the same subject (e.g., user ID) for a given subject across all prompt executions and metric recordings.
For initial A/B tests, use even splits to gather data faster.
Ensure you have enough data for statistical significance before declaring a winner.
Change only one variable between versions to understand what drives performance differences.
For new versions, start with a small percentage to minimize risk.
Decide what metrics matter before running the test to avoid cherry-picking results.
Watch for unexpected drops in other metrics when optimizing for one specific metric.

​
Common Use Cases

​
Email Marketing

Test subject lines, tone, call-to-action wording: 
// Different email styles for different user segments
response, err := client.Prompts.Execute(
	"marketing-email",
	map[string]interface{}{
		"user_id": user.ID,
		"segment": user.Segment,
	},
	map[string]interface{}{"product": "Summer Sale"},
)
Metrics: open rate, click rate, conversion rate

​
Customer Support Chatbot

Test different conversation styles: 
response, err := client.Prompts.Execute(
	"support-bot",
	map[string]interface{}{"user_id": user.ID},
	map[string]interface{}{"issue": userMessage},
)
Metrics: resolution rate, satisfaction score, escalation rate

​
Product Descriptions

Test different description styles: 
response, err := client.Prompts.Execute(
	"product-description",
	map[string]interface{}{
		"product_id":   product.ID,
		"user_segment": segment,
	},
	map[string]interface{}{"product_name": product.Name},
)
Metrics: conversion rate, time on page, add-to-cart rate

​
Next Steps

Examples

See complete A/B testing examples

Metrics Reference

Learn more about recording metrics