Experiments and variants

Creating an Experiment

To create a new experiment:

1. Navigate to the Hyppot admin panel at APP_URL/hyppot/panel
2. Enter a descriptive name for your experiment
3. Click "Add Experiment"
4. Define the experiment, its conversions, variants and audience, and then save it.
5. Click on the Activate button to start it

Configuring Tracked Conversions

Conversions you want to track within an experiment are defined within "Tracked conversions" section of experiment definition. Please note that before you can add the conversion there, you need to define it in the "Conversion Types Definition" UI:

Each conversion type can have required metrics value that act as a filter for the conversions. Only the conversions with metrics value equal to the ones defined for the experiment will be considered valid for the experiment. You can use them if you want to track only selected conversions, for example - click on the specific button (like "add to cart").

Configuring Variants

A variant represents a specific version of your feature that you want to test. Each variant has:

• A unique identifier,
• An audience configuration,
• Values for all variables defined in the experiment.

To add a variant:

1. Click "Add Variant"
2. Enter a variant identifier
3. Set values for all experiment variables
4. Configure the audience for this variant

Audience Configuration

Each variant's audience can be configured in three ways:

1. Percentage - What percentage of users should see this variant

   • The sum of percentages across all variants must not exceed 100%
   • A variant with 0% will not be shown to any users

2. Allow List - Specific users who should always see this variant

   • Users in the allow list will see this variant regardless of the percentage setting
   • Useful for testing with specific users or internal teams

3. Deny List - Specific users who should never see this variant

   • Users in the deny list will not see this variant
   • Useful for excluding certain users from the experiment

Audience Assignment

Hyppot uses a deterministic algorithm to assign users to variants, ensuring consistent assignment across sessions.

To resolve if user is assigned to an experiment variant in your code:

// Get the experiment resolver from dependency injection
var resolvedExperiment = resolver.ResolveForUser(
    new ExperimentId("your-experiment-id"),
    new UserId("user-id")
);

if (resolvedExperiment != null)
{
    // User is assigned to a variant
}

Variant variables

You can define variables of the three types:

• String (.Net string),
• Number (.Net double),
• Boolean (.Net bool).

The below example shows how to access the variable value in your .Net code:

// Get the experiment resolver from dependency injection
var resolvedExperiment = resolver.ResolveForUser(
    new ExperimentId("your-experiment-id"),
    new UserId("user-id")
);

if (resolvedExperiment != null)
{
    
    var columnCount = resolvedExperiment.Variables
        .First(x => x.Name == new VariableName("column-count"))
        .Value.Get<double>();
}

Tracking Experiment Data

Hyppot provides a comprehensive system for tracking experiment data.

Tracking Impressions

An impression occurs when a user is shown a variant of your experiment. To track impressions:

// Get the experiment tracker from dependency injection
var tracker = serviceProvider.GetRequiredService<IExperimentTracker>();

// Track impression for a specific user and experiment
tracker.TrackImpression(
    ExperimentImpression.Create(
        experimentId: "your-experiment-id",
        variantId: "variant-id",
        user: "user-id",
        eventDate: DateTime.UtcNow,
        deduplicationKey: "optional-deduplication-key"
    )
);

Impressions are deduplicated using either the deduplication key or by user ID within the impression length window. This prevents the same user from being counted multiple times in a short period.

Tracking Conversions

Conversions represent user actions that indicate success for your experiment. To track conversions:

// Get the experiment tracker from dependency injection
var tracker = serviceProvider.GetRequiredService<IExperimentTracker>();

// Track a simple conversion
tracker.TrackConversion(
    ExperimentConversion.Create(
        conversionType: "your-conversion-type",
        eventDate: DateTime.UtcNow,
        user: "user-id"
    )
);

// Track a conversion with metrics
tracker.TrackConversion(
    ExperimentConversion.Create(
        conversionType: "purchase",
        eventDate: DateTime.UtcNow, 
        user: "user-id",
        deduplicationKey: "optional-deduplication-key",
        metrics: new VariableInstance(
            new VariableName("amount"), 
            new VariableValue<double>(99.99)
        ),
        new VariableInstance(
            new VariableName("product-id"), 
            new VariableValue<string>("SKU-12345")
        ),
        new VariableInstance(
            new VariableName("is-subscription"), 
            new VariableValue<bool>(true)
        )
    )
);

Conversions are automatically associated with impressions if:

Note that you do not have to provide experiment data when tracking conversions - you just define the conversion information. Hyppot will automatically link and count conversions tracked for given experiments if:

1. The user ID matches
2. The conversion occurred after the impression
3. The conversion occurred within the configured conversion window
4. All of the required metrics match those defined in the experiment's tracked conversion.

The conversion will be associated with impression up to 24 hours from the occurrence (you can configure this time window during the application setup).

deduplicationKey is an optional parameter that can be used to identify the impression/conversion so you don't accidentally track them twice (for example when retrying after transient error).

Multiple tracked impressions for the same experiment and user will be treated as one if happened within 24 hour time window from the first occurrence (unless you provide different deduplication keys). This can also be configured during the app setup.

The order of tracking events does not matter. You can track conversion even before impression, as long as you provide correct timestamps of when the events ocurred.

Working with Metrics

Hyppot supports tracking metrics for each conversion. Metrics provide additional data points about conversions, allowing for more detailed analysis.

Metrics use the same VariableInstance type as experiment variables:

• They support the same types: String, Number (double), and Boolean
• They're created using the same VariableInstance class and value types
• They provide additional context for each conversion

Metrics can be used in two main ways:

1. As conversion filters to only count specific conversions (as shown in "Configuring Tracked Conversions")
2. As additional data points for analysis in your experiment results

To define metrics for a conversion:

var metrics = new VariableInstance[] {
    new VariableInstance(
        new VariableName("revenue"), 
        new VariableValue<double>(125.50)
    ),
    new VariableInstance(
        new VariableName("product-type"), 
        new VariableValue<string>("electronics")
    ),
    new VariableInstance(
        new VariableName("is-new-customer"), 
        new VariableValue<bool>(false)
    )
};

tracker.TrackConversion(
    ExperimentConversion.Create(
        "purchase-completed",
        DateTime.UtcNow,
        "user-id",
        deduplicationKey: null,
        metrics: metrics
    )
);

Viewing Results

After your experiment has been running for some time, you can view the results in the Hyppot admin panel:

1. Navigate to the experiment details
2. Expand the selected experiment
3. Click on the "Statistics" tab
4. View conversion rates and other metrics for each variant