How to Design Experiment

Corrily allows users to design and run various types of experiments. This article will describe the general experimentation framework, guide you through the process of setting up and running any type of experiment, as well as analyzing the results and taking appropriate actions after the experiment was concluded.

​

From Hypothesis to Experiment Design

Before running an experiment, it is important to outline what hypothesis you want to test.

This could mean running a single experiment or it could mean breaking down the hypothesis into multiple experiments. For e.g. if your hypothesis is that there may be higher willingness to pay for your products in North America, you may want to break down the hypothesis into two experiments - one experiment across US and Canada (since they are likely similar markets for your business), and another for Mexico.

If your hypothesis is a little more specific - say, there is higher willingness to pay amongst professionals in North America, it is best to define the relevant audience on the Corrily dashboard, and segment your experiment further. However, if you do not have a strong hypothesis, we recommend starting broad, learning from the experiment analytics so you can form some hypothesis, and then running more specific experiments.

Other important consideration when designing your experiment is to think through what cohorts of users do you want to experiment on. Do you want to experiment solely on new users who have never seen your pricing page before, or you want to experiment with just the free user base.

There are all important considerations in deciding who to experiment on and setting your experiments up for success.

​

Experiment Setup

​

Experiment Status Flow

Every experiment undergoes a systematic Status Flow from drafting to conclusion. This flow represents the progressive steps from initial consideration of an experiment to its final results.

1. Drafting the Experiment Configuration

   • At this stage, you simply draft the configuration of your experiment. It’s worth noting that while in the draft state, the experiment remains inactive and only serves as a saved template for any future modifications.

     

2. Manual Testing

   • Initially, set the experiment traffic to 0%. This enables manual testing without affecting production users.
   • With Corrily API, you can force experimental prices to be returned even when traffic is set at 0%. Additionally, users can be directly assigned an experimental price via the User Page on the dashboard.

     • Traffic Allocation - Step 1: Navigate to “Experiments” on the left menu and select the desired experiment. Once inside, scroll down and click on the “Traffic Allocation” edit icon as illustrated below.

       

     • Traffic Allocation - Step 2: In the subsequent window, set your experiment traffic (0% in this instance). Options are available for either automatic traffic allocation per experiment arm or manual adjustments. Confirm your settings by clicking “Save.”

       

3. Activating the Experiment

   • Once satisfied, activate your experiment. The experimental prices will instantly be served to your users.

   • Activation can be done by selecting the chosen experiment under the “Experiments” window and then clicking “Start Experiment.”

     

4. Pausing and Resuming

   • If necessary, experiments can be temporarily paused and then resumed at a later time.

     

5. Concluding the Experiment

   • When you’ve gathered sufficient data and insights, you can conclude the experiment, ceasing its operations permanently.

     

Recommendation: Avoid rolling out the experiment to 100% of your traffic immediately after manual testing. Instead, initiate with a gradual rollout, targeting 5-10% of your traffic. Monitor for at least a day to ascertain whether users are subscribing at the experimental price and if operations are proceeding as anticipated. If all appears in order, you can then scale the traffic percentage to encompass 100%.

​

Experiment Products

The first step in setting up an experiment is to add the products you want to run the experiment for. These products can be any kind of subscription plan or addon (monthly/annual, basic/advanced) and can be defined in the Products tab of the Corrily dashboard

​

Experiment Targeting

A crucial aspect of Experiment Setup Process is experiment targeting. Generally, there’s no obligation to modify prices for your entire user base. In many instances, businesses differentiate pricing based on the countries they operate in. This distinction arises from variations in Purchasing Power Parity (PPP) across countries.

Thus, the primary configuration option for experiment setup pertains to targeting. It allows specification of countries (based on purchasing power or region) or the identification of distinct countries for the experiment. Additionally, the target audience can either encompass all users or be tailor-made to focus on a specific audience found under the ‘User Management’ tab. Further details on audience creation are available at Segmentation Documentation.

Corrily’s advanced filtering system facilitates this, allowing businesses to categorize countries based on Purchasing Power, Regional distinctions, or specific country-based criteria

User audience targeting can be broad, encompassing all users, or tailored to focus on a particular group. This specified group can be designed under the ‘User Management’ tab. For comprehensive guidelines on crafting a specific audience, please consult our Audience Segmentation Documentation.

​

Experiment Policy

A crucial configuration in your experiment setup is the Experiment Policy. There are two primary options available:

• NEW_USER_ONLY: This policy focuses exclusively on new users. The idea behind this strategy is to protect your existing user base, ensuring they are not subjected to unexpected price modifications. Those who have previously seen the price or have transitioned to paying customers will remain unaffected by the experiment. Typically, this “new users only” approach is the default, offering a risk-free method, as there’s no chance of existing users identifying any package alterations.
• NEW_AND_EXISTING_USERS: Sometimes, due to a limited influx of new users or perhaps operating over an extended period, there may arise a need to update packages for all users, including the existing ones. This strategy addresses that, allowing experimental packages to be showcased to both new users and existing users who haven’t been part of any ongoing experiment. This implies that irrespective of any previous packages these users might have seen, Corrily will present the newly determined experimental packages to them.

It’s vital to select a policy that aligns with your business goals and the expectations of your user base.

Once customer is created it’ll be impossible for the customer to pay in the different currency, so it makes it impossible to do package experiments for existing users. To avoid technical issues here we’re recommending do the special preparation steps to group already paid users into the separate Audience, and to exclude that Audience from any experiment.

​

Traffic Allocation Strategies

The fourth key concept is Traffic Allocation. It’s recommended to use the default Automatic option, which will split the traffic equally among arms. Think of it as about an AB test or about ABN test. Almost everybody wants to send the same amount of traffic to each arm, but sometimes if you don’t have enough traffic or if you’re already confident in your baseline price starts and sure, there’re no risk of seasonality change, you might want to avoid sending traffic to baseline arm.

You might want to change the Traffic Split manually. For example to:

• decrease baseline arm traffic %
• or to completely disable some poorly performing arms

​

Steps Before Starting the Experiment

​

Backfill Existing Users

Before starting an experiment, it is recommended to backfill existing users into Corrily to avoid serving experimental prices to the existing user base. This also allows you to create an audience for existing paid customers and exclude them from any experiment to avoid currency issues. The backfill of the small userbase could be done using Identify API Endpoint. More information about segmentation can be found in Audiences.

​

Mitigating the Risks:

Potential Risks of Price Experimentation:

1. Payment Currency Discrepancy: One of the primary risks is the possibility of a returning paying user encountering an wrong currency when they revisit, thus hindering them from making subsequent payments.
2. Perception of Inconsistent Pricing: There exists a risk of existing users detecting a shift in pricing. Such inconsistencies can lead to discomfort or displeasure among them.

To counteract these risks, it’s advisable to implement a backfill of existing users into the Corrily system prior to initiating any price experiments. Undertaking this step ensures that existing users aren’t inadvertently subjected to new experimental pricing. Furthermore, once these users are backfilled, an Audience dedicated to current paid users can be constructed, allowing them to be exempted from experimental changes, thereby averting potential currency discrepancies.

​

Analyzing Experiment Results

​

Compare ARPU or predicted LTV across the Arms

Corrily provides the comprehensive set of charts to study the experimental prices performance on every stage of the user funnel:

• starting from conversion from free Visitors to the signed Users,
• then conversion to a trial period or to a first payment,
• and then the revenue metrics on how much did user pay over time,
• and finally the retention and charts

Using experiment Analytics you’ll be able to analyze the every step of the funnel individually, or to see the final experimentation KPI metric projection (usually, the “Predicted LTV”).

​

Look at the over-time charts to study trends

Also want to note that Experiment Analytics has “Over-time charts”, which might help you to experiment quicker. That charts help to study trends in conversion or revenue across arms. It’s not rare experiment stats stop changing over time, means nothing is going to be changed in the future with a more data coming in, so, there’s a signal to conclude the experiment faster.

​

Steps after Concluding the Experiment

​

Rollout New Prices or Packages

If the optimal arm wins, you may want to roll out the new prices to your entire user base in the experimental countries and for experimented audiences. To do this, update the prices in the Price Control Center and decide if you want to migrate existing users to allow them to see the updated prices or packages instead of the previous ones

​

Migrate Experimental Users Back to Baseline

If the baseline arm wins, you may want to reset the experimental users back to the baseline price. This can be done via Prices Migration. The migration of the users might be done using Calculate Price API Endpoint with the "override": true flag in payload.

​

Limitations and Tricky Moments

You should be aware that Corrily Data Model assumes that a single user can be part of at most one experiment at the time. If there’re multiple experiments running simultaneously in the same country, even if they’ll be testing the prices for the different products, it wouldn’t be possible for the same user to participate in both experiments. It’s strongly recommended to run no more than a single experiment in the same country, or for the same Audience at a time.