Run a Proof Impact experiment with a no-widget group
Split your visitors between a published widget and no widget at all, then compare how often each group converts.
An experiment splits the visitors of one published widget in two: half see the widget as usual, and half don't see it at all. Comparing how often each group converts tells you whether the widget makes a difference, which the ordinary Proof Impact report can't. Experiments are on Business. The workspace owner and the project's Admins run them; anyone who can open the project can read the results.
Before you start, you need a published widget and a conversion event your site already sends, with Proof Impact tracking installed. See Set up Proof Impact tracking.
Create a draft
Open Proof Impact under Analyze in your project's sidebar, click the Experiments tab, then New experiment.
- Name: what you want to find out.
- Widget: the widget to test. The no-widget group won't see it anywhere it's embedded.
- Goal: the conversion the two groups are compared on. Your primary event is chosen for you. Events with the Guardrail role, archived events and events left out of reporting aren't offered.
- Duration: 1 to 4 weeks. Whole weeks keep weekdays and weekends even between the groups.
Advanced holds the rest, each with a default. See Advanced settings.
Click Create draft. Nothing changes on your site yet. Until you start it, you can change a draft with Edit or remove it with Delete.
Start it
Select the draft and click Start experiment, then confirm. Visitors start joining groups a few minutes later, and the setup can't change from then on.
Starting fails if the widget isn't published, if an event is archived or left out of reporting, or if another experiment is already running or paused on the same widget. While an experiment runs on a widget, Auto-Optimize waits on that widget.
Let it run
The experiment shows its day (Running: day 5 of 14), how many visitors have joined, the date it ends and the date the result arrives. The numbers under it are a look at a test in progress; wait for the result before acting on them.
It ends by itself, within an hour of reaching its duration. The result arrives after the time a visitor has to convert has passed for the last visitors who joined.
You get a notification when the experiment ends this way, and another when its result is ready.
Read the result
The result opens with what the experiment found, such as The widget raises conversions or No clear difference. See What each result means.
Below it, The two groups lists Visitors, Converted and Conversion rate for Saw the widget and Didn't see the widget. Show the statistics opens the difference with its likely range, the p-value, the Data checks and how many visitors the widget was delivered to. Setup and history shows what the experiment was started with and what happened to it.
Advanced settings
Advanced holds these settings of a draft, each with a default:
- Time a visitor has to convert: how long after a visitor's first visit in the experiment their conversion still counts. It starts as your goal's own attribution window.
- Safety metric (optional): an event that mustn't get worse, such as refunds. It's offered when the project has an event with the Guardrail role.
- Value currency (optional): fill this in to compare the value per visitor too. Every conversion must carry a value in this currency.
- Smallest difference (optional): the least that counts as a result, in percentage points. Left empty, any clear difference counts.
- What you expect (optional): a note for your team.
What each result means
The result opens with one of these:
| Result | What it means |
|---|---|
| The widget raises conversions | Visitors who saw the widget converted more, and the difference is statistically clear. The result says by how much and how confident it is. |
| The widget lowered conversions | Visitors who saw the widget converted less, in the same clear way. |
| No clear difference | The two groups converted about the same. A difference this small can be chance. |
| Not enough data to tell | Too few conversions arrived to compare the groups. |
| A difference too small to count | The difference is real, but smaller than the Smallest difference you set. |
| A safety metric got worse | Your safety metric moved the wrong way for visitors who saw the widget, so it isn't counted as a win. |
| This result can't be trusted | A data check failed. The result says which one. |
The confidence in a result is the chance that the widget's true conversion rate is the higher one, estimated from the two groups' numbers. It's never shown as 100%.
Pause, resume or end early
- Pause stops placing new visitors in groups. Everyone sees the widget, and visitors already counted keep their group. The end date doesn't move.
- Resume continues with the same setup.
- End early ends the experiment before its duration. An experiment ended early can't say whether the widget helps, however its numbers look: it shows Ended early: not enough data to tell. To stop for a while, pause it instead.
An ended experiment can't be started again. Create a new draft to test again.
What visitors see
- Didn't see the widget: the widget doesn't appear on any page where it's embedded.
- Saw the widget: the widget appears as usual.
- Each visitor stays in their group on later visits in the same browser, and counts in it even if the widget failed to load for them.
- Visitors who aren't measured aren't placed in a group, such as those who declined consent or, by default, whose browser sends a Global Privacy Control signal. Bots are left out too. They all see the widget as usual, and so does a visitor whose group couldn't be looked up.
A conversion counts when your site sends it with convert() in the visitor's browser, within the time a visitor has to convert. Orders your server sends aren't linked to a group.
What happens next
A result never changes your widget: keeping it, changing it or removing it stays your decision. When a result is No clear difference or Not enough data to tell, a longer experiment, or one on a page with more visitors, can give a clearer answer.
Was this page helpful?
Read the Proof Impact report
Read Proof Impact's totals, attribution models and breakdowns, check data quality, fix a value that shows Unavailable and export the numbers.
Auto-Optimize a Proof Match widget
Let a Proof Match widget test its rules against each other and show each visitor the proof that leads to more conversions.