Experiments in CMS Orbit
Stop guessing. Let your visitors decide.
Show some visitors another version of the hero, or of a whole page, and count who goes on to do what you want, such as booking a trip. CMS Orbit tells you in plain words which version works better and when the answer is reliable.
The winner goes into the page's draft and goes live through the page's usual review. Part of the Growth and Enterprise plans.
Northwilds tested a hero that names the Lofoten ridge walk against its usual headline, counting paid bookings.
- Advice in plain words
- Each variant as visitors see it
- A person chooses the winner
How it works
From an idea to the winner, in the editor
Beside the page
Started, followed and decided from the page itself
The Test panel sits in the editor's right-hand rail. It sets up the experiment, opens each variant and, once it runs, shows the advice, each variant's visitors and rate and the buttons to pause, stop or choose the winner.
- Only what you change: components you leave alone keep following the page, so a typo fixed later is fixed in every variant.
- Fair from the start: once it starts, variants, goals and audience are fixed and the variants open read only.
- Preview links: anyone with the link can see a variant before it starts. Visits through them are never counted.
- Each variant's visitors and rate
Reading the results
How sure, how much better, and how much longer
The advice at the top says what to do: keep it running and for how long, choose a winner, keep the original, or that there's no real difference. The charts under it show why.
- Chance to beat the original: 50% means you can't tell yet.
- Uplift with its 95% range: a range from −5% to +20% means it could still go either way.
- Risk if chosen: how much conversion you could lose if the variant is in fact worse.
- Guardrails: goals that must not get worse, such as visitors who stay, turn red when a variant harms them.
- A warning when traffic is off: if variants get a different share of visitors than planned, the results say so before you trust them.
A winner is only called once every variant has at least 100 visitors, the minimum days have passed, it's at least 95% likely to beat the original and the risk is under 1% of the original's rate. You can look every day.
- How sure the result is
- Guardrails
When a winner is called
Four checks before the advice says choose
You can look every day. The advice only says choose a winner once all four are true.
100
visitors at least, in every variant
Small numbers swing too much to trust.
7 days
at least, to start with
A full week evens out weekdays and weekends.
95%
likely to beat the original
50% means you can't tell yet.
< 1%
risk if chosen
Of the original's rate, should the variant in fact be worse.
Your choice
Who takes part, and how
Privacy
Fair tests that respect your visitors
Plans
On Growth and Enterprise
Experiments are part of the Growth and Enterprise plans of CMS Orbit. On Starter, the Test panel shows which plan includes them, and an owner can upgrade under Billing in Orbit.
Your website needs version 0.11 or later of the JavaScript SDK, which runs experiments with no extra code. One more line lets visitors take part without cookies.
Questions
About experiments
Do A/B tests in CMS Orbit need a developer?
No. On Growth and Enterprise, editors test a component or a whole page with up to three variants in the editor, choose a goal and an audience, and approve the winner like any other change. Websites on the JavaScript SDK 0.11 or later run experiments with no extra code.
How long does an experiment need?
It depends on your traffic. When you set one up, CMS Orbit estimates how long it needs to find a 10% and a 20% improvement from the page's visitors and today's success rate. A week is the minimum to start with, so weekdays and weekends even out, and the results say roughly how many more days a reliable answer takes.
Do experiments need a cookie banner?
Not necessarily. By default, visitors who accept cookies keep their variant for 13 months, and before that take part for the day without one. Or run cookieless only: nothing is stored in the browser. Which fits depends on the law where your visitors are, so ask whoever looks after data protection.
Does the winner go live straight away?
The winner goes into the page's draft and the experiment ends, so every visitor sees it until the page is published. Publishing goes through the page's usual review and approvals, so nothing goes live without a person approving it.
Find out what works. Then keep it.
Start a website in CMS Orbit on the Growth plan and run your first experiment on a real page. The trial is free for 14 days.