CMS Orbit
A/B testing in a headless CMS: test a hero, keep the winner
Try another hero or a whole new page on real visitors, set a goal such as Booking paid, read in plain words when the answer is reliable and keep the winner.
6 min read
On this page
Most website changes are a guess. Someone thinks a new photo or a sharper headline will bring more bookings, it goes live, and nobody can say afterwards whether it helped. Proper A/B testing usually means another tool, a script that makes the page flicker and a results screen full of statistics.
CMS Orbit has experiments built in. You test a hero or a whole page from the editor, on real visitors, and it tells you in plain words which version works better and when you can trust the answer. This post explains how.
An example from Northwilds
Northwilds wants more people to book its trips. Its home page hero has a mountain view and the usual headline. The team thinks a hero that names the Lofoten ridge walk will do better, so they test it, counting paid bookings.
Half the visitors see the hero as it is. The other half see Variant B. A visitor who sees the new hero and later pays for the Lofoten ridge walk on another page counts as a booking, because goals on other pages count too. After a week or two, the advice at the top of the results says something like 'Variant B is 98% likely to be better than Original', with the range of how much better, and that it is safe to promote it.
Set up a test in the editor
Open the page, select the hero and choose Test in the right-hand rail, then Try another hero. The button is named after the component you selected.
- What to test: One component, such as the hero, or the whole page. A whole-page variant can change, add, remove or reorder anything on it.
- The goal: A click on a link or button in what you test, a visit to a page such as
/book/thank-you/, a form sent, visitors who stay, an event such as Booking paid or Add to bag, or a goal from Insights Orbit. - Variants: One, two or three, so A/B, A/B/C or A/B/C/D. Visitors are shared evenly between the original and the variants.
- Visitors taking part: How many visitors are in the test at all. The rest see the original and aren't counted.
Before you start, CMS Orbit estimates how long the test needs to find a 10% and a 20% improvement, from the page's visitors a day and today's success rate. With Insights Orbit connected, those come from the page's last 30 days.
You can also write down what you expect, such as 'A photo of travellers on the trail will get more trip enquiries than the mountain view'. It is shown with the results, so nobody rewrites the question once the answer is in.
Change only what you test
The variant opens in the same editor, with a bar that says changes are saved to the variant only and the live page stays as it is. Components you leave alone keep following the page, so a typo fixed on the page later is fixed in every variant too.
Each variant has a preview link to check before you start. Visits through those links are never counted. Once the test starts, its variants, goals, audience and traffic are fixed, so the results stay fair.
Who takes part
Under Audience, guardrails and schedule you can narrow it down, or leave everything as it is:
- Audience: Computers, tablets or phones, new or returning visitors, countries such as
NO, DE, GB, and visitors arriving from a campaign. - Guardrails: Goals that must not get worse. Visitors who stay is ticked to start with, so a win on clicks can't quietly cost you engaged visitors.
- Sharing the traffic: An even split gives the clearest answer. For short campaigns, such as a winter sale, Move towards the best sends more visitors to the variant doing best, keeping at least 10% on each.
- Schedule: Start and end it by itself, with a minimum of 7 days to start with, so a full week evens out weekdays and weekends.
Read the answer in plain words
The advice says what to do: keep it running and for roughly how long, choose a winner, keep the original, or that there's no real difference. The charts under it show why:
- Chance to beat the original: 50% means you can't tell yet.
- Uplift with its 95% range: A range from −5% to +20% means it could still go either way.
- Risk if chosen: How much conversion you could lose if the variant is in fact worse.
- Guardrails: Bars turn red when a variant is at least 90% likely to make one worse.
- A warning when traffic is off: If variants get a different share of visitors than planned, the results say so before you trust them.
A winner is only called once every variant has at least 100 visitors, the minimum days have passed, it is at least 95% likely to beat the original and the risk is under 1% of the original's rate. You can look every day: the advice takes that into account.
Keep the winner, through your usual review
Choose Choose the winner and pick a variant. The one the advice picked is marked suggested, and you are told if it looks like it makes a guardrail worse. The winner goes into the page's draft and the experiment ends. Every visitor sees it from then on, and publishing goes through the page's usual review and approvals. Or keep the original and leave the page as it is.
Choosing a winner needs its own permission, which SEO Managers, Administrators and Owners have. Every start, pause, stop and winner is in the audit log. AI agents can read results and draft an experiment for you to check, but only people start, pause, stop and choose winners.
Fair to your visitors
By default, visitors who accept cookies keep their variant for 13 months, and before that take part for the day without one. Or choose cookieless only: nothing is stored in the browser, and a code for the day, worked out on your server, keeps each visitor's variant until tomorrow. Which fits depends on the law where your visitors are, so ask whoever looks after data protection.
Visitors with Do Not Track or Global Privacy Control, search engines and bots always see the original and aren't counted. CMS Orbit stores no address, browser details or cookie, and each visitor's record is deleted at the latest 90 days after the experiment ends.
With Insights Orbit
When the space is connected to Insights Orbit, Open in Insights shows what else each variant changed: the pages visitors went on to, how long they stayed and the site's other goals. Insights Orbit only shows the numbers. The test is still run and decided in CMS Orbit. Read more in cookieless analytics next to your pages.
For developers
With @bifrost-orbit-cms/sdk 0.11 or later and pages rendered with CmsPage or CmsLivePage, experiments run with no code. A small inline script picks each visitor's variant before the first paint, so there is no flicker, and the page is cached once for everyone. One line in your proxy, withExperiments, lets visitors take part without cookies. Google Tag Manager, Google Analytics 4 and VWO can see each visitor's variant through the data layer, and their events can be goals.
The details are in the docs: Run experiments on your website for developers, and Run an experiment for editors.
What it takes
Experiments are part of the Growth and Enterprise plans of CMS Orbit. On Starter, the Test panel shows which plan includes them, and an owner can upgrade under Billing in Orbit. See how experiments work, compare the plans or explore CMS Orbit. The trial is free for 14 days.
Read next
SEO built into CMS Orbit, AI agents with a person in control and live products, trips and jobs on your pages.