Skip to content
Bifröst Orbit
Sign in

CMS Orbit

A/B testing in a headless CMS: test a hero, keep the winner

Try another hero or a whole new page on real visitors, set a goal such as Booking paid, read in plain words when the answer is reliable and keep the winner.

Bifröst Orbit team

6 min read

On this page

Most website changes are a guess. Someone thinks a new photo or a sharper headline will bring more bookings, it goes live, and nobody can say afterwards whether it helped. Proper A/B testing usually means another tool, a script that makes the page flicker and a results screen full of statistics.

CMS Orbit has experiments built in. You test a hero or a whole page from the editor, on real visitors, and it tells you in plain words which version works better and when you can trust the answer. This post explains how.

An example from Northwilds

Northwilds wants more people to book its trips. Its home page hero has a mountain view and the usual headline. The team thinks a hero that names the Lofoten ridge walk will do better, so they test it, counting paid bookings.

Half the visitors see the hero as it is. The other half see Variant B. A visitor who sees the new hero and later pays for the Lofoten ridge walk on another page counts as a booking, because goals on other pages count too. After a week or two, the advice at the top of the results says something like 'Variant B is 98% likely to be better than Original', with the range of how much better, and that it is safe to promote it.

Set up a test in the editor

Open the page, select the hero and choose Test in the right-hand rail, then Try another hero. The button is named after the component you selected.

  • What to test: One component, such as the hero, or the whole page. A whole-page variant can change, add, remove or reorder anything on it.
  • The goal: A click on a link or button in what you test, a visit to a page such as /book/thank-you/, a form sent, visitors who stay, an event such as Booking paid or Add to bag, or a goal from Insights Orbit.
  • Variants: One, two or three, so A/B, A/B/C or A/B/C/D. Visitors are shared evenly between the original and the variants.
  • Visitors taking part: How many visitors are in the test at all. The rest see the original and aren't counted.

Before you start, CMS Orbit estimates how long the test needs to find a 10% and a 20% improvement, from the page's visitors a day and today's success rate. With Insights Orbit connected, those come from the page's last 30 days.

You can also write down what you expect, such as 'A photo of travellers on the trail will get more trip enquiries than the mountain view'. It is shown with the results, so nobody rewrites the question once the answer is in.

CMS OrbitPage editor
The Test panel beside the Northwilds home page: the running hero test, its advice and each variant's visitors and rate.

Change only what you test

The variant opens in the same editor, with a bar that says changes are saved to the variant only and the live page stays as it is. Components you leave alone keep following the page, so a typo fixed on the page later is fixed in every variant too.

Each variant has a preview link to check before you start. Visits through those links are never counted. Once the test starts, its variants, goals, audience and traffic are fixed, so the results stay fair.

Who takes part

Under Audience, guardrails and schedule you can narrow it down, or leave everything as it is:

  • Audience: Computers, tablets or phones, new or returning visitors, countries such as NO, DE, GB, and visitors arriving from a campaign.
  • Guardrails: Goals that must not get worse. Visitors who stay is ticked to start with, so a win on clicks can't quietly cost you engaged visitors.
  • Sharing the traffic: An even split gives the clearest answer. For short campaigns, such as a winter sale, Move towards the best sends more visitors to the variant doing best, keeping at least 10% on each.
  • Schedule: Start and end it by itself, with a minimum of 7 days to start with, so a full week evens out weekdays and weekends.

Read the answer in plain words

The advice says what to do: keep it running and for roughly how long, choose a winner, keep the original, or that there's no real difference. The charts under it show why:

  • Chance to beat the original: 50% means you can't tell yet.
  • Uplift with its 95% range: A range from −5% to +20% means it could still go either way.
  • Risk if chosen: How much conversion you could lose if the variant is in fact worse.
  • Guardrails: Bars turn red when a variant is at least 90% likely to make one worse.
  • A warning when traffic is off: If variants get a different share of visitors than planned, the results say so before you trust them.

A winner is only called once every variant has at least 100 visitors, the minimum days have passed, it is at least 95% likely to beat the original and the risk is under 1% of the original's rate. You can look every day: the advice takes that into account.

CMS OrbitExperiment
The full results of the hero test, with the chance that Variant B is better and the range of how much.

Keep the winner, through your usual review

Choose Choose the winner and pick a variant. The one the advice picked is marked suggested, and you are told if it looks like it makes a guardrail worse. The winner goes into the page's draft and the experiment ends. Every visitor sees it from then on, and publishing goes through the page's usual review and approvals. Or keep the original and leave the page as it is.

Choosing a winner needs its own permission, which SEO Managers, Administrators and Owners have. Every start, pause, stop and winner is in the audit log. AI agents can read results and draft an experiment for you to check, but only people start, pause, stop and choose winners.

Fair to your visitors

By default, visitors who accept cookies keep their variant for 13 months, and before that take part for the day without one. Or choose cookieless only: nothing is stored in the browser, and a code for the day, worked out on your server, keeps each visitor's variant until tomorrow. Which fits depends on the law where your visitors are, so ask whoever looks after data protection.

Visitors with Do Not Track or Global Privacy Control, search engines and bots always see the original and aren't counted. CMS Orbit stores no address, browser details or cookie, and each visitor's record is deleted at the latest 90 days after the experiment ends.

With Insights Orbit

When the space is connected to Insights Orbit, Open in Insights shows what else each variant changed: the pages visitors went on to, how long they stayed and the site's other goals. Insights Orbit only shows the numbers. The test is still run and decided in CMS Orbit. Read more in cookieless analytics next to your pages.

For developers

With @bifrost-orbit-cms/sdk 0.11 or later and pages rendered with CmsPage or CmsLivePage, experiments run with no code. A small inline script picks each visitor's variant before the first paint, so there is no flicker, and the page is cached once for everyone. One line in your proxy, withExperiments, lets visitors take part without cookies. Google Tag Manager, Google Analytics 4 and VWO can see each visitor's variant through the data layer, and their events can be goals.

The details are in the docs: Run experiments on your website for developers, and Run an experiment for editors.

What it takes

Experiments are part of the Growth and Enterprise plans of CMS Orbit. On Starter, the Test panel shows which plan includes them, and an owner can upgrade under Billing in Orbit. See how experiments work, compare the plans or explore CMS Orbit. The trial is free for 14 days.

SEO built into CMS Orbit, AI agents with a person in control and live products, trips and jobs on your pages.

CMS Orbit

Fix photos where you use them, and keep the original

The image editor in CMS Orbit and Travel Orbit crops, straightens, adjusts and blurs photos in the media library, and keeps the file you uploaded. Here is what it does and how blurring keeps private details off your website.

5 min read

CMS Orbit

Edit pages and trips together, in real time

Several people can work on the same page or trip at once. Here is how you see each other, avoid typing over each other and talk without leaving the page.

5 min read

See it with your own team.

Orbit is free. Try CMS Orbit for 14 days with your own content, no card needed, or ask us anything.

Talk to us