theme scientist shopify ab testing app snippet

👉 Get Theme Scientist on the Shopify app store.

Theme Scientist is an automated A/B testing app for Shopify.

Highlights:

  • Easily set up A/B tests with auto theme swapping to gather daily stats
  • Analyze the success of your test with metrics like Orders, Sales, Adds to Cart, Visitors, Sessions, Page Views
  • Schedule a theme to publish at a specific time in the future

If you want more features let us know and we’ll build it 👨‍💻

FAQ

There are infinite possibilities but here are some ideas to get started:

  • Add to cart color (or size)
  • Promotion bar (like free shipping vs brand messaging vs % off)
  • Product page layout
  • Navigating to cart page after add to cart, or staying on the product page
  • Home page banner image (collage of proucts vs lifestyle image vs background + words)
  1. Duplicate your live theme (or the theme you want to set as Theme A)
  2. Make a change to this new theme (example: change the add to cart button color)
  3. Navigate to A/B Theme Tests then choose these 2 themes and start the A/B test
  4. Theme Scientist will swap themes at 12:01am every day and log the stats of each day / theme
  5. Check back after the test starts to see the analytics

Tips for an accurate test

  • The more traffic you have and the longer your tests run, the more accurate the results.
  • A good minimum time frame is 2 weeks so each theme version sees 1 day of the week.
  • Test 1 element or section at a time
  • Don’t make edits to a theme during testing. If it’s necessary, make the same changes to both themes.

Visitors and Sessions are tracked similar to Google Analytics and Shopify analytics, by setting a cookie in the user’s browser and sending events to a database.

The app will install a small pixel on your website that logs these events:

Visitors

Each unique person that comes to your site in a day is a visitor.

Sessions

A session is a period of time that a visitor hangs out on your site. A session expires after 30 minutes of activity or at midnight your local store time.

Page Views

Every time someone navigates to a page on your website, this counts as a page view.

Add to Cart

When someone clicks the add to cart button on your product page.

Duplicate clicks on the same page are not counted.

Example:

Say a person comes to your site at 9am. They check out 2 pages, add 1 product to cart, then leave.

That same person returns at 4pm, looks at 3 more pages, adds a product to cart, and purchases.

The logged stats would be:

  • 1 visitor
  • 2 sessions
  • 5 page views
  • 2 add to cart
  • 1 order
  • $xx.xx sales

*This same journey would yield 1 add to cart in Shopify analytics. Shopify tracks add to cart only once per session, not by actual adds to cart.

Analytics software have similar principles but have too many variables to yield the exact same stats.

TS Analytics are close to Shopify Analytics, with the exception of Add to Cart.

Add to Cart

Shopify counts multiple adds to cart in the same session as 1 add to cart. So the actual data point is “sessions where a product was added to cart”.

For A/B testing, it’s good to know how many times a user actually added a product to cart.

TS tracks each add to cart event, but only once per page (to prevent skewed stats from double clicks).

Bots

By default, Google Analytics includes bot traffic in their reports.

TS Analytics filters out bots.

Sales channels

Your overall sales might reflect differently in Shopify and TS because TS only tracks the Online Store channel (since that’s what we’re testing).

Orders

TS pulls order data directly from Shopify for best accuracy. GA uses a tracking script, so you might notice fewer orders / sales in GA.

TS does not count refunds. So if you have a day with refunds, you may see different totals.

For more info about analytics differences see Discrepancies between Shopify and other software .

When you start an A/B test, the 2 themes will swap at 12:01am every day until the test ends.

The reason for swapping once per day rather than multiple times per day is so that most visitors will see the same theme version during their daily interaction.

This also keeps a consistent viewing experience for people that visit your site from 2 different devices on the same day – cookies only track in the browser, so if someone checks your site on a phone, then logs into their computer later, they are essentially a “new visitor”.

Yes, but if you edit 1 theme you should edit the other theme so the only difference is the variable you’re testing.

The 2 themes involved in a test are tracked by theme ID so the same 2 themes will swap throughout the duration of the test even if you rename a theme.

If a different / 3rd theme, the app will indicate a warning in the results showing that the test stats on that day may not be accurate.

Easy – set a theme to publish at a specific time in the future and it will publish at that time.

This is useful during promotion periods to maximize your promotion accuracy and so you don’t have to rely on a human to remember (or stay up late) to publish the theme at a specific time.

If you set a theme to schedule during an A/B test, the Theme Scheduler will publish the theme you chose, and the A/B Tester will continue swapping themes at 12:01 on the next day.

If you have any questions or feature requests contact us.