Theme Scientist is an A/B testing app for Shopify.
- Theme A/B testing with daily stat tracking
- Easy Product A/B testing – set the attributes in the app and it will handle the rest
- Analyze the success of your test with metrics like Orders, Sales, Adds to Cart, Page Views
- Theme Scheduler feature to publish a theme at a specific time in the future
Theme A/B testing
- Duplicate your live theme
- Make a change to the new theme (for example add to cart button color)
- Choose the 2 themes to test and start the A/B split test
- Check back in a few days and analyze your stats
The app automatically rotates the 2 themes every day within a few minutes after midnight and logs the stats.
Product A/B testing
- Use the app to select a product
- Change the product attributes or image order on version B
- Start the test and let it run a few days
- Check the stats in the app to see which version is winning
We made it super easy so you just set up the test then kick back and wait for the stats to come in.
Tips for an accurate test
- The more traffic you have and the longer your tests run, the more accurate the results.
- A good minimum time frame is 2 weeks so each theme version sees 1 day of the week.
- Test 1 element or section at a time
- Don’t make edits to a theme during testing. If it’s necessary, make the same changes to both themes.
A/B testing gives you data driven results to make smarter decisions.
There are infinite possibilities but here are some ideas to get started:
- Add to cart color (or size)
- Promotion bar (like free shipping vs brand messaging vs % off)
- Product page layout
- Product price or title
- Product image order (maybe try a studio image vs a lifestyle image)
- Home page banner image (collage of proucts vs lifestyle image vs background + words)
Visitors and Sessions are tracked similar to Google Analytics and Shopify analytics, by setting a cookie in the user’s browser and sending events to a database.
The app will install a small pixel on your website that logs these events:
Each unique person that comes to your site in a day is a visitor.
A session is a period of time that a visitor hangs out on your site. A session expires after 30 minutes of activity or at midnight your local store time.
Every time someone navigates to a page on your website, this counts as a page view.
Add to Cart
When someone clicks the add to cart button on your product page.
Duplicate clicks on the same page are not counted.
Say a person comes to your site at 9am. They check out 2 pages, add 1 product to cart, then leave.
That same person returns at 4pm, looks at 3 more pages, adds a product to cart, and purchases.
The logged stats would be:
- 1 visitor
- 2 sessions
- 5 page views
- 2 add to cart
- 1 order
- $xx.xx sales
*This same journey would yield 1 add to cart in Shopify analytics. Shopify tracks add to cart only once per session, not by actual adds to cart.
Analytics software have similar principles but have too many variables to yield the exact same stats.
TS Analytics are close to Shopify Analytics, with the exception of Add to Cart.
Add to Cart
Shopify counts multiple adds to cart in the same session as 1 add to cart. So the actual data point is “sessions where a product was added to cart”.
For A/B testing, it’s good to know how many times a user actually added a product to cart.
TS tracks each add to cart event, but only once per page (to prevent skewed stats from double clicks).
By default, Google Analytics includes bot traffic in their reports.
TS Analytics filters out bots.
Your overall sales might reflect differently in Shopify and TS because TS only tracks the Online Store channel (since that’s what we’re testing).
TS pulls order data directly from Shopify for best accuracy. GA uses a tracking script, so you might notice fewer orders / sales in GA.
TS does not count refunds. So if you have a day with refunds, you may see different totals.
For more info about analytics differences see Discrepancies between Shopify and other software .
We use the daily swap method. This works well for template based platform sites like Shopify where you don’t have much control over your server.
This means that we swap themes on the server side, rather than the client side (like if you were doing split traffic method), so there is no performance lag and no rendering lag during page load.
When you start an A/B test, the 2 themes or products will swap at 12:01am every day until the test ends.
The reason for swapping once per day rather than multiple times per day is so that most visitors will see the same theme version during their daily interaction.
This also keeps a consistent viewing experience for people that visit your site from 2 different devices on the same day – cookies only track in the browser, so if someone checks your site on a phone, then logs into their computer later, they are essentially a “new visitor”.
We recommend running a test for at least 2 weeks so each version sees a different day of the week.
The app has a built in statistical significance calculator to help you decide how confident 1 version or the other is performing better. The more traffic and the longer the test runs, the more accurate your results.
We also have a statistical significance calculator here if you want to plug in some numbers and do some estimations.
Yes, but if you edit 1 theme you should edit the other theme so the only difference is the variable you’re testing.
The 2 themes involved in a test are tracked by theme ID so the same 2 themes will swap throughout the duration of the test even if you rename a theme.
If a different / 3rd theme, the app will indicate a warning in the results showing that the test stats on that day may not be accurate.
For accurate results, you can have only 1 active theme test.
You can run as many product tests as you’d like.
The fewer elements you change during a test, the more accurate your results will be.
For example, if you test 2 different add to cart buttons + free shipping vs no free shipping + product image on the left or right side… you won’t know which individual element converts better, or if the combination converts better.
Easy – set a theme to publish at a specific time in the future and it will publish at that time.
This is useful during promotion periods to maximize your promotion accuracy and so you don’t have to rely on a human to remember (or stay up late) to publish the theme at a specific time.
If you set a theme to schedule during an A/B test, the Theme Scheduler will publish the theme you chose, and the A/B Tester will continue swapping themes at 12:01 on the next day.
If you have any questions or feature requests contact us.