New Shopify App: Theme Scientist A/B Testing Tool

Updated December 23, 2021

Update: Theme Scientist is now operated by a new team. It’s still my favorite A/B testing app but as a team Speed Boostr is decidedly small and narrowing our focus. We partnered with PreviewX to acquire the app, allowing the app to grow faster and give it the attention it deserves (check out the latest version in the Shopify app store). PreviewX has a strong team that will continue the evolution of the app and help Shopify stores run A/B testing on their website. 👍

🎉 We launched a new app!

See it in the Shopify app store here: Theme Scientist

What it does

The key feature is A/B Testing.

A/B Testing (aka Split Testing): Testing 2 different versions of a website or page to see which one converts better.

Here’s how A/B testing works with this app:

• Set it to automatically swap 2 themes every day at midnight and track daily stats
• Tracks visitors, sessions, views, adds to cart, orders, sales, conversion rate
• Let the test run and analyze the results to see which theme converts better

The idea is to test 1 feature at a time (button color, slider vs banner, page features, etc).

For A/B testing help + ideas, check out our A/B Testing Guide for Shopify.

There’s also a Theme Scheduler feature that lets you schedule a theme to publish at a specific time in the future. This is especially useful during promotion periods so nobody as to remember to log in exactly when you want to start the promo.

Example A/B Test

I ran a test recently on a Shopify store to see whether a blue or green add to cart button performed better.

Here are the results after 2 weeks:

Theme A with the blue add to cart button had fewer sessions but more orders + sales + higher conversion rate. Blue wins!

Maybe blue feels more trustworthy, I don’t know but the data shows that more people prefer the blue add to cart in this sample.

This isn’t a huge difference, but if this data set indicates long term preferences, that boost in conversion rate translates to large gains.

Some tests yield small changes, some yield big differences. Either way you won’t know the best solution until you run an actual test.

How to set up an A/B test?

With the Theme Scientist app, it’s easy:

1. Duplicate your live theme
2. Make a change to the new theme (like changing the add to cart button color)
3. Select the 2 themes in the app and start the A/B split test
4. Check back in a few days and analyze your stats

You can let it run as long as you’d like. A longer test = more accurate data.

I like to run tests for 2 – 4 weeks. I prefer 2 weeks minimum so each theme version gets a full day of the week.

Split test methods

There are various methods to A/B testing (daily swaps, hourly swaps, per user swaps). I like daily swaps the best.

The main reason is that people will often view your site on multiple devices. You can’t control what they see across devices, so a single daily swap ensures that user has the best chance at a consistent experience on any given day.

I think having a consistent viewing experience for a user’s session is important for a good user experience + quality data gathering.

This method has worked well for me (both on Amazon and Shopify) – I’ve had great success with Splitly AB Testing on Amazon. They use the daily swap method too.

Performance

I’ve seen A/B testing tools that modify the page in real time… and I remember being able to see the page render as it was loading (like a button changing colors during load). That’s no good.

I built this app with performance in mind.

You create your A and B versions as separate themes, then the app automatically swaps the themes on the back end, so there is no performance loss on your website.

The tracking script is a tiny 4KB file that listens for events (page views, add to cart click, sessions), then sends them to the app database. The script is minified and set to load at the end of your page so it doesn’t affect page rendering.

The build story

Man this was a journey. A lot of fun but also a lot of soul crushing debugging work 😂

I built this over the course of a few months in a combination of some part time chipping away + some intense hackathons.

Most of the coding happend at Dojo coworking space in Bali.

I’d show up around 7am, blast underground techno into my noise cancelling headphones, fire up some caffeine and go hard until about noon.

Around noon I’d walk down to the beach to get some sun and watch the waves while digesting the morning session, sometimes snacking an acai bowl.

The load of other projects + Speed Boostr management dictated how much I worked on the app the rest of the day, but that was pretty much my morning routine 6-7 days a week for the few weeks I was dialed in to bust out the bulk of the project.

Challenges

The most frustrating part was probably time – keeping track of store time vs universal time, different time zones, daylight savings time, and calculating and storing time accurately.

Side note: I thought daylight savings time was dumb before… working with it in code moved it up a good notch or 2 in my book of things that suck.

The most challenging was probably building the custom analytics tracking system (plus figuring out the swap system for accurate stat tracking on each version). Shopify analytics tracks this data but I can only tap the API for all that data if the store is on the Advanced Shopify plan or higher.

That would severely limit the user base, so I decided to build my own.

I used JavaScript to track events, integrated bot detection and cookie tracking, and consulted with a database expert to help design a solid database that scales well into the future and able to handle millions of events per day.

As with any large project, I learned a ton along the way and had a grand celebration the week I finished it and submitted to the app store. 🍻🥳

Why build an A/B testing app?

As competition increases, A/B testing becomes more important.

Ads get more expensive and ecommerce gets more saturated every year. This makes A/B testing a secret weapon to maximize your conversions and get a higher ROI.

I’ve always done A/B testing on Shopify manually. Manual is boring. Plus without an automated tool the results just aren’t 100% accurate.

What I wanted was a way to test 2 different theme versions without rendering the page differences on the front end during page load (and in effect slowing the page load as well).

With auto swapping themes + analytics traffic, I can gather the stats I need to make smarter design decisions.

I’ve had clients that run a lot of promotions and I remember having to set an alarm for exactly 12:01am in a certain time zone to flip the promotion on since marketing emails went out and the promotion went live.

Not all stores need this exact timing, but for the high volume stores I worked with the difference of an hour during a promotion can have a revenue swing of hundreds or thousands of dollars.

I’ve also had situations where a client did daily hour promos, needing to switch promotion banners and collection links every hour.

If I had the Theme Scheduler feature I built, I could have just created all the different themes ahead of time, set the scheduling, and let automation handle the rest while I chilled out 😎

How I priced it

I juggled a few factors to come up with the pricing:

• This is a data heavy app so server costs are going to be a factor.
• A single A/B test can lead to a decision that generates hundreds or thousands of dollars additional revenue per month, so based on ROI potential the app could have a much higher price point.
• Smaller stores don’t have a big budget so I need a low entry point.
• Larger stores will use a lot more server resources, so I need scaling price plans.

Then I thought to myself, “as a store owner, what would I want to pay for this?”

My first answer was 0 haha! But nah I figured $20 – $50 per month is a good deal, and I’m happy to pay that if I run 1 AB test per month.

Our actual prices are even lower than that, check em out here: apps.shopify.com/theme-scientist.

Logo

2021 we retired the cartoon mad scientist for the logo in exchange for a clear A/B testing graphic, but don’t worry, the mascot still lives in the app listing and some other random places. 👨‍🔬

When I get on a good kick I’ll code or hack away on a project deep into the night, feeling like a mad scientist bringing a creation to life.

That’s where scientist came into play.

This app is about theme testing so hey Theme Scientist boom there ya go 👍

I usually don’t know exactly what I want in a logo or design, so typically I’ll do a logo contest but in this case I knew what I wanted so I worked with a designer 1 on 1.

My ideas going into the design:

• Theme testing is a science, so I wanted a scientist looking character for the logo
• It’s an app so I wanted it to be fun
• I wanted him to look a little crazy, kinda like Doc from Back to the Future (I love that character!)
• I wanted him to be holding 2 different versions of something
• Since it’s an app icon I didn’t want complex features that would get lost in a small icon view

The result:

YASSSS! I love it!

I went through a few renditions to reach that point. Here are some earlier versions:

No-mustache guy has great teeth but looks too young. Brown hair guy looks good but not as wise as gray hair man.

The mini afro design is quirky I like it but it didn’t feel right for the name. The wavy hair looks nice but I was looking for something a little more crazy (but not too crazy).

Logo design is fun.

Summary (2021 update)

If you think about it, conversion rate can make or break the profitability of an ad campaign.

Theme Scientist helps you maximize conversion rate by giving data backed statistics so you know what works.

Start your A/B testing journey with a 14 day free trial of Theme Scientist 👉 Theme Scientist Shopify A/B Testing App

My goal was to make this a great tool for increasing conversion rates and helping store owners get more value from their traffic. It’s been doing that, but in trying to narrow my focus I’ve passed the torch on the app. Don’t worry it’s in good hands and will continue to evolve and get even better.

If you want to see what else I’ve built, check out my other app Order Automator. This app automates a lot of tasks for Shopify store owners, mostly based on order processing but it’s expanded to much more.