Updated June 10, 2020
🎉 Speed Boostr just released a new app: Theme Scientist!
What it does
The key feature is A/B Testing.
A/B Testing (aka Split Testing): Testing 2 different versions of a website or page to see which one converts better.
Here’s how A/B testing works with this app:
- Set it to automatically swap 2 themes every day at midnight and track daily stats
- Tracks visitors, sessions, views, adds to cart, orders, sales, conversion rate
- Let the test run and analyze the results to see which theme converts better
The idea is to test 1 feature at a time (button color, slider vs banner, page features, etc).
Update: Theme Scientist now includes Product A/B testing.
For A/B testing help + ideas, check out our A/B Testing Guide for Shopify.
There’s also a Theme Scheduler feature that lets you schedule a theme to publish at a specific time in the future. This is especially useful during promotion periods so nobody as to remember to log in exactly when you want to start the promo.
Example A/B Test
I ran a test recently on a Shopify store to see whether a blue or green add to cart button performed better.
Here are the results after 2 weeks:
|THEME A||THEME B|
Theme A with the blue add to cart button had fewer sessions but more orders + sales + higher conversion rate. Blue wins!
Maybe blue feels more trustworthy, I don’t know but the data shows that more people prefer the blue add to cart in this sample.
This isn’t a huge difference, but if this data set indicates long term preferences, that boost in conversion rate translates to large gains.
Some tests yield small changes, some yield big differences. Either way you won’t know the best solution until you run an actual test.
How to set up an A/B test?
With the Theme Scientist app, it’s easy:
- Duplicate your live theme
- Make a change to the new theme (like changing the add to cart button color)
- Select the 2 themes in the app and start the A/B split test
- Check back in a few days and analyze your stats
You can let it run as long as you’d like. A longer test = more accurate data.
I like to run tests for 2 – 4 weeks. I prefer 2 weeks minimum so each theme version gets a full day of the week.
Split test methods
There are various methods to A/B testing (daily swaps, hourly swaps, per user swaps). I like daily swaps the best.
The main reason is that people will often view your site on multiple devices. You can’t control what they see across devices, so a single daily swap ensures that user has the best chance at a consistent experience on any given day.
I think having a consistent viewing experience for a user’s session is important for a good user experience + quality data gathering.
This method has worked well for me (both on Amazon and Shopify) – I’ve had great success with Splitly AB Testing on Amazon. They use the daily swap method too.
I’ve seen A/B testing tools that modify the page in real time… and I remember being able to see the page render as it was loading (like a button changing colors during load). That’s no good.
I built this app with performance in mind.
You create your A and B versions as separate themes, then the app automatically swaps the themes on the back end, so there is no performance loss on your website.
The tracking script is a tiny 4KB file that listens for events (page views, add to cart click, sessions), then sends them to the app database. The script is minified and set to load at the end of your page so it doesn’t affect page rendering.
The build story
Man this was a journey. A lot of fun but also a lot of soul crushing debugging work 😂
I built this over the course of a few months in a combination of some part time chipping away + some intense hackathons.
Most of the coding happend at Dojo coworking space in Bali.
I’d show up around 7am, blast underground techno into my noise cancelling headphones, fire up some caffeine and go hard until about noon.
Around noon I’d walk down to the beach to get some sun and watch the waves while digesting the morning session, sometimes snacking an acai bowl.
The load of other projects + Speed Boostr management dictated how much I worked on the app the rest of the day, but that was pretty much my morning routine 6-7 days a week for the few weeks I was dialed in to bust out the bulk of the project.
The most frustrating part was probably time – keeping track of store time vs universal time, different time zones, daylight savings time, and calculating and storing time accurately.
Side note: I thought daylight savings time was dumb before… working with it in code moved it up a good notch or 2 in my book of things that suck.
The most challenging was probably building the custom analytics tracking system. Shopify analytics tracks this data but I can only tap the API for all that data if the store is on the Advanced Shopify plan or higher.
That would severely limit the user base, so I decided to build my own.
As with any large project, I learned a ton along the way and had a grand celebration the week I finished it and submitted to the app store. 🍻🥳
Why build an A/B testing app?
As competition increases, A/B testing becomes more important.
Ads get more expensive and ecommerce gets more saturated every year. This makes A/B testing a secret weapon to maximize your conversions and get a higher ROI.
I’ve always done A/B testing on Shopify manually. Manual is boring. Plus without an automated tool the results just aren’t 100% accurate.
What I wanted was a way to test 2 different theme versions without rendering the page differences on the front end during page load (and in effect slowing the page load as well).
With auto swapping themes + analytics traffic, I can gather the stats I need to make smarter design decisions.
I’ve had clients that run a lot of promotions and I remember having to set an alarm for exactly 12:01am in a certain time zone to flip the promotion on since marketing emails went out and the promotion went live.
Not all stores need this exact timing, but for the high volume stores I worked with the difference of an hour during a promotion can have a revenue swing of hundreds or thousands of dollars.
I’ve also had situations where a client did daily hour promos, needing to switch promotion banners and collection links every hour.
If I had the Theme Scheduler feature I built, I could have just created all the different themes ahead of time, set the scheduling, and let automation handle the rest while I chilled out 😎
How I priced it
I juggled a few factors to come up with the pricing:
- This is a data heavy app so server costs are going to be a factor.
- A single A/B test can lead to a decision that generates hundreds or thousands of dollars additional revenue per month, so based on ROI potential the app could have a much higher price point.
- Smaller stores don’t have a big budget so I need a low entry point.
- Larger stores will use a lot more server resources, so I need scaling price plans.
Then I thought to myself, “as a store owner, what would I want to pay for this?”
My first answer was 0 haha! But nah I figured $20 – $50 per month is a good deal, and I’m happy to pay that if I run 1 AB test per month.
I combined those price factors with my own preferred amount (as store owner) and came up with these price plans:
When I get on a good kick I’ll code or hack away on a project deep into the night, feeling like a mad scientist bringing a creation to life.
That’s where scientist came into play.
This app is about theme testing so hey Theme Scientist boom there ya go 👍
I usually don’t know exactly what I want in a logo or design, so typically I’ll do a logo contest but in this case I knew what I wanted so I worked with a designer 1 on 1.
My ideas going into the design:
- Theme testing is a science, so I wanted a scientist looking character for the logo
- It’s an app so I wanted it to be fun
- I wanted him to look a little crazy, kinda like Doc from Back to the Future (I love that character!)
- I wanted him to be holding 2 different versions of something
- Since it’s an app icon I didn’t want complex features that would get lost in a small icon view
YASSSS! I love it!
I went through a few renditions to reach that point. Here are some earlier versions:
No-mustache guy has great teeth but looks too young. Brown hair guy looks good but not as wise as gray hair man.
The mini afro design is quirky I like it but it didn’t feel right for the name. The wavy hair looks nice but I was looking for something a little more crazy (but not too crazy).
Logo design is fun.
I’m going to keep making Theme Scientist more awesome ✊
I use the app so I have my own ideas but also I’m taking requests and feedback.
Some of the features in my app Order Automator came directly from a customer asking if I could add that feature. My answer is usually YES!
My goal is to make this a great tool for increasing conversion rates and helping store owners get more value from their traffic.
If you think about it, conversion rate can make or break the profitability of an ad campaign.
This app helps you maximize conversion rate by giving data backed statistics so you know what works.
Start your A/B testing journey with a 14 day free trial of Theme Scientist 👉 Theme Scientist Shopify A/B Testing App