This is part 2 of a multi-part post on website testing, part 1 is available here
Part 3 is going to include reporting features and final summaries.
In my continuing journey to open eyes to A/B and multivariate testing, I’m going to show you 2 more tools this week. On the docket: Visual Website Optimizer , Convert Experiments, and Optimizely
For purposes of this post, we are going to ignore Split URL tests. In my humble opinion, if you want to run split URL testing you only need Google Analytics Experiments. For this comparison I am going to focus on multivariate testing
The easiest way to do this comparison is to break this up into 3 sections.
- Creating test variations
- Setting up the test (implementation, targeting, etc.)
- Results and reporting (I’ll cover this in Post 3)
Creating Test Variations
Both tools are WYSIWYG (what you see is what you get). Simply put in the URL you want to test and it spits back the page with every element clickable, allowing you to edit each. You’ll notice the editing elements are fairly similar between the tools. Each platform lets you choose to “Track Clicks” to determine goal success and every platform lets you edit the element.
In all 3 platforms you are able to edit the HTML directly, perhaps to add custom JavaScript or CSS styling. In Optimizely you have an advantage of adding custom JavaScript directly (instead of editing the HTML first).
|
Visual Website Optimizer |
Convert Experiments |
Optimizely |
 |
 |
 |
On our homepage, I wanted to test the effects of pausing our auto-rotating banner. Our site sits on a DotNetNuke CMS and the only way to turn off the auto-rotator was to insert a line of JavaScript telling the normal rotating banner to load as PAUSED.
Since the JavaScript had to be applied at the page level, I could not utilize the Pause button highlighted in the screenshot below.
For this, I needed to edit the javascript of the page. The 3 platforms each have this option. For Optimizely I had to select an element and go to parent until I hit the Body, then I could edit the HTML (I felt this was very messy as it required me touching the base of the site for a very simple test.)
Visual Website Optimizer and Convert Experiments allow you to add custom JavaScript to the specific variations without needing to touch the overall site code. *Note, I verified with Convert Experiments and global is being renamed to avoid any confusion, it DOES apply only to the variation.
|
Visual Website Optimizer |
Convert Experiments |
Optimizely |
  |
 |
 |
Test Setup
Implementation
I’ll start with the part most people are going to hate (those that aren’t developers): implementing the code needed to actually execute the test. All 3 platforms have a similar process, add some JavaScript to the head of your website, and move on.
- Convert Experiments does execute asynchronously.
- Visual Website Optimizer has an option for async or standard
- Optimizely’s code is NOT async, but they do have an option for asynch if you contact them… however, the synchronous option was the easiest install with only 1 line.
All 3 platforms give a nifty “check code” after you are instructed to implement it. Optimizely even sent an email to me to let me know the code was implemented. This would be incredibly helpful if you work in an organization or an agency where someone else has to implement the code.
Targeting
All 3 platforms have targeting customization some way or the other; look at the table below to get a sense of how each is structured. I’ll note Optimizely DOES offer geo-targeting, but only at the Platinum level (my testing was done at the Gold level).
I found Optimizely’s targeting a little lacking, compared to the options available in the other 2 tools. But theoretically the Custom JS condition could add these functions, although it would require some serious JavaScript knowledge.
|
Visual Website Optimizer |
Convert Experiments |
Optimizely |
|
|
 |
 |
Traffic Allocation
I mentioned in part 1, if you have a particularly sensitive test to run, choosing what percentage of your visitors to test is a key element to stakeholder buy-in. All 3 platforms have a traffic allocator.
Visual Website Optimizer
And they have a nifty tool to help you estimate how long a test will need to run at a specific allocation. This tool will be great when you’re writing up your testing schedule.
Convert Experiments
Nothing fancy here, just straight to the point.
Optimizely
Similar to Visual Website Optimizer, you’re able to specify allocation among variations.
Analytics Integration
All three tools have native Google Analytics integration. This is done via custom variables. .
The main difference between the three tools in this area is that Optimizely is the only one that lists what’s in the variation. This is incredibly beneficial for future reporting when you want to look back to remember what a test was.
Example, both Convert Experiments and Visual Website Optimizer
Optimizely shows what each combination was. Granted, it’s truncated text, but at least it gives a sense.
But that’s all the reporting I’m going to show you now! Part 3 will have the final summary and all the reporting goodness.
Stay tuned!
