A/B Test Summary
Prev
Next
This conversion rate optimization summary was inspired by GoodUI FastForward. It is a way to consolidate many A/B test into typical paterns with the aim of improving the conversion rate.
Aim
- With alle tests we aimed the optimization of the conversion rate. The conversion rate was measured usually by the completed eMessages, these are the forms filled and sent from visitors. Quote requests is a particular type of eMessages – the hottest and more wanted.
- Conversions were usually not measured only on the page where the experiment took place, but anywhere on the site.
Site
- Al these tests were performed on the METTLER TOLEDO official website between 2013 and 2018
Mock ups
- A: is the control group
- B, C, D, … : are testing variations
Certainty
- Why do we compare to 1.645?
- A Z-Score of greater than 1.645 indicates that there we can be 95% confident that the conversion rate of the variation is greater than the conversion rate of the control group. This means that there is only a 5% chance that the conversion rate is less than or equal to the conversion rate of the control group.
- Why 95% Confidence?
- 95% is the confidence level that the statistics community has come to agree is the best in the most situations. It is a balance between sample size (and thus time required to run the test) and assurance that the result is correct. You can calculate for higher or lower confidence levels, but 95% will be the best for most situations. If a test needs to be completed quickly switch to a lower confidence level. For more critical applications you may want to switch to a higher confidence level.
- 😀, 🙁
- Denote statistical significant result
- (😀), (🙁)
- Denotes almost statistical significant result
- Some tests are also published on GoodUI patterns
Conversion rate optimization patterns
Pattern 1
- Category: Banner
- Teasers on a page
- Effect (lift): -3.7% eMessages
- Certainty (Z score): 0.65
- Learning: Supportive content can negatively impact triage.
Pattern 2
- Category: Design
- Content alignement and thumbnail images
- Effect (lift): +16.9% eMessages
- Certainty (Z score): 2.31 😀
- Well structured design elements and thumbnail images help visitors to navigate and convert.
Pattern 3
- Category: CTA
- Many call to actions
- Effect (lift): – 13.4% eMessages
- Certainty (Z score): -0.46
- Exagerate number of call-to-actions can be counterproductive.
Pattern 4
- Category: Information architecture
- Links to product model pages
- Effect (lift): +6.3% eMessages
- Certainty (Z score): 0.70
- Offering links to deeper information can fullfill visitor’s needs.
Pattern 5
- Category: CTA
- Duplicated buttons
- Effect (lift): -1.3% eMessages
- Certainty (Z score): -0.21
- Repeating the same of call-to-actions too many times can be counterproductive.
Pattern 6
- Category: Information architecture
- Introduction before triage
- Effect (lift): +31.4 eMessages
- Certainty (Z score): 1.98 😀
- A short introduction and an illustration is recommendet to explain and introduce complex topics.
Pattern 7
- Category: Information architecture
- Additional links
- Effect (lift): -54% eMessages
- Certainty (Z score): 1.73 🙁
- Exagerate number of additional links is counterproductive.
Pattern 8
- Category: CTA
- Instant quote
- Effect (lift): +13.3% eMessages
- Certainty (Z score): 1.33
- Getting an immediate price information is a real need.
Pattern 9
- Category: CTA
- Short-cut to Webinar
- Effect (lift): +76.8% eMessages
- Certainty (Z score): 1.41
- A short-cut button fullfill visitor’s expectation.
Pattern 10
- Category: CTA
- Button and inline form in same page
- Effect (lift): +6.0% eMessages Visitors are confused if CTA’s are contradictory.
- Certainty (Z score): 1.55 (😀)
- A short-cut button fullfill visitor’s expectation.
Pattern 11
- Category: CTA
- Quality badges and button
- Effect (lift): +3.5% eMesaages, +4.9% Quote requests
- Certainty (Z score): 0.45; 0.48
- Information about quality certificates can improve conversions.
Pattern 12
- Category: CTA
- Very long button label
- Effect (lift): -16.9 eMessages
- Certainty (Z score): -2.86 🙁
- Avoid illegible long button labels.
Pattern 13
- Category: CTA
- Button variations
- Effect (lift): B: +1.6% eLeads; C: +2.5% eLeads; D: +2.3% eLeads; E: +1.2% eLeads; F: -3.8% eLeads
- Certainty (Z score): B: 0.45; C: 0.70; D: 0.64; E: 0.32; F: 1.04
- Clear visual differienciation between two buttons is welcome.
Pattern 14
- Category: Information architecture
- Button Introduction before triage
- Effect (lift): +17.7% eMessages; -1.9% Quote requests
- Certainty (Z score): 1.58 (😀); -0.11
- A short introduction and an illustration is recommendet in complex topic.
Pattern 15
- Category: Information architecture
- Vertical and horizontal comparison
- Effect (lift): -30.6% Quote requests
- Certainty (Z score): -1.4
- Horizontal comparaison is difficult.
Pattern 16
- Category: Design
- Form design
- Effect (lift): +1.0% eMessages
- Certainty (Z score): 0.55
- No learning.
Pattern 17
- Category: CTA
- Variations of primary and secondary buttons
- Effect (lift): B: +5.3% eMessages; C: 6.2% eMessages; D: +3.2% eMessages (grey buttons was not creating an eMessage)
- Certainty (Z score): B: 0.27; C: 0.31; D: 0.16
- One short clear mainingfull button is best.
Pattern 18
- Category: Design
- Help at the end of the page to prevent bounces
- Effect (lift): +0.4% eMessages
- Certainty (Z score): 0.65
- No learning.
Pattern 19
- Category: CTA
- Quality badges
- Effect (lift): +3.2% quote requests
- Certainty (Z score): 0.74
- Quality certificates with a strong call-to-action can improve conversions.
Pattern 20
- Category: Design
- Mood image and removal of links
- Effect (lift): +7.8% eMessages
- Certainty (Z score): 1.60 (😀)
- Remove links leading away from conversion.
Pattern 21
- Category: Video
- Missing video (in China)
- Effect (lift): -13.5% quote requests
- Certainty (Z score): -0.44
- Bad user experience can lead to reduced conversions.
Pattern 22
- Category: CTA
- Promotion
- Effect (lift): +18.2% quote requests
- Certainty (Z score): 3.18 😀
- Adding a promotion with a quote request is very effective.
Pattern 23
- Category: CTA
- Overlay VS in-line form
- Effect (lift): +24.5% eMessages
- Certainty (Z score): 3.70 😀
- Avoid inline forms in favor of overlay forms.
Pattern 24
- Category: Information architecture
- Video/image preview of webinar
- Effect (lift): B: -34.7% eMessages; C: +1.2% eMessages
- Certainty (Z score): B: -1.2; C: 0.05
- Video preview of webinars reduces webinar registrations. It is better to use a meaningful image.
Pattern 25
- Category: Information architecture
- Character of the image
- Effect (lift): +10% eMessages
- Certainty (Z score): 0.81
- Human image subject is more personal and emotional.
Pattern 26
- Category: Information architecture
- Mood/hero image
- Effect (lift): -22.6% eMessages
- Certainty (Z score): -1.70 🙁
- Mood or hero images have negative impact on conversions.
Pattern 27
- Category: Information architecture
- Mood image
- Effect (lift): -5% eMessages
- Certainty (Z score): -0.73
- Mood or hero images have negative impact on conversions, even if the call-to-action is above the fold.
Pattern 28
- Category: Information architecture
- PDF preview
- Effect (lift): +9% eMessages
- Certainty (Z score): 1.31
- PDF preview can increase conversions if well made.
Pattern 29
- Category: CTA
- No form
- Effect (lift): +374% eMessages
- Certainty (Z score): 10.25 😀
- Avoid using a form can boost conversion by 4 times. Of couse the challenges is the identification of visitors.
Pattern 30
- Category: Information architecture
- Marketing information around triage
- Effect (lift): -11.1% eMessages initiated; +0.5% eMessages completed; +1.7% Hot/Warm eMessages
- Certainty (Z score): -3.54 🙁; 0.06; 0.21
- Enriching a page with too much background information can put conversions and triage at risk
Pattern 31
- Category: CTA
- Mobile form reduction
- Effect (lift): +54.3% eMessages completed
- Certainty (Z score): 3.1😀
- Reduce a form for mobile visitors from 6 fields to 1 field is very effective and can double conversions.
Pattern 32
- Category: CTA
- Service request
- Effect (lift): +8.5% eMessages; +6.4% Quote requests
- Certainty (Z score): 1.2; 0.7
- Service offering request and Service phone number influence positivly quote requests.