A/B Testing

2017 Land Rover Discovery Emails

With the arrival of the All-New 2017 Land Rover Discovery in approximately 1 month, I had been given the opportunity to A/B test a couple of content strategies within an email campaign that I have been pondering.

1.) Which is the more enticing lure, A. LOW STARTING PRICE or B. SOON ARRIVAL?

and

2.) Which visual stimuli reaps more audience activity, A. IMAGE EMPHASIS OVER TEXT or B. TEXT EMPHASIS OVER IMAGE?

The results of these 2 A/B tests will be used to craft an email campaign for the New Range Rover Velar, which launches in 4 months, based on quantitative analysis of these emails.

All variables remained the same (subject line, time of delivery, etc.), apart from content variables A and B – pertaining to their respective assessments.

Care was taken to not overwhelm, and therefore lose prospects and clients, with multiple emails filling their inboxes – so Test 1 experimented on Bellevue residents whilst Test 2 experimented on Seattle residents.

1.) Price over ETA

New Discovery Email AB Test - IMAGES 3 Final v 3 - Price not Arrival
New Discovery Email AB Test - IMAGES 3 Final v 3

A. Starting Price:

30.6% open rate ( +15.1% above industry average) | 2.9% click-through rate ( +1.1% above industry average)

B. Arrival Time:

33.4% open rate ( +17.9% above industry average) | 1.8% click-through rate ( industry average)

Although the opening rate was greater for B than it was for A, subject line and title variables were uniform between both of them. What is being tested is the recipient’s behavior once the email has been opened.

Therefore Test A is superior performer, having a significantly higher click-through rate than Test B ( by +1.1%).

These results reflect that Bellevue prospects are better enticed by an email emphasizing an appeal to A. Starting Price, rather than, B. Soon ETA.

1.) Visual over Textual

New Discovery Email AB Test - IMAGES 3 Final 2
New Discovery Email AB Test - Text Final

A. Predominately Imagery:

23.3% open rate ( +7.8% above industry average) | 2.8% click-through rate ( +1% above industry average)

B. Predominately Text:

29.1% open rate ( +13.6% above industry average) | 1.6% click-through rate ( -0.2% below industry average)

As it had been for Test 1, the opening rate was greater for B than it was for A. However, subject line and title variables were uniform between the two of them.

Email A. still performs better, having a significantly higher click-through rate than Test B ( by +1.2%).

Of course, more trials will need to be done to minimize the room for error. However these initial results reflect that Seattle prospects are better lured by an email expressing more visuals at the expense of lengthy text. A. Imagery wins over B. text.

Memorial Day Weekend Sales Event Emails

For this A/B test, my intention was to compare the open rates of emails displaying  two different subject lines.

1.) Which garners higher open rate?

 A. ‘Memorial Day Sales Event’

or

B. ‘Complimentary 3-Year Prepaid Maintenance Plan’

The variables were exactly the same, and the content of both emails were identical. However, one (A) simply lets the recipient know that it is a Holiday Sales Event, perhaps subconsciously inciting a sense of urgency in that holidays tend to be brief and could be a missed opportunity – (B) states the actual offer of the event, which is a Complimentary maintenance plan.

1.) Actual Offer over Sales Event

Memorial Day LR Bellevue Email revised

A. Actual Offer:

36.9% open rate ( +22% above industry average)

B. Sales Event Title:

30.5% open rate ( +15.6% above industry average)

The better performing Subject Line was Test A (stating ‘Complimentary 3-Year Prepaid Maintenance Plan’). The open-rate bested Test B by +6.4%.

These results reflect that Our Bellevue & Seattle prospects are more likely to open an email when it initially communicates what the offer entails, rather than merely stating the sales event title.

1.) Orange Vehicle over Red Vehicle

The psychology of colors in marketing, has always fascinated me. As if it were magic, labeling something green can denote health or positive financial undertones – or how blue can convey purity and cleanliness or tranquility.

In this A/B test I wanted to find which color of vehicle entices more shoppers to our website by assessing the click-through rate of either email.

1.) Which garners higher click-through rate?

 A. ‘Orange F-TYPE’

or

B. ‘Red F-TYPE’

I made sure every variable of either email were identical, and also that the images themselves were exact apart from their difference in color (I ensured this by editing the orange F-TYPE image to become red). According to the psychology of color in marketing, email A. (Orange Vehicle) should promote optimism or a sense of caution and slight anxiety that might capture impulsive buyers. However, email B. (Red Vehicle) should incite a sense of urgency and encourage appetite or stimulate the body – which is why fast-food and retail typically use this color.

A. Orange Vehicle (Jaguar F-TYPE):

35.4% open rate ( +19.9% above industry average) | 2.5% click-through rate ( +7% above industry average)

B. Red Vehicle (Jaguar F-TYPE):

35% open rate ( +19.5% above industry average) | 2.3% click-through rate (+5% above industry average)

Both click-through rates are above the industry average, however email A. (Orange Vehicle) wins over email B. (Red Vehicle) by a slightly higher +0.2%.

Of course, more trials will need to be done to minimize the room for error. All other variables were kept uniform between both emails, however this finding could be as a result of the customers’ preference for the Orange F-TYPE over the Red F-TYPE. Perhaps if there is a preference, this might be a reflection of the aforementioned psychology of colors play in marketing.