Before we dive into optimizing predictive analytics for images using #RealTimeML, at our neighborhood Email Service Provider, there are a few people we need to acknowledge. First, we would like to recognize the Stanford Digital Economy Lab and its managing director Christie Ko. Christie reached out to us to potentially write articles for them, and we talked about several topics in the world of Machine learning (ML). She found our blog here on CircleID and, with the collaboration of Erik Brynjolfsson, reached out to discuss how we can further help the academic community better appreciate the pervasive nature of RealTimeML in real-world environments such as email campaign building. With great humility, the entire team at Loxz is honored that they had the impetus to reach out.

The other entity we would like to acknowledge is an organization called BrightApps based in Walnut Creek, CA. Run by Greg McGregor and his team, BrightApps became the first organization we partnered with after revealing our RealTime text and image email recommendation models on a Zoom Call. It is inspiring to roll out these nascent models to strategic partners because it affirms that the need for an “in-session” real-time email recommendation is warranted. We are collaborating on a singular model, and if proven successful, we can roll it out officially as a pilot with an ESP we are targeting. These are exciting times for both organizations.

Now, let us get into Real-Time Predictions with Image Optimization in Email. Images are the key to enhanced click-through rates. Nevertheless, when building our campaigns, upon uploading an image into your email editor, you have absolutely no idea how that image will be perceived from an analytic perspective. You have a hunch based on some historical data, but that mindset is so 2020. With a real-time email recommendation engine, you no longer have to guess, so do away with dynamic emails, and focus on your target variable.

RealTimeML with Optimized Images

AI has become the universal engine of execution and enables a rapidly growing number of tasks and processes. Email is not different. To execute email campaigns effectively now and well in the future, the operational foundation of a business and the core of a company’s operating model is defined by how the company drives the execution of tasks.

Concurrently, AI is becoming a force in the arts, connecting various disciplines and media and expanding the range of artistic possibilities. Your ESPs image editor will change dramatically as well, allowing you to augment data and enhancing the styles of the images you upload. Transforming images is not necessarily new, but the kinds of style-transfer techniques used across a broad variety of subjects from media to film to music allows engineers or even campaign builders to replicate existing styles that is far more genuine then we’ve ever imagined. AI is being used to create completely new works of art and is transforming the method for crafting the work.

One of our data scientists, Mateo Martinez, credentialed at UCLA, used a Google Colab Notebook to run various optimization models on the contents of an email. The original model he built was a text-based optimization model described here last month. Today, we will describe image optimization in an email. The sheer versatility of reimaging images and, even more powerful, informing the campaign builder with real-time analytics before the campaign is deployed makes this application unique.

This model essentially provides predictive analytics on the images used for digital marketing campaigns, specifically emails. In preparing your campaign, there are significant steps to take before you run the model, and they include but are not limited to selecting your target variable, selecting the industry your organization wants to target, and then selecting a type of email campaign you will run. Note: this model was explicitly trained using automotive images.

The Data

The dataset used in this analysis is from the Stanford AI Lab Repository. The dataset initially contained 16,000 images of cars. From there, in the Data Wrangling notebook, we generated synthetic data that would be typical for an email digital marketing campaign, which will use to train this machine learning model.

The Model

We have currently developed accurate predictive analytics models on word count, images, call-to-action, links, and the development of a sentiment analysis model on the email text is upcoming. This machine learning model was developed using the Python programming language using various machine learning algorithms such as Random Forest, XGBoost, and Convolutional Neural Networks to optimize for many different variables. The model is constructed to analyze the contents of an email campaign or any HTML or JSON document and parse its contents to convert the document into an array of structured data and information. The user of this algorithm would additionally input what variable they wished to optimize for. In this model, we will use an example of Click Through Rate.

Analyzing the sentiment of an email will be described in an upcoming post. (You might have the vision to understand how vital sentiment analysis is in an email if you have ever received an email from PGE about a proposed power outage or a government office such as your local police department). For now, we will focus on images:

In the Feature Engineering notebook, we extracted image embeddings using a pre-trained Convolutional Neural Network. The specific model we used was ResNet34-a model with 34 layers, which was pre-trained on the ImageNet dataset. We then used these image embeddings as inputs into a Gradient Boosted Tree model to generate more accurate probabilities on a user-specified target variable. Given the selected variable to optimize, the model would then report the conversion probability on the target variable with its current state and provide conversion probabilities if the target variable was altered with synthetic data:

1. Convolutional Neural Network using ResNet34 & PyTorch
2. Gradient Boosted Tree Model using XGBoost

The Metadata:

• relative_im_path: the file path to the actual image
• class class_name: the full description of the vehicle
• make: the make of the vehicle
• model: the model of the vehicle
• vehicle_type: the type of vehicle
• year: the year of the vehicle
• click_through: a binary value for a click-through event
• unsubscribed: a binary value for a unsubscribe event
• open_rate: a binary value for an open rate event
• delivery_rate: a binary value for a spam event
• campain_type : [Welcome, Transactional, ‘Engagement_Campaign’, ‘No_Opener’, ‘Revenue_Based’, ‘Promotional’, ‘Survey’, Abandoned Cart]
• industry : [‘Hospitality’, ‘Real_Estate’, ‘Automotive’, ‘Industrial’, ‘Medical’]

Using historical data from the campaign builders’ history and cohorts of other marketers within the same industry, the models dives deep to find optimal images of cars. Keep in mind that if you are using a data-centric model and introduce new datasets to make your model more accurate, this also can be achieved. At the outset, you have to determine whether this model will be more data-centric or have only finite data and have to enhance the algorithm. In this case, you would be looking at a model-centric approach.

This could further be enhanced in your styling editor of images to know if a particular image would convert better with a “black-background” or background of a beach.” If your styling editor supports 3D or uses augmented images, you can run the model to see if there is a higher conversion rate with 3D images of the car you want to feature. Maybe an augmented image of the convertible Aston Martin will provide a much better CTR in the specific industry you targeted. Perhaps a convertible Porsche would convert with a beach background than a “Volvo” on a showroom floor?

How do you know? By the time your subscribers click on the image or link, it is way too late.

Let us explain the process! In your ESPs email editor, you upload the image of your choice when building out the email campaign. Remember, you can tailor this campaign and be very specific on the type of analytics you want to uncover.

In this example, you upload an image of an Aston Martin. While we channel our inner James Bond, we feel reasonably confident that the Aston Martin will convert well, but we have no real-time analytics. After optimizing your target variables, in this case, the

Target Variable=CTR and create the framework for your inputs using a dropdown: as shown here, you are still reasonably sure that the Aston Martin will convert well: You make these inputs below:

1. Click-Through-Rate
2. Campaign Type: Promotional
3. Industry: Hospitality

Voila: Here are the predictions:

Based on your selections, a “promotional” based campaign in the “hospitality” industry seems to appreciate a later model Mercedes Convertible over the Aston Martin. How absurd! Anyone in their right mind would have clicked on the Aston Martin, but they like the convertible Benz.

In our models, we do our best to explain the models as to why Image 1 was selected. In this case, image one, the Mercedes receives a 75% click-through rate while the Aston Martin claims merely a 48% CTR. That added 27% is the lift you’ll receive in CTRs. That’s the magic of realtimeML, before a campaign is deployed. In deep learning neural networks, explainability is more challenging, but we could get enough metadata from the dataset to do our best to explain why Image 1, (the Mercedes) was selected. Of course, with a reputation like MailChimp, there probably would be no need to explain the model to your clients, but a little detail about why the model chose image one would help. Here is our explanation:

Conclusion

This machine learning model recommends using an image similar to Alternate Image 1 as it has a higher click_through probability. Note: The image recommendations were derived from a Neural Network (a black box model). The reasons and patterns behind these predictions are not discernible from this Neural Network model; however, the Neural Network was trained to differentiate between vehicle types (SUV, Coupe, Van, Convertible, etc.). Thus some explainability can be derived. Given that an image of a convertible had the highest probability, it can be deduced that images of convertibles will be the most effective.

Well, that is Part III of our email recommendation series. We have two more models to share in the upcoming weeks and months. In part IV, we will show you how to optimize for links in an email and the best combination of text and links within specific industries and types of campaigns. How many links are too many in a particular industry? Will we find out?

A special thank you goes out to one of our data scientists, leading efforts on the email recommendation engine. His name is Mateo Martinez. He’ll be happy to walk you through a demo on this model or any of the previous models we’ve examined here.

Update Jun 22, 2022: Abstract to Updated Image Optimization Report: Using ML for Predictive Email Engagement Metrics within the workflow of a campaign (Published June 10, 2022)

While much of big tech has been using predictive analytics in real-time ( for example, LinkedIn will tell you how many clicks to expect from a campaign), it has yet to trickle down to the Email Industry. MailChimp has provided some insights in your dashboard after the campaign has been sent, but none of the other major ESPs have built-in predictive metrics prior to the campaign being sent.

Abstract: The goal of this machine learning predictive analytics model is to serve accurate email metric predictions and recommended images prior to a campaign send for timely feedback of email-related marketing campaigns. Whether it’s one-to-many for a promotional campaign, a highly curated segment or a one-to-one transactional email, the goal is to serve these predictions in milliseconds after the model is run by the campaign engineer. In this full report, we utilize an automotive image dataset and generate additional recommended images using a data augmentation technique and inference pipeline. This particular prediction model utilizes the CNN image-based ResNet model, and we performed transfer learning using the existing parameters and weights from an ImageNet dataset. The resultant model is then further improved using XGboost classification algorithms to increase its accuracy. The training and deployment are conducted on AWS Sagemaker. Our algorithms obtained an accuracy score of 91% using the testing dataset. The demonstration of this model is to will identify the best engagement images for the target variable chosen by the campaign engineer and provide potential or recommended images with a higher engagement rate. For demonstration purposes, this model was trained specifically using an automotive related dataset, but can be tuned for other types of imagery depending on the dataset methods.

The IO Model is 1 of 9 different models built for Email Campaign Engineers. You can read the full report here.

Published: June 10th, 2022
Author: Miu Lun (Andy) Lau, Data Scientist