How to A/B test catalogs

Knowing which design will perform best in the real world is impossible. You can guess, and use your “gut feeling”, but there’s no substitute for testing and getting objective data from your actual audience. 

Just because a design seems better doesn’t necessarily mean that it will bring in more revenue - that’s where A/B testing comes in.

A/B testing is an essential tool in any data-minded advertiser’s toolbox. It allows you to test different designs against each other to figure out what adjustments you should make to your product feed designs to reach the highest possible conversion rate.

However, it is extremely important that you do this test properly and carefully read this guide, to avoid certain mistakes that could give you invalid results, and to be able to figure out which design changes actually improve your paid social performance.

This guide outlines the correct method for setting up a test with your product feed, where you’ll gain valuable and relevant insights into what to do to optimize your Dynamic Product Ads.

Overview:

• Create new feeds with new designs in Confect (one feed for each new design being tested)

• Create new catalogs in Facebook (one new catalog for each design being tested, including a new catalog for the original design)

• Duplicate the current DPA campaign for the new catalogs (one new campaign for each catalog being tested, including a new campaign for the original design)

• Start the A/B test (using the Experiments tool in Facebook Ad Manager)

• Evaluate the test results and decide next steps

Setting up the test

This step-by-step guide ensures you’ll understand how to execute a proper A/B test and gain data that you can apply to your designs.

1. Create new feeds with new designs in Confect

Before starting the test, you’ll need some designs to compare to the original. That’s why the first step is to create a few different designs, and set up separate “Feed Automations” for each of them.

You can test the original against a single new design, but we recommend aiming for 2-3 new designs against the original.

You have the choice of testing one design variation at a time (i.e. backgrounds) or comparing completely different designs with many variations between them.

• If you test completely different designs against each other, you will find out which performs best, but not why. By trying multiple designs at the same time and comparing between them, you will likely be able to find and implement a creative with significantly improved results after just one round of A/B testing.

This is recommended for new Confect users, currently using non-customized DPAs, who want to significantly improve their ad performance in a relatively short period of time.

You can get inspired by our Ad examples, or The ultimate guide for designing DPAs for Facebook.

However, once you find a design with much higher performance than your original, it is difficult to optimize it through this method, and it’s recommended to test a single variation at a time after that.

• If you test one design change at a time, you will get definite answers to questions like: “Should my design have a logo?”, “Will a CTA improve conversions?” and etcetera. 

Step-by-step, you will be getting closer to the absolute best design for your specific brand and audience. This is much more long-term sustainable than the first option.

Also, keep in mind that different campaign objectives will require different design changes to optimize them, and different metrics to measure success. It doesn’t make sense to compare a lead generation design to a conversion design. 

Similarly, a retargeting design shouldn’t be compared to a broad-targeting design, as they go for different audiences.

2. Create new catalogs in Facebook

Once you have your new designs and “Feed Automations”, we need to create new catalogs in Facebook Ad Manager.

If you’re testing 2 new designs against the original (your currently running catalog), you’ll need to create 3 new catalogs (2 for the new design and 1 for the original).

It is extremely important to create a new catalog for the original feed that’s currently running. Facebook saves product-interest data in the catalogs. Just like they save data in campaigns, they do in catalogs as well - all the data about who is interested in which products.

In a comparison between newly created catalogs, and a catalog that has been running for some time, the older catalog will ALWAYS win - giving you non-reliable data!

Also, don’t delete your currently running catalog, as it stores the valuable learning data.

Follow these steps to create new catalogs:

• In Facebook Ads Manager top-left menu, open the “Commerce Manager”

• Click “Add Catalog”

• Select “Ecommerce” as your catalog type

• Make sure that “Upload Product Info” is selected. Then, name the new catalog so you know which is which, and click “Create”.

• Click “View Catalog”

• In the left-hand menu, click “Data sources”
• Select “Data feed” under “Add items”, and click “Next”

• When asked “Are you ready to upload your spreadsheet or file?”, select “Yes” and click “Next”
• Click on “Use an URL”
• Enter the URL of the feed - either the original URL of your feed, or one of the URL provided by the Confect “Feed Automation”, based on which catalog you’re creating
• Make sure to leave the “Username” and “Password” fields blank

• We recommend selecting “Hourly” updates, as this is how often Confect updates the feed

• Name your data source to keep the A/B test neat and organized, select the “Default Currency”, and click “Next”

• Finally, click “Save feed & Upload” - your new catalog is now created

Follow these steps once for your original feed, and once for every new Confect feed you’ve created.

If you’re testing the original against 2 new designs, you should have the following catalogs in your Commerce Manager:

• Your current catalog (has been running for some time, will not be included in the test)

• New Confect designs #1

• New Confect designs #2

• New catalog for current design (same images as “Your current catalog”, but is brand-new, so it isn’t optimized = fair comparison)

It is extremely important to not include the original catalog, which has been running for some time, in the test, but instead create a new duplicate of it.

In this example image, the catalogs to be used in the test are highlighted.

You are now ready to move on to creating campaigns for your test.

3. Duplicate the current DPA campaign for the new catalogs

It’s important that all the catalogs, and their corresponding campaigns, have the same campaign settings, so the only changing variable is the design itself. That is why it’s best to duplicate the existing DPA campaign.

Go into your “Ads Manager” under “Campaigns”.

First, turn off your current campaign, so there’s no overlap.

Duplicate your existing campaign once for each catalog (one for each new Confect feed, and one for your original feed).

So, if your test consists of your original feed being compared to 2 new Confect feeds, you’ll need to duplicate your current DPA campaign 3 times.

Do not publish them yet, otherwise you won’t be able to change the catalog. To individually edit them, click on “Edit each campaign”.

Make sure to rename them to be able to keep track of the catalog being used, and then click “Save to draft”.

Now, you will have to change the catalog being used in every campaign individually. Make sure to set the catalog according to the name of the campaign for an organized test.

Scroll down to “Catalog” and make sure to change this to the correct one for this campaign.

Remember to change the catalog on every single copy created (highlighted on the left) and only after doing this, click “Publish”.

Turn off all campaigns, as you’ll be using the Experiments tool in the next step to start the A/B test.

So, if you’re testing 2 new feeds against the original, you should have the following DPA campaigns:

Old campaign with the original feed catalog 

• Turned off, will not be running while testing

New campaign with a copy of the original feed catalog

• Using the original feed designs, but changed to a newly created catalog

New campaign with Confect feed catalog no. 1

• Using a catalog with the first feed variation from Confect

New campaign with Confect feed catalog no. 2

• Using a catalog with the second feed variation from Confect

4. Start the A/B test

You now have the foundation for your A/B test ready. The next step is to create and start the test in Facebook Ad Manager so you can compare the performance and reach conclusions.

To do this, simply follow the guide by Facebook: Create an A/B test in Experiments tool

• In this guide, in step 4., select “Test another campaign”, instead of “Duplicate an existing campaign”, as we’ve already duplicated your campaign and assigned them different catalogs.

5. What to do after the test has been completed?

Once the entire duration runs its course (usually 7-30 days), you will be able to compare the performance and see the best-performing variant. This depends on the metric you’re optimizing for, but with DPAs, you’ll most likely be concerned about the “Conversion Rate” and “Return On Ad Spend”.

Since the only difference between these campaigns is the design, you can see which creative brings in the most results.

You can now go into your old catalog (the one that has been running for some time), and replace the original feed URL with the winning-feed URL. 

You can do this under “Commerce Manager” -> “Catalogs” (Click on the old one) -> “Data sources” -> “Data feeds” -> “Settings” -> “Schedules” (Click “Edit” and replace the original URL)

• If you want to keep optimizing further (which we highly recommend), use the winning-feed from this test as the baseline for future tests. So treat the winning-feed as the “original” and repeat this guide.