This little research was inspired by an article by Mark Ritson in Marketing Week, titled “Synthetic data is suddenly making very real ripples“. The article discusses a new approach to quantitative marketing research. Instead of surveying hundreds to thousands of people, this approach uses data from large language models like ChatGPT. According to research conducted by several academics who have performed comparative analyses of data from this method and traditional methods, the difference is surprisingly small.
The case in the article focused on building a perceptual map—a chart that illustrates the similarities between brands and how they are perceived by customers. The illustration below (sourced from the paper linked above) shows such maps built using various large language models and standard marketing research approaches. As you can see, they are very close; the paper claims a 90% similarity.
So, I decided to try building a similar perceptual map for my area of specialization—ecommerce—and analyze leading ecommerce platforms.
eCommerce platforms I selected initially:
- Magento
- Shopify
- SAP Hybris
- BigCommerce
- WooCommerce
- Shopware
- Commercetools
- Salesforce
- VTEX
- Spryker
- Oro
- Elastic Path
- Optimizely
- Kibo
- Sana Commerce
I used ChatGPT with Advanced Data Analysis, which is quite handy since it can assess not only individual similarities but also build a whole matrix and then draw the chart.
As a test, I ran a prompt based on the model from the research. In the context of ecommerce platforms, it sounded like this:
I want you to act as a person filling out a survey. I will ask you a question and you must answer using only an integer, no words. You will reply with an integer between 0 and 10. My first question is in your opinion, how similar are the ecommerce platforms BigCommerce and and Shopify on a scale of 0 to 10 where 10 means very similar?
ChatGPT answer was 8.
Then I asked it to build a matrix of similarities for all the platforms listed and construct a perceptual map.
Here is what I have got as a result:
The closer the platforms are on this chart, the more similar they are perceived to be.
What do we see here:
- Shopify is similar to BigCommerce
- Oro is close to Spryker
- Optimizely is close to Elastic Path
- Magento has some similarity with Hybris and Salesforce
- Salesforce is not far from Commercetools
Then I tried to slightly modify the approach and asked ChatGPT to specifically assess the similarity of ecommerce platforms as brands, through the eyes of consumers. The results are below.
Very close to original chart, not many differences.
Upon reviewing the matrix, I found that many platforms have very low similarity with all other platforms, probably reflecting their lower popularity and lack of data about them in the language model database. So, I asked ChatGPT to exclude such platforms. Here is the resulting reduced chart.
The original similarities listed earlier still stand, just with a bit less noise.
So, that’s how online platforms are perceived by people, at least if we trust synthetic research.
Do you agree or not?
Reference: ChatGPT transcript