Recommender systems
In the domain of ai recommendation systemsMachine learning models have been widely used to recommend similar sampleswhether it be products, content, or even suggestions for similar contacts. Most of these pre-trained models are open source and can be used without having to train a model from scratch. However, due to the lack of Big Data, there is no open source technology that we can rely on for complementary product recommendation.
In the following article I propose a framework (with code in the form of easy to use library) that exploits the LLM for the discovery of complementary products in an inexpensive manner.
My goal in introducing this framework is for it to be:
- Scalable
It is a framework that should not require supervision during its execution, that is not at risk of breaking, and whose result should be easily structured so that it can be used in combination with additional tools. - Affordable
It should be affordable to find the complement of thousands of products with Minimum spend (approximately $1 per 1000 products calculated, using Groq's pricing system)also, without the need for any fine tuning (this means it could even be tested on a single product).
***The full zeroCPR code is open source and available at my Github repositoryFeel free to contact me if you need any help or if you need any features. In this article, I introduce the framework (and its corresponding library). CPR zero and a new incitement technique I call DataFrame String for list reasoning.
Before we dive into the theory behind the zeroCPR framework, let’s understand why current technology is limited in this very domain:
Why do neural networks excel at recommending similar products?
These models excel at this task because neural networks innately group samples with common features into the same spatial region. To simplify, if, for example, a neural network is trained on human language, it will assign words or sentences that have similar characteristics to the same spatial region. Similar meaningsFollowing the same principle, if you train with customer behavior in mind, customers who share similar behavior will organize themselves into similar spatial regions.
Models capable of recommending similar sentences are called semantic models, and both are Light and accessibleallowing the creation of recommendation systems that rely on the similarity of language rather than customer behavior.
A retail company lacking customer data can easily recommend similar products by exploiting the capabilities of a semantic model.
What about complementary products?
However, recommend complementary products It's a completely different task. As far as I know, There is no open source model Retail companies train their custom companion recommendation systems based on their data, resulting in models that are difficult to generalize and are industry-specific.
CPR zero It represents Zero-shot complementary product recommenderThe operation is simple. When you receive a List of your available products and reference productsI tried to find out if there are any complementary products on their list that could be recommended.
Large language models can easily recommend complementary products. You can ask ChatGPT to show you which products can be paired with a toothbrush and it will likely recommend you floss and toothpaste.
However, my goal is to create a enterprise level tool which can work with our custom data. ChatGPT may be right, but it generates unstructured output that cannot be integrated with our product listing.
He ZeroCPR Framework It can be summarized as follows, where we apply the following 3 steps for each product on our product list:
1. List complementary products
As explained, the first hurdle to overcome is finding real complementary products. Since similarity models are ruled out, we need to use an LLM. The execution of the first step is quite straightforward. Given a Input product (e.g. Coca-Cola)create a list of valid complementary products that a user can purchase with it.
I asked the LLM to generate a perfectly parsable list using Python: once parsed, we can visualize the result.
The results are not bad at all: all are products that are likely to be purchased. In pair with Coca-ColaHowever, there is a small problem: THESE PRODUCTS MAY NOT BE IN OUR DATA.
2. Matching the products available in our data
The next step is to try to match each complementary product suggested by the LLM to a corresponding product in our dataset. For example, we want to match “Nachos” to the closest possible product in our dataset.
We can perform this comparison using vector search. For each LLM product, we will compare it to the semantically most similar one in our dataset.
As we can see, the results are far from exact. “Nachos” will be paired with “SALT AND PEPPER MUSHROOM GAME”while the closest match with “Burgers” is “BEACH CABIN STOOLS S/2”. Some of the matches are valid (we can look at Napkins), but if there are no valid matches, a semantic search will still match it with an irrelevant candidate. cosine similarity threshold From experience, this is a terrible method for selecting valid options. I will use an LLM to validate the data again instead.
3. Select the correct plugins using Chain-of-DataFrame
The goal now is to validate the matching from the previous step. My first attempts to match products recommended by an LLM were frustrated by the lack of consistency in the result. Despite being a 70B model, when I was passing in the message a list of products to match, the result was less than desirable (with combinations of errors in the format and a very unrealistic result).
However, I have noticed that when entering a list of products and asking the model to… Reason in each sample and generate a score (0 or 1): (following the format of a pandas dataframe and applying a transformation to a single column), the model is much more reliable (in terms of format and output). I call this induction paradigm Chain-of-Dataframe, in reference to the well-known pandas data structure:
To give you an idea of the data frame chain request, in the following example, {Product name} is coca-cola, while the {supplementary list} is he Column called recommended_product We can see it in the image below:
A customer is doing shopping and buys the following product
product_name: {product_name}A junior shopping expert recommends the following products to be bought together, however he still has to learn:
given the following
complementary_list: {complementary_list}
Output a parsable python list using python, no comments or extra text, in the following format:
(
(, , <0 or 1>),
(, , <0 or 1>),
(, , <0 or 1>),
...
)
the customer is only interested in **products that can be paired with the existing one** to enrich his experience, not substitutes
THE ORDER OF THE OUTPUT MUST EQUAL THE ORDER OF ELEMENTS IN complementary_list
Take it easy, take a big breath to relax and be accurate. Output must start with (, end with ), no extra text
The result is a multidimensional list that can be easily parsed and immediately converted back into a Pandas Dataframe.
Observe the reasoning and score columns generated by the model to find the best complementary productsWith this last step we have been able to filter out most of the irrelevant matches.
***The algorithm may look similar to CHAIN OF TABLES: EVOLUTION OF TABLES IN THE CHAIN OF REASONING FOR UNDERSTANDING TABLESbut I think the above proposal is much simpler and uses a different structure. Feel free to comment if you think otherwise.
4. How to handle little data: Filling with the closest substitute
There is One last question we need to addressDue to a lack of data, the number of recommended products is likely to be minimal. In the example above, we can recommend 6 complementary products, but there may be cases where we can only recommend 2 or 3. How can we improve the user experience and increase the number of valid recommendations, given the limitations imposed by our data?
