LLM-KT: A Flexible Framework for Improving Collaborative Filtering Models with Built-in LLM-Generated Features

Collaborative filtering (CF) is widely used in recommender systems to match user preferences to items, but often struggles with complex relationships and adapting to changing user interactions. Recently, researchers have explored the use of LLM to improve recommendations by leveraging their reasoning skills. LLMs have been integrated into various stages, from knowledge generation to candidate classification. While effective, this integration can be costly and existing methods, such as KAR and LLM-CF, only improve contextual CF models by adding textual features derived from LLM.

Researchers from HSE University, MIPT, Ural Federal University, Sber ai Lab, AIRI and ISP RAS developed LLM-KT, a flexible framework designed to improve CF models by incorporating LLM-generated features into intermediate layers of the model. Unlike previous methods that rely on direct input of LLM-derived features, LLM-KT integrates these features within the model, allowing it to reconstruct and use the embeddings internally. This adaptive approach requires no architectural changes, making it suitable for various CF models. Experiments on the MovieLens and amazon datasets show that LLM-KT significantly improves on baseline models, achieving a 21% increase in NDCG@10 and performing comparably to state-of-the-art context-aware methods. .

The proposed method introduces a knowledge transfer approach that improves CF models by incorporating LLM-generated features within a designated internal layer. This approach allows CF models to intuitively learn user preferences without altering their architecture, creating profiles based on user-item interactions. LLMs use messages tailored to each user's interaction data to generate preference summaries or “profiles,” which are then converted into embeddings with a pre-trained text model, such as “text-embedding-ada-002.” To optimize this integration, the CF model is trained with an auxiliary pretext task, combining the original model loss with a reconstruction loss that aligns the profile embeddings with the internal representations of the CF model. This setup uses UMAP for dimensional alignment and RMSE for reconstruction loss, ensuring the model accurately represents user preferences.

The LLM-KT framework, built on top of RecBole, supports flexible experimental setups, allowing researchers to define detailed pipelines through a single configuration file. Key features include support for integrating LLM-generated profiles from multiple sources, an adaptive configuration system, and running batch experiments with analytical tools to compare results. The internal structure of the framework includes a Model Wrapper, which oversees essential components such as the Hook Manager for accessing intermediate representations, the Weights Manager for tuning control, and the Loss Manager for custom loss adjustments. This modular design streamlines knowledge transfer and tuning, allowing researchers to efficiently test and refine CF models.

The experimental setup evaluates the proposed knowledge transfer method for CF models in two ways: for traditional models that use only user-item interaction data and for context-aware models that can use input features. Experiments were performed on amazon's “CD & Vinyl” and MovieLens data sets, using a train validation test split of 70-10-20%. Reference CF models included NeuMF, SimpleX, and MultVAE, while KAR, DCN, and DeepFM were used for contextual comparisons. The method was evaluated with classification metrics (NDCG@K, Hits@K, Recall@K) and AUC-ROC for click rate tasks. The results showed consistent performance improvements across all models, with versatility and accuracy comparable to existing approaches such as KAR.

The LLM-KT framework offers a versatile way to enhance CF models by embedding LLM-generated features within an intermediate layer, allowing models to leverage these embeddings internally. Unlike traditional methods that input LLM features directly, LLM-KT enables seamless knowledge transfer between multiple CF architectures without altering their structure. Built on the RecBole platform, the framework allows flexible configurations for easy integration and adaptation. Experiments on MovieLens and amazon datasets confirm significant performance improvements, demonstrating that LLM-KT is competitive with state-of-the-art methods on context-aware models and applicable on a broader range of CF models.

look at the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram channel and LinkedIn Grabove. If you like our work, you will love our information sheet.. Don't forget to join our SubReddit over 55,000ml.

(Sponsorship opportunity with us) Promote your research/product/webinar to over 1 million monthly readers and over 500,000 community members

Sana Hassan, a consulting intern at Marktechpost and a dual degree student at IIT Madras, is passionate about applying technology and artificial intelligence to address real-world challenges. With a strong interest in solving practical problems, he brings a new perspective to the intersection of ai and real-life solutions.