How the modern rise of ai has completely revolutionized search applications…
The recent rise of generative ai and the arrival of large language models (LLM) has led many to wonder about the evolution of search engines. Will dialogue-based LLMs replace traditional search engines, or will these models' tendency to hallucinate make them an unreliable source of information? The answer to these questions is currently unclear, but the rapid adoption of ai-centric search systems such as you.com and ai/” rel=”noopener ugc nofollow” target=”_blank”>perplexity.ai indicates widespread interest in augmenting search engines with modern advances in language models. Ironically, however, We have been intensively using linguistic models in search engines for years.! The BERT proposal (1) led to a step-function improvement in our ability to evaluate semantic textual similarity, resulting in these language models being adopted by a variety of popular search engines (including Google!). In this overview, we will discuss the components of such ai-powered search systems.
Search engines are one of the oldest and most widely used applications of machine learning and artificial intelligence. Most search engines are basically made up of two basic components (depicted above):
- Recovery– From the set of all possible documents, identify a much smaller set of candidate documents that could be relevant to the user's query.
- Classification– Use further analysis to sort the set of candidate documents so that the most relevant documents are displayed first.
Depending on our use case, the total number of documents we search can be very large (for example, all products on Amazon or all web pages on Google). As such, the search retrieval component must be efficient: Quickly identifies a small subset of documents that are relevant to the user's query.. Once we have identified a smaller set of candidate documents, we can use more complex techniques: like larger neural networks or more data — to optimally organize the…