There is a race towards language models with wider context windows. But how good are they and how can we know?
This article was originally published in ai/p/long-context-llms” rel=”noopener ugc nofollow” target=”_blank”>Art Fish Intelligence.
The context window of large language models (the amount of text they can process at once) has been increasing at an exponential rate.
In 2018, language models such as BERT, T5and GPT-1 Up to 512 tokens could be accepted as input. Now, in the summer of 2024, this figure has increased to 2 million tokens (in publicly available LLM). But what does this mean for us and how do we evaluate these increasingly capable models?
The recently released ai-studio/” rel=”noopener ugc nofollow” target=”_blank”>The Gemini 1.5 Pro model can accept up to 2 million tokensBut what do 2 million tokens mean?
If we estimate that 4 words are approximately equal to 3 tokens, it means that 2 million tokens can (almost) fits the entire Harry Potter and Lord of the Rings series.
(The total word count of the seven books in the Harry Potter series is 1,084,625The total word count of the seven books in The Lord of the Rings series is 481.103. (1,084,625 +…