Sprinklr improves performance by 20% and reduces cost by 25% for machine learning inference on AWS Graviton3
This is a guest post co-written with Sprinklr's Ratnesh Jamidar and Vinayak Trivedi. sprinklers The mission is to unify silos, ...
Tag: inference
PyramidInfer: Enabling efficient KV cache compression for scalable LLM inference
LLMs like GPT-4 excel at language understanding, but struggle with high GPU memory usage during inference, which limits their scalability ...
KV-Runahead: Scalable Causal LLM Inference Using Parallel Key-Value Cache Generation
The large language model or LLM inference has two phases, the request (or preload) phase to generate the first token ...
OpenELM: A Family of Efficient Language Models with Open Source Training and Inference Framework
The reproducibility and transparency of large language models are crucial to promote open research, ensure the reliability of results, and ...
Talaria: Interactive Optimization of Machine Learning Models for Efficient Inference
On-device machine learning (ML) moves cloud computing to personal devices, protecting user privacy and enabling intelligent user experiences. However, tailoring ...
Scale AI training and inference for drug discovery through Amazon EKS and Karpenter
This is a guest post co-written with the leadership team of Iambic Therapeutics. ai/" target="_blank" rel="noopener">Iambic Therapeutics is a drug ...
Use Kubernetes Operators to gain new inference capabilities in Amazon SageMaker that reduce LLM deployment costs by 50% on average
We are pleased to announce a new release of amazon SageMaker for Kubernetes operators using the AWS Drivers for Kubernetes ...
Boost inference performance for Mixtral and Llama 2 models with new Amazon SageMaker containers
In January 2024, amazon SageMaker launched a new version (0.26.0) of Large Model Inference (LMI) Deep Learning Containers (DLCs). This ...
Meta unveils its new custom AI chip as it races to catch up
Meta, hell-bent on catching up to its rivals in the generative ai space, is spending ai-investment-priority-150500360.html">thousands of millions in their ...
OctoAI wants to make it easier to deploy private AI models with OctoStack
ai/">OctoAI (formerly known as OctoML), today announced the launch of OctoStack, its new end-to-end solution for deploying generative ai models ...
