LLM CPU-GPU I/O-Aware inference reduces latency on GPUs by optimizing CPU-GPU interactions
LLMs are driving important advances in research and development today. There has been a significant shift in research objectives and ...
Tag: inference
Combining large and small LLMs to increase inference time and quality | by Richa Gadgil | December 2024
Implementation of speculative and contrastive decodingLarge language models are composed of billions of parameters (weights). For each word it generates, ...
Advancing Medical AI: Evaluating OpenAI's o1 Preview Model and Optimizing Inference Strategies
Medprompt, a runtime steering strategy, demonstrates the potential to guide general-purpose LLMs to achieve cutting-edge performance in specialized domains such ...
Unlock cost savings with the new scale down to zero feature in SageMaker Inference
Today at AWS re:Invent 2024, we are excited to announce a new feature for amazon SageMaker inference endpoints: the ability ...
Speed up your AI inference workloads with new NVIDIA-powered capabilities in Amazon SageMaker
This post is co-written with Abhishek Sawarkar, Eliuth Triana, Jiahong Liu and Kshitiz Gupta from NVIDIA. At re:Invent 2024, we ...
Easily deploy and manage hundreds of LoRA adapters with SageMaker efficient multi-adapter inference
The new efficient multi-adapter inference feature of amazon SageMaker unlocks exciting possibilities for customers using fine-tuned models. This capability integrates ...
Neural Magic Releases 2:4 Sparse Llama 3.1 8B: Smaller Models for Efficient GPU Inference
The rapid growth in the size of ai models has brought with it significant computational and environmental challenges. Deep learning ...
Amazon SageMaker Inference now supports G6e instances
As demand for generative ai continues to grow, developers and enterprises are looking for more flexible, cost-effective, and powerful accelerators ...
Towards Low-Bit Communication for Parallel Tensor LLM Inference
This paper was accepted into the Efficient Natural Speech and Language Processing (ENLSP) Workshop at NeurIPS 2024. Tensor parallelism provides ...
This UC Berkeley AI article presents Pie: a machine learning framework for transparent performance sharing and adaptive expansion in LLM inference
The use of large language models (LLM) has revolutionized artificial intelligence applications, enabling advances in natural language processing tasks such ...
Recommended.
