Large language models (LLMs) with billions of parameters have dramatically transformed ai applications. However, its demanding computation during inference has posed significant challenges for implementation on resource-constrained devices. Despite recent trends favoring alternative activation functions such as GELU or SiLU, known for higher computation, this study strongly advocates reestablishing ReLU activation in LLMs. We show that using the ReLU activation function has a negligible impact on convergence and performance, while significantly reducing computation and weight transfer. This reduction is particularly valuable during the memory-bound inference step, where efficiency is paramount. By exploring sparsity patterns in ReLU-based LLMs, we reveal the reuse of activated neurons to generate new tokens and, leveraging these insights, we propose practical strategies to substantially reduce the LLM inference computation by up to three times, using activations of ReLU with minimal performance tradeoffs.
