Role of LLMs like ChatGPT in scientific research: Integrating scalable AI and high-performance computing to address complex challenges and accelerate discovery in diverse fields

In the contemporary landscape of scientific research, the transformative potential of ai has become increasingly evident. This is particularly true when applying scalable ai systems to high-performance computing (HPC) platforms. This exploration of scalable ai for science underscores the need to integrate large-scale computational resources with vast data sets to address complex scientific challenges.

The success of ai models like ChatGPT highlights two main advances crucial to their effectiveness:

1. The development of transformer architecture.
2. The ability to train on large amounts of data at Internet scale.

These elements have laid the foundation for important scientific advances, as seen in initiatives such as black hole modeling, fluid dynamics, and protein structure prediction. For example, one study used artificial intelligence and large-scale computing to develop models of black hole mergers, taking advantage of a data set of 14 million waveforms on the Summit supercomputer.

A clear example of the impact of scalable ai is drug discovery, where transformer-based language models (LLMs) have revolutionized chemical space exploration. These models use extensive datasets and fine-tuning on specific tasks to autonomously learn and predict molecular structures, thereby accelerating the discovery process. LLMs can efficiently explore chemical space by employing tokenization and mask prediction techniques, integrating pre-trained models for molecules and protein sequences with fine-tuning on small labeled datasets to improve performance.

High-performance computing is indispensable for achieving these scientific advances. Different scientific problems require different levels of computational scale, and high-performance computing provides the infrastructure to handle these diverse requirements. This distinction sets ai for science (AI4S) apart from consumer-focused ai, which often deals with sparse, high-precision data from expensive experiments or simulations. Scientific ai requires handling specific features of scientific data, including incorporating known domain knowledge such as partial differential equations (PDEs). Physics-Informed Neural Networks (PINNs), Neural Ordinary Differential Equations (NODEs), and Universal Differential Equations (UDEs) are methodologies developed to meet these unique requirements.

Scaling ai systems involves both model- and data-driven parallelism. For example, training a large model like GPT-3 on a single NVIDIA V100 GPU would take ages, but using parallel scaling techniques can reduce this time to just over a month on thousands of GPUs. These scaling methods are essential not only for faster training but also for improving model performance. Parallel scaling has two main approaches: model-based parallelism, necessary when models exceed the memory capacity of the GPU, and data-driven parallelism, which arises from the large amount of data required for training.

Scientific ai differs from consumer ai in its accuracy and data handling requirements. While consumer applications may rely on 8-bit integer inferences, scientific models often need high-precision floating-point numbers and strict adherence to physics laws. This is particularly true for simulation surrogate models, where integrating machine learning with traditional physics-based approaches can yield more accurate and cost-effective results. Neural networks in physics-based applications may need to enforce boundary conditions or conservation laws, especially in surrogate models that replace parts of larger simulations.

A critical aspect of AI4S is adapting to the specific characteristics of scientific data. This includes handling physical constraints and incorporating known domain knowledge such as PDEs. Soft penalty constraints, neural operators, and symbolic regression are methods used in scientific machine learning. For example, PINNs incorporate the PDE residual norm into the loss function, ensuring that the model optimizer minimizes both the data loss and the PDE residual, leading to a satisfactory physical approximation.

Parallel scaling techniques are diverse, including data parallel and model parallel approaches. Data-parallel training involves splitting a large batch of data across multiple GPUs, each of which processes a portion of the data simultaneously. On the other hand, parallel model training distributes different parts of the model across multiple devices, which is particularly useful when the model size exceeds the memory capacity of a single GPU. Spatial decomposition can be applied in many scientific contexts where data samples are too large to fit on a single device.

The evolution of ai for science includes the development of hybrid ai-simulation workflows, such as cognitive simulations (CogSim) and digital twins. These workflows combine traditional simulations with ai models to improve the accuracy of predictions and decision-making processes. For example, in neutron scattering experiments, ai-driven methods can reduce the time required for experimental decision-making by providing real-time steering and analysis capabilities.

Several trends are shaping the landscape of scalable ai for science. The shift towards mixed-expert (MoE) models, which are sparsely connected and therefore more cost-effective than monolithic models, is gaining ground. These models can handle many parameters efficiently, making them suitable for complex scientific tasks. The concept of an ai-powered autonomous laboratory is another exciting development. With integrated research infrastructures (IRI) and foundation models, these labs can perform experiments and analysis in real time, accelerating scientific discoveries.

Limitations of transformer-based models, such as context length and computational expense, have renewed interest in linear recurrent neural networks (RNNs), which offer greater efficiency for long token lengths. Additionally, operator-based models for solving PDEs are becoming more prominent, allowing ai to simulate entire classes of problems rather than individual instances.

Finally, the interpretability and explainability of ai models must be considered. As scientists remain cautious about ai/ML methods, it is crucial to develop tools to elucidate the rationale behind ai predictions. Techniques such as Class Activation Mapping (CAM) and attention map visualization help provide insights into how ai models make decisions, fostering trust and broader adoption in the scientific community.

Sources