This is a customer post written jointly by ICL and AWS employees.
ICL is a multinational manufacturing and mining corporation based in Israel that manufactures unique mineral-based products and meets the essential needs of humanity, primarily in three markets: agriculture, food, and engineered materials. Their mining sites use industrial equipment that must be monitored because machinery failures can result in lost revenue or even environmental damage. Due to the extremely harsh conditions (high and low temperatures, vibrations, salt water, dust), it is difficult to connect sensors to these mining machines for remote monitoring. Therefore, most machines are monitored manually or visually continuously by workers on site. These workers frequently review camera footage to monitor the status of a machine. Although this approach has worked in the past, it does not scale and incurs relatively high costs.
To overcome this business challenge, ICL decided to develop internal capabilities to use machine learning (ML) for computer vision (CV) to automatically monitor its mining machines. As a traditional mining company, the availability of internal resources with data science, CV or ML skills was limited.
In this post, we discuss the following:
- How ICL developed internal capabilities to build and maintain CV solutions that enable automatic monitoring of mining equipment to improve efficiency and reduce waste
- A deep dive into a mining filter solution developed with the support of the AWS Prototyping program
Using the approach outlined in this post, ICL was able to develop a framework on AWS using Amazon SageMaker to create other use cases based on vision extracted from approximately 30 cameras, with the potential to scale to thousands of such cameras at its sites. production.
Building internal capabilities through AWS Prototyping
Building and maintaining machine learning solutions for business-critical workloads requires sufficiently trained personnel. It is often not possible to outsource such activities because it is necessary to combine internal knowledge about business processes with the creation of technical solutions. Therefore, ICL approached AWS for support in their journey to create a CV solution to monitor their mining equipment and acquire the necessary skills.
AWS Prototyping is an investment program in which AWS brings specialists to customer development teams to create mission-critical use cases. During this engagement, the customer's development team learns the underlying AWS technologies while building the use case over the course of 3 to 6 weeks and receives hands-on help. In addition to the corresponding use case, all the client needs is 3 to 7 developers who can spend more than 80% of their work time creating the aforementioned use case. During this time, AWS specialists are fully assigned to the customer's team and collaborate with them remotely or on-site.
ICL Computer Vision Use Case
For the prototyping engagement, ICL selected the use case to monitor their mining filters. A screen is a large industrial mining machine where minerals dissolved in water are processed. Water flows in several lanes from the top of the machine to the bottom. The flow is controlled individually for each lane. When the flow leaves the lane it is called overflow, which indicates that the machine is overloaded. The overflow inflows are minerals that the filter does not process and are lost. This must be avoided by regulating the influx. Without a machine learning solution, overflow must be monitored by humans and potentially takes time to be observed and managed.
The following images show the inputs and outputs of the CV models. The raw image from the camera (left) is processed using a semantic segmentation model (middle) to detect the different lanes. The model (right) then estimates coverage (white) and overflow (red).
Although the prototyping engagement focused on a single type of machine, the general approach of using cameras and automatically processing their images while using CV is applicable to a broader range of mining equipment. This allows ICL to extrapolate the insights gained during prototyping to other locations, camera types and machines, and also maintain the ML models without requiring any third-party support.
During the engagement, AWS specialists and ICL development team would meet every day and jointly develop the solution step by step. ICL data scientists would work independently on assigned tasks or receive hands-on pair programming support from AWS ML specialists. This approach ensures that ICL data scientists not only gain experience in systematically developing machine learning models using SageMaker, but also in integrating these models into applications and automating the entire lifecycle of such models, including automated retraining or monitoring. of models. After 4 weeks of this collaboration, ICL was able to move this model to production without needing additional support in 8 weeks, and has since built models for other use cases. The technical approach to this commitment is described in the next section.
Monitoring mining screens using CV models with SageMaker
SageMaker is a fully managed platform that addresses the entire lifecycle of an ML model: providing services and features that support teams working on ML models, from labeling their data in Amazon SageMaker Ground Truth to training and model optimization as well as hosting ML models. for production use. Prior to the compromise, ICL installed the cameras and obtained images as shown in the images above (leftmost image) and stored them in an Amazon Simple Storage Service (Amazon S3) bucket. Before you can train models, you need to generate training data. The joint ICL-AWS team addressed this in three steps:
- Label the data using a semantic segmentation labeling job in SageMaker Ground Truth, as shown in the following image.
- Preprocess the labeled images using image augmentation techniques to increase the number of data samples.
- Divide the labeled images into training, testing, and validation sets, so that the performance and accuracy of the model can be properly measured during the training process.
To achieve production scale for ML workloads, automating these steps is crucial to maintaining training quality. Therefore, whenever new images are tagged with SageMaker Ground Truth, the preprocessing and splitting steps are automatically executed and the resulting datasets are stored in Amazon S3, as shown in the model training workflow in the following diagram. Similarly, the model deployment workflow uses SageMaker resources to update endpoints automatically whenever an updated model is available.
ICL uses several approaches to deploy ML models in production. Some involve their current ai platform called KNIME, which allows them to quickly deploy models developed in the development environment into production by industrializing them into products. Various combinations of using KNIME and AWS services were analyzed; The previous architecture was the most suitable for the ICL environment.
SageMaker's built-in semantic segmentation algorithm is used to train models for screening grid area segmentation. By choosing this built-in algorithm instead of a self-built container, ICL does not have to deal with the undifferentiated heavy lifting of maintaining a convolutional neural network (CNN) and at the same time can use such a CNN for its use case. After experimenting with different configurations and parameters, ICL used a fully convolutional network (FCN) algorithm with a pyramid scene analysis network (PSPNet) to train the model. This allowed ICL to complete model construction within a week of the prototyping commitment.
Once a model has been trained, it must be deployed so that it can be used for evaluator monitoring. Based on model training, this process is fully automated and orchestrated using AWS Step Functions and AWS Lambda. Once the model is successfully deployed to the SageMaker endpoint, the incoming images from the cameras are resized to fit the model's input format and then fed to the endpoint for predictions using Lambda functions. The prediction result of semantic segmentation and overflow detection are stored in Amazon DynamoDB and Amazon S3 for further analysis. If an overflow is detected, Amazon Simple Notification Service (Amazon SNS) or Lambda functions can be used to automatically mitigate the overflow and monitor the corresponding lanes in the affected filter. The following diagram illustrates this architecture.
Conclusion
This post described how ICL, an Israeli mining company, developed its own computer vision approach for automated monitoring of mining equipment using cameras. We first show how to approach such a challenge from an enablement-focused organizational point of view, then provide a detailed look at how the model was built using AWS. Although the monitoring challenge may be unique to ICL, the general approach of building a prototype together with AWS specialists can be applied to similar challenges, particularly for organizations that do not have the necessary knowledge of AWS.
If you would like to learn how to create a production-scale prototype of your use case, contact your AWS account team to discuss a prototyping engagement.
About the authors
Markus Bestehorn He leads engineering and prototyping teams for customers in Germany, Austria, Switzerland, and Israel for AWS. He has a PhD in computer science and specializes in building complex machine learning and IoT solutions.
David Abekasis He leads the data science team at ICL Group with a passion for educating others on data analytics and machine learning while helping solve business challenges. She has a master's degree in data science and an MBA. She was fortunate to investigate spatial and time series data in the field of precision agriculture.
Ion Cleopas is a senior machine learning prototyping architect with a master's degree in data science and big data. He helps AWS customers create innovative ai/ML solutions by enabling their technical teams on AWS technologies by co-developing prototypes for challenging machine learning use cases, paving their path to production.
Miron Perel is a senior director of machine learning business development at Amazon Web Services. Miron advises generative ai companies in building their next generation models.
