How BQA Optimizes Education Quality Reporting Using Amazon Bedrock

Given the value of data today, organizations across various industries are working with large amounts of data in multiple formats. Manually reviewing and processing this information can be a challenging and time-consuming task, with room for potential errors. This is where intelligent document processing (IDP), coupled with the power of generative ai, emerges as a revolutionary solution.

Improving IDP's capabilities is the integration of generative ai, which leverages large language models (LLMs) and generative techniques to understand and generate human-like text. This integration allows organizations to not only extract data from documents, but also interpret, summarize, and generate insights from the extracted information, enabling more intelligent and automated document processing workflows.

He Education and Training Quality Authority (BQA) plays a vital role in improving the quality of education and training services in the Kingdom of Bahrain. BQA reviews the performance of all education and training institutions, including schools, universities and vocational institutes, thereby promoting the professional advancement of the nation's human capital.

BQA oversees a comprehensive quality assurance process, which includes establishing performance standards and conducting objective reviews of education and training institutions. The process involves the collection and analysis of extensive documentation, including self-assessment reports (SER), supporting evidence, and various media formats from the institutions being reviewed.

The collaboration between BQA and AWS was facilitated through the Cloud Innovation Center (CIC) program, a joint initiative of AWS, Tamkeenand leading universities in Bahrain, including Bahrain Polytechnic and University of Bahrain. The CIC program aims to foster innovation within the public sector by providing a collaborative environment where government entities can work closely with AWS consultants and university students to develop cutting-edge solutions using the latest cloud technologies.

As part of the CIC program, BQA has created a proof-of-concept solution, leveraging the power of AWS services and generative ai capabilities. The main objective of this proof of concept was to test and validate the proposed technologies, demonstrating their feasibility and potential to optimize BQA data management and reporting processes.

In this post, we explore how BQA used the power of amazon Bedrock, amazon SageMaker JumpStart, and other AWS services to optimize the overall reporting workflow.

The challenge: streamlining self-assessment reports

BQA has traditionally provided education and training institutions with a template for the SER as part of the review process. Institutions must submit a review portfolio containing the complete SER and supporting material as evidence, which at times did not fully adhere to established reporting standards.

The existing process had some challenges:

Inaccurate or incomplete submissions – Institutions may provide incomplete or inaccurate information in submitted reports and supporting evidence, resulting in gaps in the data needed for a comprehensive review.
Missing or insufficient supporting evidence – Supporting material provided as evidence by institutions often did not substantiate claims made in their reports, calling the evaluation process into question.
Requires a lot of time and resources – The process required spending a lot of time and resources to manually review submissions and follow up with institutions to request additional information if necessary to rectify the submissions, which slowed down the overall review process.

These challenges highlighted the need for a more agile and efficient approach to the submission and review process.

Solution Overview

The proposed solution uses amazon Bedrock and the amazon Titan Express model to enable IDP functionalities. The architecture seamlessly integrates multiple AWS services with amazon Bedrock, enabling efficient data extraction and comparison.

amazon Bedrock is a fully managed service that provides access to high-performance base models (FMs) from leading ai startups and amazon through a unified API. It offers a wide range of FMs, allowing you to choose the model that best suits your specific use case.

The following diagram illustrates the architecture of the solution.

The solution consists of the following steps:

The relevant documents are uploaded and stored in an amazon Simple Storage Service (amazon S3) bucket.
An event notification is sent to an amazon Simple Queue Service (amazon SQS) queue to line up each file for further processing. amazon SQS acts as a buffer, allowing different components to reliably send and receive messages without being directly coupled, improving the scalability and fault tolerance of the system.
The SQS queue invokes the AWS Lambda text extraction function, processes each queued file, and uses amazon Textract to extract text from the documents.
The extracted text data is placed in another SQS queue for the next processing step.
This new queue containing the extracted text invokes the text digest Lambda function. This function sends a request to SageMaker JumpStart, where a Meta Llama text generation model is implemented to summarize the content based on the provided message.
In parallel, the InvokeSageMaker Lambda function is invoked to perform comparisons and evaluations. Compares the extracted text to the BQA standards the model was trained on, evaluating text compliance, quality, and other relevant metrics.
The summary data and evaluation results are stored in an amazon DynamoDB table.
On demand, the InvokeBedrock Lambda function invokes amazon Bedrock to generate generative ai summaries and comments. The feature constructs a detailed message designed to guide the amazon Titan Express model in evaluating the university's presentation.

Rapid engineering with amazon Bedrock

To harness the power of amazon Bedrock and ensure that the generated output met the desired structure and formatting requirements, a carefully crafted message was developed according to the following guidelines:

Presentation of evidence – Present the evidence presented by the institution under the relevant indicator, providing the model with the necessary context for the evaluation.
Evaluation criteria – Describe the specific criteria against which the evidence should be evaluated.
Evaluation instructions – Instruct the model as follows:
- Indicate N/A if the evidence is irrelevant to the indicator.
- Evaluate the university's self-assessment based on the criteria
- Assign a score from 1 to 5 for each comment, citing evidence directly from the content.
Response format – Specify your answer in bulleted form, focusing on relevant analysis and evidence, with a limit of 100 words.

To use this request template, you can create a custom Lambda function with your project. The function should handle the retrieval of required data, such as the indicator name, the evidence presented by the university, and the rubric criteria. Inside the function, include the request template and dynamically fill in the placeholders (${indicatorName}, ${JSON.stringify(allContent)}and ${JSON.stringify(c.comment)}) with the recovered data.

The amazon Titan Text Express model will then generate the evaluation response according to the instructions provided, adhering to the specified format and guidelines. You can process and analyze the model response within your function, extracting the compliance score, relevant analytics, and evidence.

The following is an example message template:

for (const c of comments) {
        const prompt = `
        Below is the evidence submitted by the university under the indicator "${indicatorName}":
        ${JSON.stringify(allContent)}
    
         Analyze and Evaluate the university's eviedence based on the provided rubric criteria:
        ${JSON.stringify(c.comment)}

        - If the evidence does not relate to the indicator, indicate that it is not applicable (N/A) without any additional commentary.
        
       Choose one from the below compliance score based on evidence submitted:
       1. Non-compliant: The comment does not meet the criteria or standards.
        2.Compliant with recommendation: The comment meets the criteria but includes a suggestion or recommendation for improvement.
        3. Compliant: The comment meets the criteria or standards.

        THE END OF THE RESPONSE THERE SHOULD BE EITHER SCORE: (SCORE: COMPLIANT OR NON-COMPLIANT OR COMPLIANT WITH RECOMMENDATION)
        Write your response in concise bullet points, focusing strictly on relevant analysis and evidence.
        **LIMIT YOUR RESPONSE TO 100 WORDS ONLY.**

        `;

        logger.info(`Prompt for comment ${c.commentId}: ${prompt}`);

        const body = JSON.stringify({
          inputText: prompt,
          textGenerationConfig: {
            maxTokenCount: 4096,
            stopSequences: (),
            temperature: 0,
            topP: 0.1,
          },
        });

The following screenshot shows an example of the response generated by amazon Bedrock.

Results

Implementing amazon Bedrock enabled institutions to realize transformative benefits. By automating and streamlining the collection and analysis of extensive documentation, including SERs, supporting evidence, and various media formats, institutions can achieve greater accuracy and consistency in their reporting processes and preparation for the review process. . This not only reduces the time and cost associated with manual data processing, but also improves compliance with quality expectations, thereby improving the credibility and quality of your institutions.

For BQA, the implementation helped achieve one of its strategic objectives focused on optimizing its reporting processes and achieving significant improvements in a variety of critical metrics, substantially improving the overall efficiency and effectiveness of its operations.

Key expected success metrics include:

Faster response times to generate self-assessment reports that are 70% accurate and compliant, leading to greater overall efficiency.
Reduction of the risk of errors or non-compliance in the reporting process, enforcing established guidelines.
Ability to summarize lengthy submissions into concise bullet points, allowing BQA reviewers to quickly analyze and understand the most pertinent information, reducing evidence analysis time by 30%.
More accurate compliance feedback functionality, allowing reviewers to effectively evaluate submissions against established standards and guidelines, while achieving a 30% reduction in operational costs through process optimizations.
Improved transparency and communication through seamless interactions, allowing users to request additional documents or clarifications with ease.
Real-time feedback, allowing institutions to make necessary adjustments promptly. This is particularly useful for maintaining the accuracy and integrity of the shipment.
Improved decision making by providing insights into data. This helps universities identify areas for improvement and make data-driven decisions to improve their processes and operations.

The following screenshot shows an example of generating new reviews with amazon Bedrock.

Conclusion

This post outlined the implementation of amazon Bedrock at the Education and Training Quality Authority (BQA), demonstrating the transformative potential of generative ai to revolutionize quality assurance processes in the education and training sectors. For those interested in exploring the technical details further, the complete code for this implementation is available in the following GitHub repository. If you are interested in running a similar proof of concept with us, please submit your challenge idea to Bahrain Polytechnic either University of Bahrain CIC website.

About the author

Maram Al Saegh is a Cloud Infrastructure Architect at amazon Web Services (AWS), where she helps AWS customers accelerate their journey to the cloud. Currently, he is focused on developing innovative solutions that leverage generative ai and machine learning (ML) for public sector entities.