Capital markets trading teams face numerous challenges throughout the post-trade lifecycle, including delays in trade settlements, accounting errors and inaccurate regulatory reporting. For derivatives trading, it is even more challenging. Timely settlement of derivatives transactions is an onerous task. This is because transactions involve different counterparties and there is a high degree of variation between documents containing trading terms (such as trade date, value date and counterparties). We commonly see the application of OCR screen removal solutions in capital market organizations. These applications have the disadvantage of being inflexible and high maintenance.
artificial intelligence and machine learning (ai/ML) technologies can help capital markets organizations overcome these challenges. Intelligent document processing (IDP) applies ai/ML techniques to automate data extraction from documents. Using IDP can reduce or eliminate the need for time-consuming human reviews. IDP has the power to transform the way capital market back-office operations work. It has the potential to increase employee efficiency, improve cash flow by accelerating business deals, and minimize operational and regulatory risks.
In this post, we show how you can automate and intelligently process derived commits at scale using AWS ai services. The solution combines amazon Textract, a fully managed machine learning service to effortlessly extract text, handwriting and data from scanned documents, and AWS Serverless technologies, a set of fully managed event-driven services to execute code, manage data and integrate applications, all without managing servers.
Solution Overview
The life cycle of a derivatives transaction involves multiple phases, from transaction investigation to execution, clearing and settlement. The solution presented in this post focuses on the trade clearing and settlement phase of the derivatives trading lifecycle. During this phase, the transaction counterparties and their agents determine and verify the exact commercial terms of the transaction and prepare for settlement.
The following figure shows a derived example confirming the document.
We build the solution using event-driven principles as shown in the diagram below. Derived confirmation documents received from customers are stored in amazon Simple Storage Service (amazon S3). An event notification about S3 object upload completion places a message into an amazon Simple Queue Service (amazon SQS) queue to invoke an AWS Lambda function. The function invokes the amazon Textract API and performs a fuzzy match using document schema mappings stored in amazon DynamoDB. A web-based user interface has been created to review the document processing process and update schemas to train services for new formats. The web UI uses amazon Cognito for authentication and access control.
The process flow includes the following steps:
- The user or business application uploads an image or PDF to the designated S3 bucket.
- An event notification about S3 object upload completion places a message into an SQS queue.
- An event upon receiving a message invokes a Lambda function which in turn invokes amazon Textract
StartDocumentAnalysis
API for information extraction.- This call initiates an asynchronous analysis of the document to detect elements within the document, such as key-value pairs, tables, and forms.
- The call also returns the ID of the asynchronous job and saves the job ID and the amazon S3 document key to a DynamoDB table.
- Upon completion of the job, amazon Textract sends a message to an amazon Simple Notification Service (amazon SNS) topic and places the resulting JSON in the designated S3 bucket for classification analysis.
- A Lambda function receives the payload from amazon SQS and performs a fuzzy match using Sorenson-Dice analysis between amazon Textract JSON results and DynamoDB document configuration mappings. The Sorenson-Dice analysis step compares the two texts and calculates a number between 0 and 1, with the former indicating no match and the latter indicating an exact match.
- Once the analysis is complete, a Lambda function writes a merged and cleaned JSON result to the original S3 bucket and inserts the analysis results back into the DynamoDB table.
- amazon API Gateway endpoints make it easy to interact with the web-based user interface.
- The human-in-the-loop UI application provides a human-in-the-loop function to analyze the document processing process and intervene as necessary to update document configuration mappings.
A human-present process was applied to visually compare the reconciled results with their locations in the input documents. End users can verify the accuracy of the results and accept or reject the findings. When new counterparts and formats are introduced, ML learning helps users create new schema mappings in the human user interface for further processing.
What is the human being in the circuit?
A human-in-the-loop process combines supervised machine learning with human involvement in training and testing an algorithm. This practice of bringing human and machine intelligence together creates an iterative feedback loop that allows the algorithm to produce better results.
You can apply human technology in all types of deep learning ai projects, including natural language processing (NLP), computer vision, and transcription. Additionally, you can use the human function in conjunction with ai content moderation systems to quickly and efficiently analyze user-generated content. We call this human-in-the-loop decision making, where ai flags content and human moderators review what has been flagged.
The harmonious relationship between people and ai has several benefits, including:
- Accuracy – In the context of document processing, there are limitations to the amount of analysis that can be automated. ai can miss content that should be flagged (a false positive) and can also incorrectly flag content that may be harmless (a false negative). Humans are essential in the content moderation process because they can interpret things like context and multilingual text.
- Increased efficiency – artificial intelligence can save a lot of time and costs by sifting through and trimming large amounts of data. The task can then be passed to humans to complete a final classification. Although you cannot automate the entire process, you can automate an important part, saving time.
Looking to the future: the art of the possible
amazon Textract is an AWS service that uses ML to automatically extract text, handwriting, and data from any document.
amazon Textract can extract information from a wide variety of documents, including scanned paper records, forms, IDs, invoices, reports, certificates, legal documents, letters, bank statements, charts, handwritten notes, and more. Supported formats include common file types such as PNG, JPEG, PDF, and TIFF. For formats like Word or Excel, you can convert them to images before sending them to amazon Textract. Content is extracted in seconds and then indexed for search through an easy-to-use API.
The Queries feature within the amazon Textract Analyze Document API gives you the flexibility to specify the data you need to extract from documents. The queries extract information from a variety of documents, such as pay stubs, immunization cards, mortgage notes, and insurance cards. You don't need to know the document's data structure (table, form, nested data) or worry about variations between document versions and formats. The flexibility that Queries provides reduces the need to implement post-processing and the reliance on manual review of extracted data.
Conclusion
Automating derivatives confirmation increases the capacity of the trading team by saving processing time. In this post, we show common challenges in processing derivative commits and how you can use AWS Intelligent Document Processing Services to overcome them. Most administrative operations in capital markets involve document processing. The approach shown in this publication sets a pattern for many administrative document processing use cases, benefiting the capital markets industry by reducing costs and improving staff productivity.
We recommend a thorough review of Security on amazon Textract and strict adherence to the guidelines provided. For more information on solution pricing, review the pricing details for amazon Textract, Lambda, and amazon S3.
“Thanks to amazon Textract and Serverless services, we have been able to create an end-to-end digital workflow for derivatives processing. We expect straight-through processing rates to increase to over 90%, reducing operational risks and costs associated with manual interventions. “This automation provides the resilience and flexibility needed to adapt to evolving market structures, such as T+1 settlement times.”
– Stephen Kim, CIO, Head of Corporate technology, Jefferies
About the authors
Vipul Parekh, is a Senior Client Solutions Manager at AWS, guiding our capital markets clients to accelerate their cloud business transformation journey. He is a GenAI ambassador and a member of the AWS ai/ML technical field community. Prior to AWS, Vipul held various roles at leading investment banks, leading transformations spanning front office to back office and compliance areas.
waves of paradise, is a senior technical program manager at AWS. She comes to AWS with more than 30 years of experience in financial services, media & entertainment, and CPG.
Saby Sahoo, is a Senior Solutions Architect at AWS. Saby has more than 20 years of experience in the field of IT solutions design and implementation, data analytics and ai/ML/GenAI.
Sovik Kumar Nath is an ai/ML Solutions Architect with AWS. He has extensive experience designing end-to-end business analytics and machine learning solutions in finance, operations, marketing, healthcare, supply chain management, and IoT. Sovik has published papers and holds a patent on ML model monitoring. He holds a double master's degree from the University of South Florida, University of Freiburg, Switzerland, and a bachelor's degree from the Indian Institute of technology, Kharagpur. Outside of work, Sovik enjoys traveling, taking ferries and watching movies.