Taipy or how to remove major obstacles in your data and artificial intelligence projects

Over the years, I have been involved in the implementation of many “intelligent software” projects that have demonstrated great benefits to major organizations. At the center of these different software projects were algorithms based on mathematical programming, simulation and heuristics, as well as ai models based on ML and generative ai. Most of these projects generated a substantial return on investment for these organizations; some have even shaped the future of their company.

Despite all the hype around ai and data, many organizations (outside the software industry) struggle to implement a successful ai strategy. Most of the CIOs/CDOs involved have produced mostly “standard” data projects (data lakes/warehouses/data management/Dashboarding), some have implemented multiple ai pilots, and very few have produced deployed projects that show ROI. substantial for your company.

One could consider the distribution of companies in terms of ai penetration as a heavily left-skewed fat-tailed distribution.

The purpose of this article is not to list all the obstacles that prevent greater penetration of ai projects within companies. To do this, I would recommend these two enlightening articles:

Why companies fail at machine learning

ai-can-help-leaders-make-better-decisions-under-pressure”>How ai can help leaders make better decisions under pressure

Instead, we focus on two major gaps in the current approach to software implementation.

Huge hole 1: a very isolated environment

It is interesting to visualize the different groups involved in a typical ai project.

Of course, there are valid reasons to have these different roles, let alone the need for specialization. However, it is worth noting that:

• In a real project, the gap between data scientists and end users is substantial.
• Each silo uses different technology stacks. It is not uncommon for data scientists to develop primarily in Python, while IT developers use JavaScript, Java, Scala, etc.
• There has never been a wider variety of programming skills between and within each silo.

Gaping Hole 2: Get buy-in from end users/business users

As highlighted in a ai-picture” target=”_blank” rel=”noreferrer noopener”>Previous article, end users seem to have disappeared from the ai ​​landscape. It is about data, technologies, algorithms, testing, implementation, etc. As if all ai projects necessarily replace fully human experts. I am convinced that the future of ai in industry lies in hybrid collaboration between business users and ai software.

However, end users are an integral part of ai software development. Not fully involving them during the development process puts you at risk of your software not being used when the system goes live.

Our strategy is to ensure that these two steps are implemented:

1. A fluid interaction of the end user with the algorithms.
2. And easy tracking of business user satisfaction

How to fill gap 1?

Some obvious directions are:

1. Standardize as much as possible in a single programming language.
2. Provide an easy-to-learn and easy-to-use programming experience to cater to all programming levels.

Python is the ideal candidate for this. It is at the heart of the ai ​​stack and is ideal for integrating with other environments.

There are many Python libraries available and they provide an easy learning curve (including reduced code); Unfortunately, they often suffer from performance issues and lack of customization.

Consider, for example, the development of graphical interfaces: one has the option of using full-code libraries like Plotly Dash (or even Java Script development) or easy-to-develop libraries like Streamlit or Gradio. However, these libraries do not scale in terms of performance and will place you in a strict framework that prohibits most customization.

A Python developer shouldn't have to arbitrate so much between programming productivity and performance/customization.

We spend a lot of time designing/implementing our product, Guy, to go a step further by ensuring ease of development while providing a huge leap in performance and customization. Here are two examples of performance issues (among many others) resolved with Taipy:

How to fill gap 2?

It is crucial to address the two salient points mentioned above:

1. Seamless end-user interaction with back-end algorithms
2. And easy tracking of business user satisfaction.

Address point 1: the end user needs to interact with the algorithm/back-end.

For this it is essential:

• Provide variables/parameters that the end user can control via the GUI.
• Allow the end user to run backend algorithms using these different parameter values, which will generate different results.
• Provide the ability to compare these different executions and track KPI performance over time.

At Taipy we have introduced the 'script' concept that addresses all of the above requirements.

One scenario consists of the execution of the algorithm/pipeline where Taipy stores all data elements (data sources, data outputs)

Taipy scenario registration allows the end user to:

• keep track of all your executions,
• review a past scenario, understand its results, scan its input data, etc.

Address point 2: easy tracking of business user satisfaction

Another great benefit of Taipy's Scenario feature is that it bridges the gap between the end user and data scientists. Taipy's scenario log is a goldmine for data scientists as they can access all end-user runs. Additionally, the end user can tag any of these scenarios and share them with data scientists for examination.

This feature of the scenario can dramatically increase end-user acceptance of the software. Unfortunately, in practice, testing of ai algorithms is usually limited to a few test cases and the use of drift detection. More is needed to ensure high software acceptance. And Taipy's scenarios will help a lot here.

Below are some examples of Taipy ai applications that allow the business user to explore pre-generated scenarios.

Conclusion

To conclude with, Guy has proven to be instrumental in the success of ai projects for leading corporations, offering an efficient and easy-to-use Python framework. With the launch of Taipy Designer, we continue to democratize ai development, focusing on accessibility for data analysts and ensuring the seamless integration of ai into business processes.

This article was originally published in ai-data-project-challenges-with-taipy”>Guy.

Thanks to taipy team for the educational/thought leadership article. taipy team has supported us in this content/article.

<!– ai CONTENT END 2 –>