SynthEval: A Novel Open Source Machine Learning Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data

Computer vision, machine learning, and data analytics in many fields have seen an increase in the use of synthetic data in recent years. Synthetic means imitating complicated situations that would be difficult, if not impossible, to record in the real world. Information about individuals, such as patients, citizens or customers, along with their unique attributes, can be found in tabular records at the personal level. These logs are ideal for knowledge discovery tasks and the creation of advanced predictive models to assist in decision making and product development. However, the privacy implications of tabular information are substantial and should not be openly disclosed. Data protection rules are essential to safeguard people's rights against harmful designs, blackmail, fraud or discrimination if sensitive data is compromised. While they can slow down scientific development, they are necessary to prevent that damage.

In theory, synthetic data improves on conventional anonymization methods by allowing access to tabular data sets while protecting people's identities from prying eyes. In addition to strengthening, balancing, and reducing data bias, synthetic data can improve downstream models. Although we have achieved notable success with text and image data, it is still difficult to simulate tabular data, and the privacy and quality of synthetic data can differ greatly depending on the algorithms used to create it, the parameters used for optimization and evaluation. methodology. In particular, it is difficult to compare current models and, by extension, objectively evaluate the effectiveness of a new algorithm due to the lack of consensus on evaluation methodologies.

A new study by researchers at the University of Southern Denmark presents SynthEval, a novel evaluation framework in the Python package. Its purpose is to facilitate the easy and consistent evaluation of synthetic tabular data. Their motivation comes from the belief that the SynthEval framework can significantly influence the research community and provide a much-needed answer to the evaluation scenario. SynthEval incorporates a large collection of metrics that can be used to create user-specific benchmarks. With the push of a button, users can access predefined cue points in the presets, and the provided components make it easy to create their own unique setups. Adding custom metrics to benchmarks is very easy and there is no need to edit the source code.

The main feature of SynthEval is a robust shell for accessing a large library of measurements and condensing them into evaluation reports or reference configurations. The metrics object and the SynthEval interface object are the two main components that do this. The first specifies how the metrics modules are structured and how the SynthEval workflow can access them. The evaluation and benchmarking modules are primarily hosted in the SynthEval interface object, which is an object that can be interacted with. If non-numeric values ​​are not provided, the SynthEval utilities will determine them automatically. They take care of any data preprocessing that is necessary.

In theory, it only takes two lines of code to perform evaluation and benchmarking: create the SynthEval object and call any of the methods. The command line interface is another way SynthEval is available to you.

The team has provided multiple ways to get the metrics to make SynthEval as versatile as possible. There are now three presets available, or metrics can be manually selected from the library. Bulk selection is also an option. If you specify a file path as the default value, SynthEval will attempt to load the file. If users use any non-standard configuration, a new configuration file will be saved in JSON format for repeatability.

As an additional useful feature, the SynthEval benchmark module allows simultaneous evaluation of multiple synthetic representations of the same data set. The results are combined, evaluated internally, and then sent. Thanks to this, the user can easily and comprehensively evaluate various data sets using various metrics. Generative model abilities can be comprehensively evaluated with the use of datasets generated by frameworks such as SynthEval. When it comes to tabular data, one of the biggest hurdles is maintaining consistency when dealing with fluctuating percentages of numerical and categorical data. This issue has been addressed in previous assessment systems in various ways, for example by limiting the metrics that can be used or limiting the types of data that can be accepted. In contrast, SynthEval constructs equivalents of mixed correlation matrices, uses similarity functions instead of classical distances to account for heterogeneity, and uses the empirical approximation of p-values ​​to try to represent the complexities of real data.

The team employs the linear classification strategy and a custom evaluation setup in the SynthEval benchmark module. It seems that generative models find it difficult to compete with baselines. The “random sample” baseline in particular stands out as a formidable opponent, ranking near the top overall and with privacy and usefulness scores that are unmatched anywhere else in the benchmark. The findings make it clear that ensuring high utility does not automatically mean good privacy. When it comes to privacy, the most useful data sets (non-optimized BN and CART models) are also among the worst ranked, posing unacceptable identification risks.

Each of the metrics accessible in SynthEval takes into account the heterogeneity of the data set in its own way, which is a limitation in itself. Preprocessing has its limits and future metric integrations must take into account the fact that synthetic data is too heterogeneous to meet this. The researchers intend to incorporate additional metrics requested or provided by the community and aim to continue improving the performance of the various algorithms and framework that already exists.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter. Join our Telegram channel, Discord channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our SubReddit over 40,000 ml