How to Assess the Quality of Gen AI Output?

Is there a single metric better than any other one, to evaluate the quality of Gen AI output? Intuitively, it seems that the answer is a resounding no. However, in the context of tabular data synthetization, such a metric actually exists and has been known for a long time. It was never implemented in moderate or high dimensions due to the complexity of the problem. Thus it remained mostly of topic for academic research and theoretical analysis.

Not anymore. Not only I tested it successfully on many use cases (telecom, insurance, healthcare, cybersecurity, education), but it is now available as a Python library (open source). It has also been tested by several of the participants in my Gen AI certification program, and the Python library called genAI-evalution was actually turned into production code by one of them.

Several illustrations are available on my GitHub repository (the Deep Resampling files), but the first use with detailed description can be traced to the following project: Generative AI Technology Break-through: Spectacular Performance of New Synthesizer.

Many evaluation metrics poorly capture the subtle interdependencies among features in your dataset, resulting in numerous false negatives: some output rated as very good, when it is indeed very bad. The purpose of my new distance is to completely fix this issue, once and for all.

To download the code, check out technical paper #29, here . Happy to share the link to the Python library with anyone interested. Everything is free, and a lot better than what you can find on the market with a high price tag.