Fabiana Clemente demonstrates a pipeline that profiles real data, generates Great Expectations rules, trains a GAN, and validates the resulting synthetic data.
2
Synthetic data generation needs separate environments for profiling tools and the generator because their NumPy and pandas requirements can conflict.
3
The right evaluation depends on the intended use of the synthetic data, since privacy, fidelity, and utility involve trade-offs.
Summary
Fabiana Clemente presents a hands-on workflow for generating and checking synthetic tabular data. She combines YData Synthetic for GAN-based generation, pandas-profiling for exploratory analysis, and Great Expectations for automated validation. The example uses a cardiovascular disease dataset from Kaggle. Profiling identifies column types, warnings, distributions, and duplicate rows, then produces a JSON profile that feeds later steps. Fabiana explains why categorical and numerical columns need different preparation, and why conditional GANs can help with imbalanced variables and mode collapse. She also discusses training choices such as batch size, epochs, and learning rate. The generated data is compared with the original through profile reports and an expectation suite. A five-epoch demonstration produces visibly poor results, including invalid values, while the expectation report records 85.29 percent success. Fabiana is clear that evaluation cannot rely on one universal score. The useful metrics depend on whether the data is intended for machine learning, privacy, or another downstream task.
The workflow joins profiling, generation, and validation
Fabiana builds the workflow from three open source components. YData Synthetic generates records with generative architectures, especially GANs. pandas-profiling provides a one-line exploratory analysis with summaries, histograms, distributions, and warnings. Great Expectations turns data rules into checks that can be applied to synthetic output. The point is to carry information from the original dataset through the whole process. Profiling creates a reusable JSON file, the generator uses inferred column information, and the expectation suite tests whether the generated records remain close enough to the source. This gives the process a repeatable structure instead of relying only on visual inspection.
The profiling and generation steps should run in separate virtual environments because the packages depend on different versions of NumPy and pandas. Fabiana uses one Conda environment with Python 3.8 for pandas-profiling and Great Expectations, and another environment containing YData Synthetic. She keeps separate notebook kernels for the profiling and synthetic-generation work. The notebooks and setup files are available in YData's open source GitHub repository. She also recommends a GPU for training the synthetic-data model and warns that TensorFlow needs the correct CUDA drivers to identify it. CPU execution remains possible, although it takes much longer.
Profiling exposes problems that affect synthetic data
The cardiovascular disease example begins with a pandas DataFrame and a pandas-profiling report. The report summarizes minimums, maximums, histograms, and other column behaviour. Fabiana's preferred section is the warnings section, which identifies duplicate rows in the original dataset. That finding matters because source-data problems can affect the synthetic output and its later assessment. The profile can also be exported to JSON, which allows an automated pipeline to reuse inferred values and warnings without opening an HTML report. She treats profiling as something to repeat throughout the workflow, rather than as a one-time exploratory step.
An initial profile can generate an expectation suite
Fabiana uses the profile to create an automated Great Expectations suite. She selects univariate information such as histograms, maximums, medians, and standard deviations, while leaving out multivariate interactions and correlations for this example. After creating a Great Expectations project context, she saves the suite and its profile information. The generated rules apply per column and can later test the synthetic dataset. This avoids manually writing every basic constraint from the source data. The result is a set of checks for fields, ranges, and other values that should remain plausible when records are generated from the original dataset.
Column types and conditional variables shape generation
YData Synthetic uses the profile's inferred variable types to separate numerical and categorical columns. Fabiana says this matters because the two types need different encodings and preparation. The selected architecture is a conditional GAN, with the target variable, cardio, used as the condition. Conditional generation is especially useful for imbalanced data because an ordinary GAN can suffer mode collapse and produce too many records from the majority class. The cardiovascular dataset is relatively well balanced, so the condition is less necessary here. Fabiana also notes that dates, missingness, and similar behaviours may need preprocessing before they are passed to the generator, while outliers should generally be retained.
GAN training has a difficult stability and tuning problem
Fabiana explains the generator and discriminator as competing networks. The generator turns a noise vector into fake records, while the discriminator classifies records as real or fake. Training improves both sides, but the system can become difficult to optimize as the architecture grows. Her example uses a Wasserstein conditional GAN with gradient penalty to make training more stable. She focuses on batch size, epochs, and learning rate. A batch that is too large can let the generator overfit and produce data that is too close to the original. Too few epochs leave the model undertrained, while too many can damage learned patterns. The useful settings depend on dataset size and column count.
Fabiana rejects the idea that synthetic data can be tuned with one generally correct metric. If the data will replace real data for machine learning, one option is to compare models trained on real and synthetic data through a train-synthetic-test-real setup. Image work may use a fidelity score such as FID, while privacy-focused applications need different measures. The intended downstream use determines what counts as a good result. She recommends treating generation as a pipeline that supports repeated experiments, tests, and versioned validations. Privacy, fidelity, and utility also trade off against one another, so improving one can reduce another.
Fabiana compares real and generated profile reports, then runs the saved expectation suite against the synthetic CSV. The demonstration model was trained for only five epochs, so it has not converged. The report shows 85.29 percent success, with required fields present and no new or missing values, but some constraints fail. For example, the systolic-pressure field has values below the stated minimum of 60 and outside the allowed range ending at 240. These checks quickly expose records that look synthetic or violate business rules. She also gives a country-and-city example: a valid country list does not guarantee that a generated city belongs to the selected country, so hierarchical relationships need explicit expectations.
"It really depends a lot and that's why the optimization or the design of the optimization ends up not being just a single metric usually."Fabiana Clemente43:54
Who should watch
You are building a tabular synthetic-data pipeline and want a concrete sequence from profiling through validation.
Your generator produces plausible-looking records, but you need automated checks for ranges, missingness, and business relationships.
You need to choose evaluation metrics for synthetic data based on machine-learning use, privacy, or fidelity rather than one generic score.
