A fine-tuned LLM reached an AUC of 91.6 on a telecommunications customer-churn dataset, close to XGBoost at 93.
2
The LLM needed tabular records translated into natural-language prompts and labels expressed as yes or no.
3
Adding customer IDs did not help, and removing XGBoost's most important feature reduced both models by roughly the same amount.
Summary
Sebastian Cattes tested whether a fine-tuned large language model could classify tabular data as well as XGBoost. He chose a recently published telecommunications customer-churn dataset so the examples would not already be in the model's training data. Each CSV row was translated into a natural-language sentence, then used to fine-tune a sequence-to-sequence model to predict yes or no. The LLM reached an AUC of 91.6, close to XGBoost's 93, although logistic regression performed better on this dataset. Untuned ChatGPT performed worse than random after 100 trials, so fine-tuning was necessary. Customer IDs added only noise, and removing the most important feature caused a similar drop for both models. Cattes sees a possible use for combining tabular and text data by translating the table into text and adding it to an existing text dataset. He also discusses the experiment's cost, prediction speed, and questions about combining churn datasets or using model explanations and hallucinated information to enrich data.
The test needed data that the language model had not already seen
Cattes avoided standard benchmark datasets because a language model might have encountered them during training. He wanted a dataset published after the model's training period, from a domain where the model could plausibly know something useful. Highly technical sensor data with hundreds of anonymous columns would not provide that opportunity. The experiment therefore used a recently uploaded customer-churn dataset from a telecommunications provider. This choice let him test whether general knowledge could add anything beyond the information contained in the table itself.
The experiment converted each table row into a natural-language classification example
The pipeline translated each CSV record into a sentence using a codebook rather than leaving it as a comma-separated row. Cattes used OpenAI APIs to create the translated prompts and fine-tune the model. Instead of predicting a numeric zero or one, the sequence-to-sequence model was trained to predict the words yes or no. A train-test split was then used to compare the fine-tuned model with XGBoost on the churn task.
The fine-tuned LLM came close to XGBoost on customer churn
Using AUC as the classification metric, XGBoost reached 93 while the other model reached 91.6. Cattes considered the LLM's result reasonable and the gap relatively small. He also tested ChatGPT without fine-tuning by giving it one translated customer record at a time and asking whether the customer would return. After 100 attempts, those predictions were worse than random, which led him to conclude that fine-tuning was needed. Logistic regression performed better than the LLM on this particular dataset.
The tests did not show that the LLM had hidden knowledge about individual customers
Cattes added the customer ID to each prompt to test whether the model might know something about the dataset or its records. The change had no effect on the results, and he described the ID as noise. He then removed the feature that XGBoost considered most important from both the XGBoost input and the LLM prompt. Performance fell by roughly the same amount for both models. The LLM did not compensate for the missing variable.
The result was stable across the train-test seeds he tried
Cattes changed the train-test split seed while changing only that part of the experiment. The results did not change materially. He described the findings on this single dataset as reasonable first-experiment results. This does not establish that the LLM is generally better, since the dataset was limited and logistic regression already outperformed it, but the comparison showed that the translated and fine-tuned approach could produce a competitive classification score.
Tabular-to-text conversion could help when a dataset also contains text
Cattes points out that mixed datasets are often handled by extracting features from text and adding those features to a conventional model. His alternative is to translate the tabular fields into a sentence, append that sentence to the other text, and use a fine-tuned language model for prediction. This approach adds preparation work compared with directly training XGBoost, but it may offer one way to combine structured and unstructured inputs in the same model.
The experiment was inexpensive but prediction speed was a concern
Cattes estimated that the whole experiment cost about 100 dollars, which was less than expected. The OpenAI API was slower than he expected during prediction: about five minutes for 1,000 predictions. He left open whether that speed was acceptable for a production use case or could be improved. His next experiments would include combining multiple churn datasets, testing whether Shapley values identify the parts of a prompt used for classification, and checking whether information from the LLM could enrich the original tabular data.
