Foundation models make it fast to prototype natural-language data tasks, but enterprise deployment exposes problems with customization, SQL quality, and privacy.
2
Number Station built the open-source NSQL model family around SQL-specific training, with model sizes that let teams choose between quality, speed, and local hosting needs.
3
NSQL used raw SQL pre-training followed by instruction tuning on about 300,000 curated instruction, schema, and SQL pairs, and its larger models outperformed available open-source alternatives on text-to-SQL evaluation.
Summary
Laurel Orr explains why natural-language interfaces to enterprise data are easy to prototype and difficult to deploy. A general model can turn a business question into SQL, but it does not know company-specific definitions, often makes mistakes in advanced SQL, and may not be allowed to send schemas or business questions to a third-party service. Number Station addressed these limits with NSQL, an open-source family of SQL-focused models. The training process first exposed models to raw SQL, then instruction-tuned them on about 300,000 curated examples containing instructions, schemas, and SQL outputs. Orr describes automated SQL parsing and schema checks, manual review of poor questions, model-assisted labeling, and query augmentation. NSQL includes several model sizes for different hardware and latency constraints. Orr reports that the larger models beat available open-source models on the Spider benchmark and that the Llama 2-based version matched GPT-4 on the reported evaluation. She also stresses that production systems need validation, auditing, and safeguards around generated queries.
Foundation models make data-task prototypes much faster to build
Orr describes foundation models as large neural networks trained with self-supervision, where the training data does not need manual labels. Their in-context learning lets one model handle many tasks through changes to the input text. For structured data, she gives examples such as cleaning data and formatting dates. This changes the cost of early experimentation. A team can prototype a task and build a demo in a day, whereas older approaches could require months of labeled-data work. Natural-language interfaces also let people who do not know how to program try tasks and build applications.
Enterprise text-to-SQL fails when business meaning is hidden in the schema
A general model cannot infer what an enterprise column means when its name is opaque. Orr gives a property-data example in which a user asks for properties with a satisfaction score above 75 percent. The correct SQL depends on a company definition of satisfaction score, such as a calculation involving work orders and another column. The definition is known by data engineers, but it is not present in the table name or in the model's general training. Fine-tuning, descriptions, and examples can teach the model this business logic.
Advanced SQL and privacy create separate production barriers
Orr says general models often fail on nuanced SQL, including queries that compare values with the average for each city and require nested logic or partitioning. A model can understand the business request and still produce invalid or incomplete SQL. Enterprises also need to keep schemas and business questions inside their own networks. That requirement rules out some third-party API arrangements, even when the model appears capable. A deployable system therefore has to address both query quality and where inference happens.
NSQL trades generality for SQL expertise and local deployment options
Number Station released a suite of open-source models for academic and commercial use. Their focused training is intended to make them better at SQL generation and to give users a base they can fine-tune on company data. Orr says the models give up some generality, including knowledge of other programming languages, in exchange for more SQL-specific behavior. The suite includes different model sizes so an enterprise can choose a quality, latency, and hosting trade-off that fits its hardware.
Most of the NSQL work went into creating and cleaning training data
Orr says data creation accounted for 90 percent of the effort. The first training phase used SQL gathered from open-source repositories so the model could learn SQL keywords and query patterns through next-token prediction. The second phase used about 300,000 curated instruction, schema, and SQL pairs to teach the model to follow user requests. Cleaning included checking that SQL was executable and that schemas matched queries. The team also discarded or relabeled sources with poor questions, used open-source models to generate questions from SQL, and made deterministic query changes for augmentation.
Specialized models can close the gap with much larger general models
Orr reports that NSQL's larger models outperformed available open-source foundation models on the Spider text-to-SQL benchmark, including models trained specifically on code. The Llama 2-based version also closed the gap with proprietary systems and matched GPT-4 on the reported comparison. Her explanation is that a smaller model can focus its capacity on SQL while a proprietary general-purpose model has to cover many tasks. The result depends on the specialized data and training rather than model size alone.
Instruction tuning teaches the model how to respond to a user's request
The training ablation separates the base model, the model after SQL pre-training, and the model after instruction tuning. SQL pre-training produces an initial improvement because the model begins to recognize SQL. The largest reported gain comes from instruction tuning, which teaches the model to connect a user's wording with the required SQL output. Orr says more high-quality data improves both phases, but instruction examples are what turn SQL knowledge into useful request-following behavior.
Generated SQL needs validation, auditing, and multi-step interaction
Orr does not present generated SQL as safe to execute without controls. A product should check whether a query is executable and whether it touches sensitive columns, and it should provide auditing for users and data engineers. She says the goal is to help engineering teams and give analysts more self-service access, not to remove engineering review. For complicated requests, a multi-step process can build a query gradually and expose intermediate checkpoints. Single-shot questions remain useful building blocks, but users may not be able to express deeply nested queries in one prompt.
"The goal here is not to replace engineering teams because these models, again, are noisy. It's more to help them do their job and allow business analysts to be more self-service."28:28
Who should watch
You are building a natural-language interface over company data and need to understand why a convincing demo can fail on real schemas and business definitions.
Your team is deciding whether to use a hosted model or deploy a smaller open-source model inside its own network.
You are curating text-to-SQL data and want practical guidance on parsing queries, checking schemas, relabeling questions, and augmenting examples.
