Phi-3 Text-to-SQL

The database has nothing to do with it. Running the SQL on the data takes about a millisecond. The wait is the model writing the query. This is Phi-3-mini, a 3.8 billion parameter model, and on this free Space it runs on 2 CPU cores with no GPU. For every question it reads the whole schema and then writes the SQL one token at a time, and a CPU is slow at that. So the amount of data does not change anything here. The model is what takes the time.

It is small on purpose. Five departments, 13 employees, 6 products, and 75 sales rows, so a little under 100 rows in total. The point is to watch the model write correct SQL for a schema it knows, not to query a huge warehouse.

Yes, quite a bit. On the held-out questions the fine-tuned model gets about 75 percent of queries exactly right when you run them, versus roughly 42 percent for the base model on its own. It also writes valid SQL every time.

With QLoRA, on 50 hand-written question and SQL pairs, plus 12 kept back for testing. It only updated about 0.12 percent of the model's weights, which comes out to a 9 MB adapter file. The whole run took around 3 minutes on a 6 GB laptop GPU.

Hugging Face used to give Spaces a free T4 GPU, but that costs money now. Running on CPU keeps the demo free, and that is the trade for the speed. On a GPU this same model answers in a second or two.

The fine-tuned model is merged into the base weights and quantized down to a 4-bit GGUF file, then served with llama.cpp. That file is about 2.4 GB, roughly a third of the full-size model, and it loses almost no accuracy on this task.

Not in this demo. It was trained on this one fixed schema, the tables you see in the sidebar. A different schema would need its own round of fine-tuning, or a much larger general model.