Watch on YouTubeThe chatbot retrieves relevant documents from Qdrant and sends them to an LLM to generate a response.
A separate embedding pipeline processes about 500,000 lithium-related documents because the built-in LangChain and Qdrant workflow was too slow.
Raahul Dutta fine-tuned Falcon 7B on the team's data and used prompt templates to generate policy papers with sections such as an executive summary, recommendations, introduction, conclusion, and source documents.
Raahul Dutta describes a chatbot built at Elsevier to reduce the manual work involved in reviewing research about topics such as lithium policy. The system embeds a large document collection, stores the vectors in Qdrant, retrieves the most similar documents for a user prompt, and passes them to an LLM. The team used its own embedding pipeline for about 500,000 documents because the built-in LangChain and Qdrant process was too slow. They uploaded vectors with Rust and tuned Qdrant index settings for search latency. For generation, the team tested several models and fine-tuned Falcon 7B on its dataset. A Gradio interface exposes the workflow, while prompt templates produce structured policy papers with source documents attached. The demonstration shows the architecture, implementation code, prompt configuration, and generated output. Dutta is direct about the system's practical purpose: helping policy researchers review the generated document before sending it onward.
Elsevier has many research articles, and clients ask for information about a topic over a recent period. Dutta gives lithium as an example. A team would read the lithium-related documents, write a policy paper, and forward it to organisations such as the EU and UN. The chatbot is intended to reduce this expensive manual effort. It reads the relevant documents and generates a policy paper that people can review.
In the described LangChain architecture, a user prompt first goes to an embedding model. The resulting embedding is used to search a vector database. Dutta says the system uses Qdrant to return the three or four most similar documents, according to the configured parameters. Those retrieved vectors and documents then go to the LLM, which generates the response. The team changed parts of this standard architecture to improve prompt quality and reduce latency.
The team had about 500,000 documents related to lithium. Dutta says the built-in LangChain and Qdrant solution was too slow for creating all the vectors, so the team built a pipeline based on a Kubeflow pipeline and a separate processing approach. It could produce embeddings for thousands of documents in a couple of minutes. The resulting vectors were stored in NumPy before being uploaded to Qdrant.
The team chose Qdrant because Dutta found its documentation useful and because it supports multiple languages and clients, including Rust and Python. They used Rust to upload the 500,000-vector database in minutes. They also fine-tuned index optimizer and memory-map threshold values to improve search latency. Qdrant was deployed in Docker in their cluster, where it could scale with load and scale down to zero.
Dutta describes tests with several models. Vicuna 13B and OPT 6.7B produced good results, while BLOOM did not. The team was also trying RedPajama and had fine-tuned Falcon 7B on its dataset with good results. Their framework accepts different model types and uses parameter-efficient fine-tuning by targeting selected model layers. Dutta says the Falcon 7B fine-tuning run used half a million data items and took about 30 minutes.
The notebook demonstration initializes a local Qdrant client, sets the collection name and content payload, and calls a LangChain initialization method when the conversation starts. The LLM endpoint is supplied through a URL and parameters. Gradio provides the interface. Users can select from prompt settings held in a dictionary, then generate the document through the chatbot workflow.
The team tested different prompts for different models. For Falcon, the generated output includes a disclaimer, title, executive summary, recommendations, introduction, and conclusion. Dutta says the system also appends the source documents used for the article. The resulting document is intended for the organisation's professionals to review before it is provided to clients or other recipients.
"We are trying to reduce this manual effort, expensive manual effort, with LangChain, Vector DB and LLM model."01:13