Sankalp Gilda explains how tsbootstrap preserves time-series structure when generating bootstrap samples for confidence intervals and model validation.
2
Uri Shamay presents GeniA, an open-source engineering assistant that retrieves relevant tools before calling them, while warning that it is not ready for unsupervised production work.
3
Raahul Dutta walks through LLM risks including prompt injection, insecure plugins, package hallucination, denial of service, model theft, and poisoned training data.
Summary
This Mini Summit combines three talks. Sankalp Gilda introduces tsbootstrap, a Python library for resampling time-series data with methods such as moving, stationary, circular, and residual bootstrapping. The goal is to preserve temporal dependencies that ordinary IID resampling destroys. Uri Shamay presents GeniA, an open-source engineering assistant that connects an LLM to operational tools. It uses a vector database to retrieve only the functions relevant to a request, which helps with the limited context window. Shamay is direct about its limits: function-calling models can select the wrong tool, and GeniA needs stronger access controls before it can safely operate production systems. Raahul Dutta focuses on LLM security. He demonstrates how suffix-based attacks can bypass model refusals and describes risks from untrusted plugins, fabricated packages, exposed secrets, repeated API calls, and poisoned models. The closing discussion returns to the gap between impressive demos and production systems with adequate safeguards.
Time-series bootstrap samples must preserve temporal dependence
Sankalp Gilda explains that ordinary bootstrapping resamples observations as if they were independent and identically distributed. That shuffles temporal order, destroys autocorrelation, and breaks the assumptions behind statistical analysis. tsbootstrap creates new series that retain time structure, so users can pass them into forecasting algorithms and estimate confidence intervals. Gilda presents the library as a bridge between data scientists who need production-ready code and machine learning engineers who may not work deeply with time-series methods.
Block bootstrapping trades statistical bias against variance through block length
Block bootstrapping samples consecutive observations instead of individual points. Moving blocks can overlap, circular blocks can wrap from the end of a series back to its beginning, stationary blocks draw lengths from a geometric distribution, and non-overlapping blocks skip overlaps. The block length affects the trade-off between bias and variance. Gilda notes that blocks can still create implausible joins when the data has trends, breakpoints, or uncorrected seasonality.
Residual bootstrapping uses a fitted model to handle more complex series
Residual bootstrapping first fits a time-series model, resamples the residuals, and adds them back to the fitted values. Gilda shows an autoregressive example where the generated series look more sensible than samples made by simply joining blocks. He says this approach can handle changing distributions and complex dependencies better, and it is less sensitive to block size. tsbootstrap also allows residual and block bootstrapping to be combined.
GeniA retrieves only relevant tools before asking the model to act
Uri Shamay describes GeniA as a virtual engineering team member that can connect an LLM to company services such as Kubernetes, cloud providers, build systems, databases, and logs. Loading every available API into the model would exceed its context window. GeniA instead stores tool definitions in a vector database, retrieves the functions related to a user's request, and supplies a smaller set to the model. New tools can be added through a YAML file, and some APIs can be connected through a URL without additional code.
GeniA is an assistant because autonomous production operation remains unsafe
Shamay says a high-level request such as troubleshooting production would require the model to inspect systems including Grafana, logging services, Kafka, and Redis. In tests with GPT-3.5 and GPT-4, the model sometimes failed to call the relevant functions even with carefully written prompts. GeniA did not yet have SSO or role-based access control, so Shamay recommends private Slack channels for the early version. The project can run powerful actions, such as restarting a Kubernetes cluster, but it should not receive broad production authority without stronger safeguards.
Raahul Dutta describes prompt injection as a major barrier to using LLMs in critical systems. A model connected to tools may execute instructions embedded in data, and small changes to an attack string can bypass rule-based filters. He refers to suffix-based attacks that change a model's refusal into a harmful answer. Dutta says there is active research in this area, but he does not present a complete defence that stops prompt injection.
LLM applications can expose enterprise systems through plugins and generated packages
Dutta warns that an untrusted LLM application could insert logic between enterprise data and the model, sending information to an attacker's server. He also describes package hallucination, where a model recommends a package that does not exist. An attacker can publish a malicious package under that name, causing a developer to install it while following generated code. Similar concerns apply to plugins that execute generated Python or connect to private data. His advice is to inspect the code and verify packages before using them.
Attackers can raise costs, steal models, or poison model outputs
Dutta describes denial-of-service prompts that ask an LLM to repeat an action hundreds or thousands of times. This can increase API bills or consume the allocation of a self-hosted system. He also discusses model theft and the difficulty of tracking the licences and origins of open-source models. With poisoned models, an attacker can insert false information during fine-tuning or ingestion and publish the model for others to use. He mentions watermarking and differential privacy as areas of work intended to reduce these risks.
The speakers separate coding demos from production engineering
In the closing discussion, Uri Shamay says many LLM engineering projects are impressive demos but do not yet account for security in enterprise production systems. Sankalp Gilda says generated code is currently most useful for small, specific pieces and still needs checks for quality, fairness, and accuracy. Raahul Dutta adds that research on factuality and security is active, but current applications should not be trusted without review. The group expects more tools for prompt testing and monitoring, while admitting that this area is still immature.
"LLM security is a major concern. Many people including myself believe it's the main reason why companies hesitate to adopt that new technology for critical systems."Uri Shamay26:26
Who should watch
You are building confidence intervals or validating forecasting models on dependent time-series data and need resampling methods that do not destroy temporal order.
You are connecting an LLM to internal engineering tools and need a concrete account of tool retrieval, context limits, permissions, and failure modes.
You are reviewing an LLM application for production use and want examples of prompt injection, malicious packages, insecure plugins, denial-of-service prompts, and data poisoning.
