Anushrut Gupta, Hasura, Senior Product Manager, Generative AI
“Over the last 3 months, summarize the top billing issues faced by my enterprise customers within the first 30 days of their onboarding.”
On the surface, building an internal AI customer intelligence application that can answer questions like this is a perfect use-case for Gen AI. However, building a production ready app that retrieves the data (RAG) before hauling it off to your favorite LLM for summarization soon becomes a terrible engineering experience.
The data is spread across 3 places: a tickets database (eg: elastic), a CRM (eg: salesforce) and your user-accounts transactional database (eg: postgres). In production, your app can’t access the data from these databases directly. Given security & privacy concerns, your app won’t have direct access to these databases. Making independent retrieval requests to each of these sources and then joining them in memory might be prohibitively expensive and needs a level of query planning to do efficiently. Moving all data into one location is expensive to build, maintain and govern Predictable quality is further made hard because underlying data formats and storage interfaces are continuously changing. Different types of user queries might require additional filtering and joining of data, which becomes hard to generalize.
APIs solve almost all of these very well known challenges. APIs offer standardization and security. APIs can provide a stable contract to interact with underlying data.
And in all likelihood, you already have APIs on these internal and external data sources.
Ironically, while APIs have become a necessity for other parts of the stack, they are clearly not the first thing that AI engineers building RAG reach for.
In this talk, we’ll discuss: Why API based retrieval doesn’t work well for RAG What we need from our existing internal and external APIs to make them RAG ready How we can get existing APIs to become RAG ready without needing to rebuild the APIs
This talk will be technical, with code demos (possibly with some live coding!) and end with key resources (reference architectures, API best practices, tools/technologies) that attendees can take back to their work.
