You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG.

Test it out. If it really and truly doesn't work, search for a script that creates question and answer pairs automatically with gpt-4. Then try using that for qLoRA. I have never heard of anyone successfully using that for a private document knowledgebase though. Only for skills like math, reasoning, Python, etc. I think the issue is that you need a LOT of data and it needs to repeat concepts or any facts you need to learn many, many times in different supporting ways.

What absolutely does not work is trying to just feed a set of documents into fine tuning. I personally have proven that dozens of times because I had a client who is determined to do it. He has been mislead.

What it will do is learn the patterns that are in those documents.

I used this simple example to teach about RAG, the importance of the system prompt and prompt injection. The notebook folder has a few more examples, local models can even do natural language SQL querying now.

I'll probably fix some things this week and then either update it or start from scratch. Guided generation, structured extraction, function calling and multi-modal are things I wanted to add and chainlit looks interesting.

Retrieval-Augmented Generation for Large Language Models: A Survey

https://arxiv.org/abs/2312.10997

The photos of this post are also good for a high level look

https://twitter.com/dotey/status/1738400607336120573/photo/2

From the various posts I have seen people claim that phi-2 is a good model to start off from.

If you just want to do embeddings, there are various tutorials to use pgvector for that.

Then you start making more specific queries? How old is he, how tall is he, etc.

And the game is you run a “questionnaire AI” that can look at a blob of text, and you ask it “what kind of questions might this paragraph answer”, and then turn around and feed those questions and text back into the system.

Is that a 30,000 foot view really of how this works?

You take the users question either embed it directly or augment it for embedding (you can for example use LLM to extract keywords form the question), query the vector db containing the data related to the question and then feed it all of LLM as: here is question form the user and here is some data that might be related to it.

Then you craft a prompt for the model along the lines of "you are a helpful assistant, you will provide an answer based on the provided information. If no information matches simply respond with 'I don't know that'".

Then, you take all of your documents and divide them into meaningful chunks, ie by paragraph or something. Then you take these chunks and create embeddings for them. An embedding model is another type (not an llm) that generates vectors for strings of text often based on how similar the words are in _meaning_. Ie if I generate embeddings for the phrase "I have a dog" it might (simplified) be a vector like [0.1,0.2,0.3,0.4]. This vector can be seen as representing a point in a multidimensional space. What an embedding model does with the word meaning is something like if I want to search for "cat" that might embed as a vector [0.42]. Now, say we want to search for the query "which pets do I have" first we generate embeddings for this phrase, the word "pet" might be embedded as [0.41] in the vector. Because it's based on trained meaning, the vectors for "pet" and for "dog" will be close together in our multidimensional space. We can choose how strict we want to be with this search (basically a limit to how close the vectors need to be together in space to count as a match).

Next step is to put this into a vector database, a db designed with vector search operations in mind. We store each chunk, the part of the file it's from and that chunks embedding vector in the database.

Then, when the LLM is queried, say "which pets do I have?", we first generate embeddings for the query, then we use the embedding vector to query our database for things that match close enough in space to be relevant but loose enough that we get "connected" words. This gives us a bunch of our chunks ranked by how close that chunks vector is to our query vector in the multidimensional space. We can then take the n highest ranked chunks, concatenate their original text and prepend this to our original LLM query. The LLM then digests this information and responds in natural language.

So the query sent to the LLM might be something like: "you are a helpful assistant, you will provide an answer based on the provided information. If no information matches simply respond with 'I don't know that'

Information:I have a dog,my dog likes steak,my dog's name is Fenrir

User query: which pets do I have?"

All under "information" is passed in from the chunked text returned from the vector db. And the response from that LLM query would ofc be something like "You have a dog, its name is Fenrir and it likes steak."

(Seems like this is what reinforcement training is, but I am just not sure? Everything seems to mush together when talking about gpts logic)

It fascinates me how much variance there is in peoples searching skills.

some people think they are talking to a person when searching e.g 'what is the best way that i can {action}' I think the number one trick is to forget grammar and other language niceties and just enter concepts e.g. 'clean car best'

The same term was the name of the device + something about the power source.

The result from the client network - my phone/client computer nothing related to the search for 4-5 pages.

Same search from work - second result was what I was looking.

So it seems there is a relation with your search history, but somehow connected with the related search history from the same ip/network.

It must be possible to create a fixed set of google searches and rate the location based on the results. So you could physically travel to a Starbucks 20miles away to get the best results for the 'best USB-C dongle reddit'.

https://kagi.com/search?q=ml+rag

https://www.google.com/search?q=ml+rag

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

With regards to its efficacy, I haven’t gone to production with it yet but I was reasonably impressed.

I uploaded 100 legal case documents to Bedrock via Claude and could push it pretty hard asking about the various cases and for situations across the knowledge base.

It did feel like it broke down and got confused at a certain point of complexity of questioning, but I still think it’s already useful as a “copilot” or search engine and surely it will only improve over time.

I had tried to suggest continuous pre-training to my client but it seemed expensive and when I mentioned that he lost interest and just kept wanting me to do fine tuning.

Also to clarify, did you do the continuous pre-training or RAG? And did you compare the efficacy of one or the other or both?

I got a toy demo up and running with continuous pre-training but haven’t evaluated it unfortunately.

We did all we could to steer him toward a correct path of understanding. Sadly we launched a working product but he doesn't understand it and continues to miss represent and miss sell it.

After continuing to give him time and follow up with him (I tend to personally do this with Clients like this), I can tell he is starting to realize his lack of understanding...

LlamaIndex is pretty good.

We originally started out building features with LangChain (loading chains from YAML sounded good—it felt like it would be easy to get non-engineers to help with prompt development) but in practice it’s just way too complicated. Nice idea, but the execution feels lacking.

It also doesn’t help that LangChain is evolving so rapidly. When we first started using it a lot of code samples on the internet couldn’t be copy/pasted because of import paths changing, and at one point we had to bump by ~60 patch versions to get a bug fix, which was painful because it broke all kinds of stuff

Disclaimer: I work at deepset

[1] https://github.com/deepset-ai/haystack

Things get messy when the number and type of documents increase. Below are the reasons why you may need advanced RAG.

1. Intelligent Data Parser 2. Chunking efficiently 3. Choice of embedding models 4. Query transformation 5. RAG technique 6. Prompt design 7. Feedback loop

Check out my blog on the 27 parameters, considerations and techniques one could follow to build a State-of-the-Art Chatbot.

https://www.lyzr.ai/27-parameters-techniques-considerations-...

So here is the quick guide.

For simpler usecases - inbuilt vector database RAG is sufficient For more complex ones - LlamaIndex or Langchain options are suitable For enterprise grade production use cases - Lyzr's SOTA RAG architecture comes in handy

https://www.microsoft.com/en-us/research/group/dynamics-insi...

Not quite. It does work, albeit likely not optimal.

See https://github.com/bublint/ue5-llama-lora

1. RAG: Most popular and works really well on smaller datasets. It is limited by number of vectors/embeddings. A typical embedding could be of 1000 tokens in size. Llamaindex did a lot of engineering on this and their techniques work pretty well. The problem with large datasets is almost always that users don't like writing long prompts/queries so the answers are more generic.

2. Finetuning + RAG: You can finetune a model on the expected outputs. If your datasets have the knowledge which might already be on open internet (blogposts, articles, anything non proprietary), then finetuning would work really well in combination with RAG, especially for large datasets. It may not work if you are working on a proprietary knowledge hard to find on open internet.

3. Continual pretraining - large large datasets, and when the knowledge is proprietary. I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Needs a model that is trained on their data and then Instruction Tuning of top of that. Most likely you wont need to do this

Wow so RAG is basically a toy for demos and low effort MVPs. 70GB is tiny, it’d barely qualify as “big data” 20 years ago.

Is anyone trying more advanced stuff like knowledge graph augmented generation to try to expand on that?

- kg DBs have the same retrieval scale problem, and often built on top of the same DBs

- traditional kg mining would still use LLMs, except instead of storing rich text ("in a clearly friendly and playful manner, the dog chased the cat") or less-rich high-dimensional chunk embeddings of the same, they get discretized down to much lossier rdf ontologies like (dog,chased,cat). There are uses, like building an entity index, but I wouldn't use for core RAG operations like answering questions accurately

- kg can be useful in the sense of, after chunking & embedding, adding *additional* linkages, such as adding summarization hierarchies and then linking related summaries back to source citation chunks. So runtimes can look up not just similar chunks, but also those that are a logical hop away, even if the embeddings are dissimilar, and via pre-summarization, incorporate a lot more insight. Though that's not a traditional kg, it highlights needing non-vector linkage tracking.

We are working on real-time large-scale projects like teaching LLMs to understand the news as it breaks as part of louie.ai, and if folks have projects like that, happy to chat as we figure out our q1+q2 cohorts. It's a fascinating time -- we had to basically scrap our pre-2023 stack here because of the significant advances, and been amazing being able to tackle much harder problems.

A simpler way here is just train the model unsupervised so all the knowledge is there in the model, and instruction tune it on the use-cases you want. Simpler from human effort perspective. Somewhat costly though the cost of storing that many vectors would be more than training the model itself. Everything else requires a lot of custom effort. Knowledge graph augmentation is probably the next step in the hype cycle, but it does not solve the fundamental human problem of writing fewer letters. (Training solves as changing 1-2 keywords do the trick if the generic string does not get the answer. See how Chatgpt changes answers if you tweak your prompt a bit). In a way RAG is an engineering solution to what is basically a data problem. It works for many cases, but when it does not, people will have to solve it via data science.

> Wow so RAG is basically a toy for demos and low effort MVPs

I would not say it's for demos or low effort MVPs. Many companies wont have that amount of data. You can also segregate it by teams. Eg: customer support has one, sales has one, product has one. Then, a golden use case is for parsing user docs. We created one for GST queries in India that works quite well.[1]. It's a search engine, but points to right docs at the source when you ask about any clause. Useful for CAs only and addresses a very narrow use case.(it's a market need as the notifications are published in PDF format and not indexed by Google)

[1]:https://clioapp.ai/gst-search

On the other hand I've got a terabyte of text extracted from LibGen - let's say I can ignore the half that is fiction and I can dedupe the rest further by 80% - that's still 100gb. On top of that I've got 300gb of text extracted from court documents and that's just from California! I haven't even downloaded the federal dump yet, let alone the other 49 states. Even if I limited myself to just the US Code and Federal Code of Regulations, that's hundreds of millions of tokens of very dense text. Embedding based RAG has been pretty much useless in each of these cases but maybe I just suck at implementing the retrieval part.

What's the data size on the GST search engine? Do you have any examples of more complex queries?

The only thing that has even been remotely useful to tackling the kind of questions I want to ask of my data sources is having the LLM generate search queries, navigate the knowledge graph, and rank the relevance of retrieved snippets but that often takes dozens if not hundreds of LLM calls which is incredibly slow and expensive.

Or would you read it bit by bit, taking notes and summarising as you went along. Then compiling your notes/summaries into a final report?

Because I don't see why we expect these models to be so superhuman when a 100K context would already be considered superhuman memory.

Imagine me regurgitating 100k tokens worth of dialogue at you and expecting you to take into account every thing I said. I know I couldn't do it, ha ha.

I'm talking about computers, not humans.

With a large amount of data, a large amount of data can be "relevant" with a loose query.

I think in those situations it's fine, using a model with an extra large context and similarity etc filters quite tight.

Developing it to realise when there are too many results and to prompt the user to clarify or be more specific would help.

Companies that want to trawl data like this can just deal with it and pay hardware that can run model for >100k context.

If >all

It's totally doable, but not "out of the box".

LLM work best on use cases where the working context is the length of a short research paper (or less). Building with LLM is mostly a exercise in application engineering on how to get them the most relevant context at the right time and how to narrow its scope to produce reliable outputs.

Fine tuning can help specialize the LLM model to perform better, but AFAIK, the training sets are relatively small (in big data terms)

You can do something using image embeddings to get what you want.

You have to upload your documents to S3, create a “Knowledge Base” then sync your documents into a vector database like OpenSearch or PineCone. You are then good to go via their playground or the AWS API.

I made a video here describing the process, check around 14 minutes in:

https://ensembleanalytics.io/blog/introducing-bedrock-knowle...

Bedrock is a decent product I think. All of the models in one place (apart from the big dogs from OpenAI) and a common API across them.

I agree it is high and it is not exactly serverless. You pay by the hour.

Bedrock itself is charged based on tokens exchanged.

I think Pinecone is a cheaper database for hobby and small business projects, though I haven't looked into it.

You can reduce a lot by committing to a 6-month contract, but it won't get cheaper than about ~$5,000/mo.

That's prohibitively expensive for small projects.

Fine tuning GPT 3.5 is much cheaper.

AWS OpenSearch has fairly good integration so you could look up costs for that. It’s not the cheapest AWS service to run and not exactly serverless as you pay by the hour.

This isn't the same as the training data the LLM is trained on. As a result, it doesn't take advantage of the entire document set. So, if you have a billion documents, only 1-3 chunks will be picked for the final answer. When you know a question spans many many documents, the answer is never going to cover that.

You could make a recursive also where you parse all the chunks, generate summaries of those and then pass them to the next chunk sequentially and so on. But you can imagine how expensive and slow that will be. It might still work for you but this is a very lossy approach.

If you have that much data in your vdb, and the user is querying a slice, it just won't do to have them ask something too generic, they need to be asked to be specific.

Ie "what was that thing from last year?" "Sorry, could you be more specific?" "Oh uh the thing to do with the financial reports" "Financial reports, yes, what about them?" "It was something to do with the way we generate them" "Ah yes, the generation of financial reports; last year in December your team leader said on Slack...etc"

The most feature complete implementation I've seen is h2ogpt[0] (not affiliated).

The code is kind of a mess (most of the logic is in an ~8000 line python file) but it supports ingestion of everything from YouTube videos to docx, pdf, etc - either offline or from the web interface. It uses langchain and a ton of additional open source libraries under the hood. It can run directly on Linux, via docker, or with one-click installers for Mac and Windows.

It has various model hosting implementations built in - transformers, exllama, llama.cpp as well as support for model serving frameworks like vLLM, HF TGI, etc or just OpenAI.

You can also define your preferred embedding model along with various other parameters but I've found the out of box defaults to be pretty sane and usable.

[0] - https://github.com/h2oai/h2ogpt

We've built what we call an Unstructured Data Transformation Layer. Think about it like an assembly line from raw text to tables in your data warehouse.

We don't use Llamaindex, we have our own (proprietary) piece of tech that does this. We can and have been outputting gold-standard tables on top of a lot of different types of context (legal docs, call transcripts, customer reviews, chat logs, emails, blog posts, social media posts, etc...) and looking to expand to more interesting domains soon.

If anyone wants to hear more hit me up at tom [at] flexor [dot] ai (does this still work or are scrapers smart enough nowadays to just grep for this too lol)

Fine tuning does result in degradation of the overall model (https://twitter.com/xaiguydotagi/status/1737082280835703142) and so various RAG techniques may be desirable. As others have mentioned, LlamaIndex is a neat solution to build RAG pipelines: https://docs.llamaindex.ai/en/stable/optimizing/production_r...

I think harpercarrol link is a pretty good one, but it basically just feeds in the documents for completion, which isn't a good approach. The dataset needs to represent how you want to use it.

This one might also be helpful https://www.deeplearning.ai/short-courses/finetuning-large-l...

Honestly surprised how almost everyone is saying to use RAG (on its own). One strong benefit to RAG is the data can change, but has lots of failure modes.

People often use hybrid search (fuzzy or bm25 etc alongside embedding search) which I suppose is still RAG.

But fine-tuning models to be better at RAG is valuable as well, increasing accuracy.

https://ragntune.com/blog/Fine-tuning-an-LLM-to-be-good-at-R...

Ideally, I'd try both. Fine tune on both the documents (create a question / answer dataset with gpt4) and rag instruction fine tune it.

Then

make ingest /path/to/folder/with/files

Then chat to the LLM.

Done.

Docs: https://docs.privategpt.dev/overview/welcome/quickstart

Cheshire Cat [0] looks promising. It's a framework for building AI assistants by providing it with documents that it stores as "memories" that can be retrieved later. I'm not sure how well it works yet, but it has an active community on Discord and seems to be developing rapidly.

The main perk over the cloud options is that you can point it at any language model, including fully local—my local install pointed at my local Ollama running Mistral.

[0] https://github.com/cheshire-cat-ai/core

Feed it a subreddit. Post results.

Other than OpenAI, there is the newly introduced „continued pre-training“ of Amazon Bedrock - but I haven’t tried.

RAG: I think that‘s a fundamentally flawed concept, but RAGfluencers will disagree. ;-)

I built a vector database from the documents, and I query the questions against it, which is very fast. This is the RAG (retrival augmented generation) step others mentioned.

The results, which are literal extracts from the documents, but short ones, are given to the model, which produces an answer. This is the slow part.

I used many of Langchain's tools to manage the whole process.

You can try it on Supawiki, with one of the featured wikis. Then, if you are ok with a solution that hosts your documents for you, you can upload them and use our solution.

https://neuml.github.io/txtai/

i.e given a text, always return back a certain set of fields. For some keys here is the possible set of enums etc. One shot prompting does work but curious how others approach this if you have training data on hand.

[0]: https://github.com/outlines-dev/outlines [1]: https://github.com/1rgs/jsonformer

Crude, I know, but it’s compatible with every model. Which is useful if you want to compare the many different models out there.

LLMs are so much better for this than LSTMs now.

Aside from that, take a look at llama.cpp grammars.

I've seen that there are a lot of approaches, but none that has gained traction, there isn't a clear consensus...

- https://notebooklm.google/ (US or VPN): uses the "gemini pro" model.

- poe.com: You need to "create a new bot", disable "Make bot publicly accessible," and then "add a knowledge source." this offers many models, although the best ones require a subscription.

The conventional approach is suitable for software engineer where they may not be less familiar with AI. The configurable approach is suitable for ML engineer where they have sophisticated uses and would want to configure chunking, indexing and retrieval strategies.

[1]: https://github.com/embedchain/embedchain

Would a five years old laptop work? Do you need a beefy GPU? Are you using some prepackaged software?

I uploaded an AWS study guide, and I'm asking for example questions for my testing. As far as I can see, I can tell no way to determine if it's pulling from the guide, or if it's from GPT's data.

I've tried out working with custom documents in two different ways for different types of data:

* Once using LlamaIndex + Chroma[0] to transcribe and then conversationally query video contents (using GPT 3.5 or 4 as the backing LLM).

* Once using GPT Plus, uploading long-form PDFs of my own fiction books to the GPT's knowledge base. I use this to help me remember character names and timelines (not always accurate, so results need to be treated with caution) and help brainstorm story or tech ideas for my world.

Both work for what I'm using them for. I feel like option one is more customizable and easier to tweak for the types of results I would want, if I have concrete requirements about what kind of output I'm looking for. Option two has a lower barrier to entry and is just a little lower effort (no need to run your own app).

For the next iteration, I'd like to try out AWS Bedrock and compare the workflow and results.

[0] https://www.daily.co/blog/search-your-video-content-library-...

There are also examples that combine semantic retrieval and structured retrieval.

I'm very happy with its results, even though the system is still young and a little bit janky. You can use it with either GPT API, or your local models through LiteLlm. (I'm running ollama + dolphin-mixtral)

Based on this assumption, I have a couple of questions:

1. Is chunking the most significant factor in RAG quality?

2. If there are no limitations, would humans that are experts in that dataset, be the best people to create chunks?

ChatGPT-4 is powered by GPT-4 Turbo at this point, so it has a context window that is much larger than 4096 tokens, whether it knows it or not. The ChatGPT application may limit the context size from reaching the full 128k to keep costs down, and it will be using some of the context window for its own purposes, but it's certainly able to maintain context across more than 4096 tokens of conversation.

ChatGPT is a consumer facing service that wraps the GPT-4 foundational model but at some point will likely wrap gpt-4-turbo.

Signing up for OpenAI API access or Azure OpenAI Services will grant you access to this model (with some rate-limits in place given its a preview model).

https://www.anthropic.com/index/claude-2-1-prompting

Why? This seems like a quick road to remote-execution-vuln-as-system-service.

> and voila we can search for anything with natural language.

What if your model keeps misunderstanding you?

For instance, if I search for a specific product name but accidentally mistype it, the resulting encoded vector might not be similar to the vector for the correctly spelled product name ?

In your example sum of embeddings for non-typo and typo version should have close cosine distance if it trained correctly.

Then create an index about the metadata of each doc. So that you can ask the RAGbot what all it can answer about.

Another way to ensure it stays on-domain is to generate synthetic questions & check for similarity against user queries. There's a whole rabbit hole of query decomposition to avoid straying off topic as well.

Early next year I’m preparing something similar for my team, so I’ll surely look into the useful links/recommendations posted by fellow HNers :-)

Instead you can extract text embeddings from your documents, put them in a vector DB, and then you have a super search. You can convert your search query to an embedding, search the DB and keep the e.g. 10 closest matches.

https://www.github.com/jerpint/Buster

Tried this summer, and kinda worked!

For the first (fine tuning) follow “AI Jason” on YouTube. He has some great tutorials.

For the second (RAG or similar), fire up a cloud VM with GPUs or use Ollama locally and read through the LlamaIndex docs on how to build a RAG pipeline.

As several other comments in this thread have already indicated: this is almost always the wrong direction. Which is confusing because it's the direction everyone always assumes they should go in at first.

The approaches that does work is surprisingly simple: take the user's question, search for snippets of your documents that appear to be about that question, then paste all of those snippets into the prompt along with the user's question and see what answer you get.

This is known as RAG: Retrieval Augmented Generation. It's a very powerful approach.

We use RAG at my job, but we don’t do any preprocessing on the message from the user, so the results are not always great for us.

Do any of you have experience using a small local model just for extracting keywords from messages which you then use for the retrieval? And then feed the search result and your prompt into OpenAI or whatever as normal.

Embedding and chunking large amounts of documents is expensive though, in both compute and storage.

The other trick I've been planning to explore is using an LLM to turn the user's question into a small number of normal FTS search queries and then run those to try and get context data.

I have also been working on this. I still fail to see why this approach isn't the default frankly. There's little benefit to vector databases.

Also maybe try to include tags or categories when you index and then you can filter on those when doing the vector search. Might get a similar effect from BM25.

Also llamaindex does RAG better than some other solutions.

so is the default to just run the RAG search index on every prompt and if it returns nothing then you get the plain answer from the base model otherwise you get the augmented answer?

At least, that's what I've rapidly been concluding.

I'm still holding out a bit of hope for there being more though.

Semantic Kernel is MS's response to LangChain and LlamaIndex - available for .NET, python and Java.

Using their Memory support (using Azure Cognitive Search), it gives a powerful RAG quickstart, which you can combined with Azure Document Intelligence to chunk your source documentation into memories that your foundational models can later use.

(Disclaimer: Its only very recently gone 1.0 and is still likely to undergo API change as the LLM domain itself is still rapidly evolving - I've substantially forked the project for my own needs, but I hope that as it stabilises, I can contribute PRs for some of my more advanced use cases).

Likely I misunderstood about how RAG works with Azure AI Studio, so sorry in advance

Check it out, specifically steps 3 and 4. As with almost every Microsoft CLI tool and SDK, it's clunky... and you can tell everyone is rushing this AI shit out as fast as they can to stay in the game. But what you want should be doable.

An under-appreciate benefit of RAG is the ability to have the LLM cite sources for its answers (which are in principle automatically/manually verifiable). You lose this citation ability when you finetune on your documents.

In Langroid (the Multi-Agent framework from ex-CMU/UW-Madison researchers) https://github.com/langroid/langroid we’ve implemented a number of RAG techniques: our DocChatAgent uses a combination of lexical and semantic retrieval, reranking and relevance extraction to improve precision and recall: https://github.com/langroid/langroid/blob/main/langroid/agen... All the code is laid out clearly so can be tweaked. We have companies using Langroid in production (e.g. for customer-support); they especially like the RAG and multi-agent features.

One of the interesting techniques in Langroid is a numbering trick for the relevance extraction stage (having the LLM extract verbatim relevant parts of passages) — instead of having the LLM “parrot” out the relevant portions, thus wasting time and tokens, we have it just spit out the relevant sentence numbers from a pre-annotated passage. We use a tool/function-call for this, leveraging Langroid’s task loop that seamlessly works for tool handling as well as sub-task handoff.

Many interesting RAG applications often require more than simple question-answering (e.g extract structured info, match doc against requirements etc) and these scenarios benefit immensely from having multiple agents (so you get separation of concerns, modularity and easier state management). Langroid simplifies this type of multi-agent setup with its unique conversational task loops, e.g https://langroid.github.io/langroid/examples/agent-tree/

Colab quick start that builds up to a 2-agent system for extracting structured info from a document:

https://colab.research.google.com/github/langroid/langroid/b...

Among many other things, Langroid also has full support for the OpenAI Assistants API, so you could use the “built-in” RAG from this API, which is a convenience but is a black box, I.e you don’t know what retrieval algo it is using, how it is filling context, and the tokens consumed.