Transcript for:
RAG System Building Guide

welcome to the AI Automators NAND RAG Master Class Unlike a lot of RAG courses that are heavy on theory this course is super practical and hands-on We're going to go very quickly into actually building out solutions and you'll be able to learn the key concepts as you go So I encourage you to build along in parallel as you're working through the course and to use documents that you're familiar with because to really understand whether your RAG system is accurate or not depends on your understanding of the documents that you're actually feeding into it and whether you're getting quality answers back These are the full workflows that we're going to build during this course spread across seven different lessons The beginner and intermediate sections of this course are completely no code and sections of the advanced course require some vibe coding with an AI model So there isn't any programming experience required for this Some experience of NN would be an advantage But if you follow along with me step by step you'll quickly get a handle on how this brilliant platform works And if you make it to the end of the course this is what you're going to end up with A fully working and highly accurate rag system with the vector store built-in superbase which also holds our record manager And this system has a fully fleshed out rag data ingestion pipeline You'll learn how to create and update vectors from files in a Google Drive folder You'll also learn how to delete vectors for files that are dropped into the recycling bin here We also have a web scraping trigger for this rag pipeline So you can crawl entire sites on a daily basis and ingest them into your vector stores This system handles a number of different file types And crucially it uses Mistral OCR to actually scan PDFs that are not machine readable And we have a sophisticated record manager to make sure that we're not duplicating data in our vector store And finally we generate contextual chunks to provide the most accurate retrieval So you're going to learn how to actually create this entire data ingestion pipeline in this course and I walk through it all step by step And then when it comes to our rag agent itself not only can it handle metadata filtering but you're also going to learn how to use hybrid search as well as how to rerank the results that come back from your vector stores So this is absolutely the best practice in Rag from contextual embeddings hybrid search ranking to make sure you're getting the highest quality retrieval from your system These workflows are available to download and import from our community the AI automators but they're not needed to get the most from this course as I go through everything step by step But check out the link in the description if you'd like to get a head start You can jump to the various sections of the masterass using the timestamps in the video Our beginner section kicks off with lesson one where we demonstrate what rag is And in lesson two we use nadn's AI agent feature to create a simple rag agent In lesson three then we move on to an intermediate level where we set up a rag pipeline to ingest documents In lesson four we extend that pipeline to handle different types of files and formats And then we extend the whole system to support web scraping using firecrawl AI And then onto our advanced section In lesson six we implement hybrid search We implement re-ranking with coheres 3.5 reranker model And then in lesson seven we implement a pattern called contextual retrieval While you can jump ahead to the various lessons the n workflows do build upon each other So I do recommend going through it in sequence I'll make sure to bookmark this video so you can come back to it later I guarantee you this is the most comprehensive NAN rag course on YouTube so it'll be worth it I put a huge amount of effort into this so I'd really appreciate it if you gave the video a like and subscribed to our channel And let's get started with lesson one building out a very simple rag agent In this lesson we're going to be using the easiest possible means to set up a Rag system just to get up and running So we're not going to worry about chunking or embeddings or anything complex And let's not even worry about an automation platform So let's just demonstrate what Rag can actually do There are lots of different rag style web apps that you can use and from my experience notebookm is absolutely one of the best So you just go to notebookm.google.com and once you sign in you land on this page and then you just click create new notebook and this is the rag aspect of it here You need to add sources because this chatbot within notebook LM is going to be grounded in the data that you provide it So let's upload some files to test this out And you may have seen a McLaren Formula 1 car in the background here Let's use the Formula 1 regulations to test this out So let's grab a couple of documents and upload them to notebook LM and then we can chat to these documents So this one is the technical regulations These are the financial regulations and these are the sporting regulations So let's start there So those have all downloaded and I just drag them in here There's the first one and there's another So there's three documents Then it takes a while to process them So what's happening in the background here is it's actually breaking down these documents embedding them in a vector database so that we can actually have a conversation with them in this chat interface here and you can see at the bottom it says three sources So if we ask a question what is the plank assembly which is a part of an F1 car It's then able to provide an accurate answer based off the sources that you uploaded So based on the sources it's a key component fitted below the central surfaces of the floor body So you have lots of these citations So number one is referring to the technical regulations and this is the chunk that was actually indexed And if you keep going down you can see number two still the technical regulations down here Number 11 is a different section of the technical regulations So you get the idea It's fully grounded in the documents If I ask a different question now what's the weather like in New York it's going to look into those documents and it's not going to find anything that's of relevance based on the sources provided and or conversation history There's no information about the weather in New York and then it explains what the documents are about So that's what's great And if you ask another question which is semi-related what size tire is fitted on an Audi A4 and you're getting back an accurate answer based on the sources provided There is no information regarding the tire size So the LLM isn't going into its training data to actually figure that out because the LLM is most likely trained on a huge corpus of information and it probably does have a good idea of what tire size fits on an Audi A4 but it's not actually speculating It's not using that training data It's only based on the information that's in context via these sources That's what Rag is So as of the time of recording this there is no API to use Notebook LM because it has a really good Rag engine built into it If you want to use Rag for your AI agents or AI workflows then you'll need to find another solution So the most basic way to get this type of functionality in your AI agents or workflows is to use the likes of OpenAI's vector store and that's what we'll do now So if you go to the playground of OpenAI which I'll link below and if you click on dashboard and then assistance you can create a similar assistance to what we just did in Notebook LM except this time you can actually programmatically interact with this via an AI automation platform like N8N or make.com So if you click create on the top right and we'll call this one our F1 advisor give it some system instructions You are an F1 expert You are tasked with answering a question using the provided information and we're going to upload those documents Your goal is to provide an accurate answer based on this information If you cannot answer the question using the provided information say sorry I don't know This piece is absolutely crucial because if you don't put this in it's going to fall back on its training data and it'll speculate answers and this is where hallucinations kick in So if we click save to that So on the right hand side here enable file search and then click add files and here you're going to drag in those files So we'll do the same thing that we did last time So at the bottom right click attach and then that creates a vector store for this F1 advisor assistant and it uploads those three files to it At this point now we have our agent that we've provided instructions We can specify a model Let's go with GPT4.1 We've uploaded files to a vector store that's attached to the assistant And then on the bottom here for temperature let's decrease this to something like 0.4 And the reason for this is the lower the temperature is the less random it becomes and the more deterministic it becomes which is what you want in a rag agent You don't want it speculating or getting creative Okay So then that's all set up Click on playground then and that brings you into this assistant in the playground Similar again to what we did with notebook LM And now we can ask questions What is the plank assembly plank assembly is the component in F1's car often referred to as just the plank Now you'll see that that's a lot less of a detailed answer than what we got from notebook LM And this may be down to a number of things It could be the number of chunks that were returned from the vector store Could be different LLMs that are trained behind the scenes And that's a lot of what we get into with rag is fleshing out what's happening behind the scenes to get to the right type of outcome So let's now ask an irrelevant question What is the current weather in New York and you get sorry I don't know Perfect So it's not speculating So now let's ask our Audi A4 tire size question cuz this is somewhat related in the sense that it's a car and these are motorsport regulations and it comes back with sorry I don't know which is ideal This is our rag agent now set up There is a bit of a lack of transparency in the sense that we can't see the chunks that were retrieved to help generate this text nor do we have citations but at least we are getting a response that's somewhat accurate So now to bring this into an automation platform if you are brand new to NADN then to quickly get started with this masterass I do recommend signing up to a cloud account on their platform And then once you become familiar with NADN you could self-host it locally or on cloud infrastructure like Railway or LSTO And if you are following along with the build I do recommend pausing the video regularly so you can make sure your configuration is aligned to what I have in the video If we go to N8N and let's just kick off workflow So we just trigger it manually and then type in open AAI and then we're going to message an assistant as you can see there And from here then we can choose the assistant or F1 advisor and for memory let's set it to thread ID and we're getting an error actually So instead let's just use the memory connector and we'll manage the memory on the NAN side So we'll save that And now you can see memory here So we just click that and just use simple memory And let's set the context window length to say 20 So we'll remember the 20 most recent exchanges within this conversation The longer you set the memory the more expensive it gets because there's more information in context and sometimes it can kind of pollute the accuracy of the responses as well So we'll leave it at 20 So we'll save that and let's go again repost the message So it's gone off to the assistant and now you can see the response and that's a similar response to what we were getting in open eyes playground The plank assembly is fitted to the underside of the car and then the question of what's the weather in New York we're getting back sorry I don't know which is ideal So that's definitely the simplest way to set up a rag system in N8N where you can just drag and drop files and you don't need to worry about chunking or embedding The openi assistant is a type of agent in the sense that you have memory It has knowledge because we've uploaded files and you can also add tools So there's function calling or tool calling It is still in beta though and it's soon to be replaced by the responses API within OpenAI So it's not really used that much within N8N So while it is pretty easy to get a basic rag system set up simply by creating an assistant and dropping in some files and then just dropping in this node here and then you have a conversation with it It does have a lot of shortcomings and the best way to kind of explain this is if I ask this question here of what arrow elements are on the back wing you'll see that this is going to the assistant it's going to look into the files in the vector store and it's not going to find any good matches and it's going to return with sorry I don't know Whereas if I ask that same question in notebook LM I'm getting back a much more thorough answer And the key reason is this first sentence Based on the sources provided the backwing is referred to as the rear wing in the regulations So notebook LM was able to actually infer backwing equals rear wing and was able to search the vector store either for that term or it has a more sophisticated means of retrieving data So this does point to limitations within the OpenAI's vector store approach I'd recommend that you set this up yourself Create an assistant in the playground upload some files and then inn create a quick workflow to hit that assistant and ask it some questions on the data And what you'll find is it answers really well for certain types of questions Whereas for other types of questions it may say it doesn't know like it like we have here It may hallucinate an answer or it may give incomplete answers So in this lesson we'll be using NEDN's AI agent feature along with the superbase vector store to chat with these documents So let's remove out this message assistant And if we click on the plus and we can type in AI agent and from here then we'll leave the source with the prompt as is If you click on add option and go to system message and then let's copy in our system message that we used within the OpenAI assistant So you're an F1 expert you need to answer the questions and we provided an out So if the answer is not in context it can just say "Sorry I don't know." So let's connect up our chat interface And then under chat model we already have an OpenAI connection here from the previous lesson So click on OpenAI chat model and let's use 4.1 which is that one And similar to what we did with the OpenAI assistant let's reduce the temperature so that we get a more consistent and less creative response We'll set it to 04 Okay we'll save that In terms of memory we'll add the same kind of simple memory again Choose our context window length of 20 Okay Okay and let's test it out So we haven't loaded any documents yet So if we say what's the plank assembly it's not really going to know And interestingly it actually does provide an answer This is bad really but I think the reason is we haven't provided it a grounding so that it could actually follow the system message So here we're saying answered based off the provided information but we haven't given it any information So let's give it some information Let's refresh the memory and then click on the plus icon for tool And if you type in vector you'll see lots of different vector stores down the right hand side here So before we set up superbase let's just choose the simple vector store Now this isn't for production use because actually what's happening with the simple vector store is it's just saving the data in memory So the minute that your NAN instance reloads you're going to lose all the data But it's a good example just to actually test this out So we'll call this one our F1 regulations We'll give it a description because descriptions are very useful for agents to understand when and how to call tools This is a repository of F1 regulation Okay So let's save that And what you'll see then is you have this leg to set an embedding model And this is a key part of vector databases is that the information needs to be processed and vector embeddings need to be created that represent that information So while that sounds a bit complex if you just click on the plus and then you just need to specify an embedding model I'll explain it as we go So let's keep using OpenAI We'll use their embedding model as well And let's use their text embedding three small model We'll save that And now if we ask the same question again what is the plank assembly i'm hoping we don't get a response because it should go to the vector store which it did It's not going to get any results because there's nothing in there And we are actually getting a response So this is bad So this is where the prompt engineering side of Rag kicks in We need to convince the LLM not to respond if it's not getting any relevant information back from the Rag store And you can see in the logs here it did go to the vector store It queried it with plank assembly but it got nothing back which is a standard use case where there might not be anything relevant So it needs to be able to handle that and it needs to basically say "Sorry I don't know." So if you click on expression let's just open this up You are an F1 expert You are tasked with answering a question and let's be more specific here using the information retrieved from the attached vector store Your goal is to provide an accurate answer based on this information only If you cannot answer the question using the provided information or if no information is returned from the vector store say sorry I don't know Okay And let's test it out again So let's refresh the memory and there we go Sorry I don't know So that is the prompt engineering side of rag systems that you really need to work through and there are techniques that you can use to actually validate the responses from a rag system to catch out any hallucinations or any examples where it's generating content from its training data as opposed to what was provided to it with the grounding Okay so let's load up some information now into this vector store So let's make some space here And if we click the plus icon we'll add a trigger which will just be a manual trigger And then from here with the plus if you just type in simple vector store and what we want to do is add documents to this So just click on add documents And you'll see then there's various different operations that you can use So we'll specify insert documents And we're going to use the same memory key that we used previously which was already pre-selected And here you'll be specifying an embeddings batch size We'll just leave it at 200 as is Okay So save that And then similar to what we have to do down here which is to specify an embeddings model we need to do the same here as well So just click on plus back to open AI And it's crucial that you use the same model for both ingesting documents into your vector store and querying or retrieving documents from your vector store If there's a mismatch in the model or the dimensions of the model then you'll get some really inaccurate results So that's the model and we haven't specified dimensions This defaults to 1536 So that should be fine So there's that And then with this simple vector store if you click on the plus where it says document you can then load your PDF document So we'll use the default data loader which loads data from the previous step in the workflow So we just need to load the PDFs here So that's done there The type of data is JSON We'll load all the input data And we can also specify metadata as well but we'll leave that for the moment So let's save that and let's move things up a little bit So you'll see that this is still showing as red there there's an issue which is we need to provide it a text splitter So this gets to the chunking required of documents So if we're to take this document for example it's 120 pages long and there's just a mountain of information here And the way rag works is that it chunks the document or it splits it and it splits it based off a specific strategy So it could split it per sentence So if we look at this section here if you were splitting for sentence it's just looking for a full stop That's a long sentence there So that could be one chunk then this could be another chunk this could be another chunk etc Or I could split it per paragraph So this could be one chunk that would be one chunk that would be one chunk Or there's other strategies like recursive character splitting So based off a chunk size it'll work its way back and look for a delimiter So let's say if it was a thousand characters let's say that's a thousand characters it'll work its way back to maybe the delimiter of a new line which might be to there for example So there's lots of different strategies So here if we click on the plus we'll use that recursive character text splitter which is recommended for most use cases And for chunk size we'll set it to a,000 characters with an overlap of 200 And what it means is it's going to start this chunk 200 characters into the previous chunk So there's a little bit of an overlap between the chunks so that we can try to get more of the context of the information in the actual segment But let's say this is one chunk for example and then you have an overlap of 200 characters for the next chunk It's not like this would be the next chunk If this is the first chunk then the second chunk would start here and then it would be a th00and characters from there for example So this piece of text would appear in two chunks and that's how the overlap works Okay So let's save that Now this workflow won't yet work because we haven't loaded any PDF So let's just do that now Let's click plus And just to make this as simple as possible let's just load a single PDF And these are public documents So if I rightclick and copy the link address there's our PDF So I should be able to actually download this So we'll use a HTTP request and we'll just get that file So let's test that step And there's our binary Perfect We click view There's the PDF So if we come back into our data loader the type of data could be binary and that's what we're getting from the previous stage You can actually specify a data format there as well So you can specify PDF but it's going to pick it up there anyway You can see file extension is PDF Okay so let's test the workflow It's downloaded the data So it's loading that data chunk by chunk into the vector store and it's just finished And there's some really interesting things here So if we zoom in you'll see that this document is 179 pages long Here you'll see 683 items were actually processed So that's 683 chunks that that document was split into by this data loader using this recursive character text splitting strategy It went to OpenAI four times because if we click here you can see that the embedding batch size is 200 So it's passing chunks in batches of 200 to OpenAI to return vector embeddings And if you double click on the vector store itself and click on logs you can see those interactions So if you click on the first one this is the input And if you click on the output you can see the vectors that were returned by that service So all of these numbers essentially represent the text that was sent to it And if you click on the second one you can see that the text changes because these are different chunks that were sent in batches Okay so we now have data in our vector store So if we come back to our agent and if we ask the same question again we'll just clear the memory What is the plank assembly it now has data to base its answer off And there we go We're getting a decent answer And if you dive into the AI agent and click on the logs you can see it went to the chat model which then triggered the tool i.e the vector store It passed the query which was plank assembly which is really interesting because that's not what I asked I asked what is the plank assembly so it rewrote that query and it stripped away words that are somewhat irrelevant to the actual retrieval And that's a really good strategy that we're going to dive into a bit further in this course And then based on that query we're getting chunks in response And those chunks contain information about the plank As you can see there the geometry of the plank was conformed to RV plank with a tolerance of plus or minus.5 mil So those chunks are relevant to the query and we're only getting four of them And the reason for that is if you double click on this tool you can see that we've set a limit of four So we could increase that now to let's say 20 And if we ask the same question again now 20 chunks from the document are being returned to GPT4.1 for it to base its answer off And then you'll need to evaluate whether you're getting a more thorough answer because there's more information being retrieved from the vector store So how many chunks should you retrieve from the vector store is very subjective If you retrieve too many you're possibly adding noise to the LLM and it might not actually produce a great answer If you retrieve too few you're going to end up with partial answers to the question So it really does depend And then also the the newer LLM models that are on the market have longer context windows So you can load up more and more information into their context and they are getting better at actually reasoning around how to actually answer the question based off the information they were provided And back to my irrelevant questions what's the weather like in New York interestingly it didn't even go to the Vector store So obviously GPT 4.1 has made a determination that based off the system prompt you are an F1 expert Clearly what's the weather like in New York is totally irrelevant If I ask my Audi A4 question what size tire is fitted on an Audi A4 we're getting the same thing because the LLM knows there are no Audi A4s in Formula 1 Finally let's ask our other question What aero elements are on the back wing openai's assistant failed to actually answer this one So now it is going to the vector store It has deemed that this is relevant and we have got back an answer So clearly this is now better than OpenAI's assistant and we're getting quite a lot of detail back there And you can see on the right hand side here it it didn't substantially rewrite the query It's still arrow elements on the back wing but there's obviously enough semantic relationship between rear wing and backwing to actually pull this information So as we continue on in this course we break apart the rag process more and more In the beginner section we used OpenAI's assistant and we were totally removed from the embedding and the chunking of the data itself and how you query that data Whereas as you build it out like this you have more control you have more configurations around the number of chunks that are retrieved for example or when we were loading the data we have more control over the size of the chunks and the overlap between those chunks So of course our issue here is that we're using this simple vector store which is just loaded into memory on nadn and that will be purged when the nadn server reboots So we need a more persistent place to store all of our information and that's why if you come down to the agent and search for vector store you get all of these vector store options So you can store them with pine cone superbase hostg has a pg vector store quadrant is another one I've used a good bit So let's choose superbase and we'll get up and running with that So click on superbase and from here then we'll need to set credentials So we'll need to create a superbase account Go to superbase.com and if you have an account log in if you don't click start your project and create an account and this will bring you into your project dashboard where you can create a new project or go into an existing project And from here then click on SQL editor and you'll see this link for quick starts So just click that and there's an option for lang chain So this is a quick start script to get your vector database up and running So just click that and then you can see the run option on the right hand side here So if you click that and you can see success at this point Now our vector database is set up So if you click on database on the left hand side you can see you have a documents table It has an ID It has content which is the text of a chunk It has associated metadata and then the actual vector embedding itself And just a point of note this quick start script creates a vector database with 1536 dimensions I'll get into that in a bit more detail in future lessons because you might need a vector database with a different number of dimensions So we have our vector database set up And if you click this button here it'll load up the table view And clearly there's nothing actually available yet but we can now start populating this So the next thing we need is an API key so that we can create our connection on N8N So if you go back to the project overview scroll down and you can see connecting to your new project So if you click on view API settings you have your project API keys here So back into N click create new credential And the two fields you need are host and your service ro secret So you can see the host up here We'll copy that and you paste it in there And service ro secret secret is this one So you can reveal and copy and drop it in there and then you click save and you are now successfully connected to your superbase account So you can close the credential And here we're going to be retrieving documents as a tool for the agent So that's fine Under table name we can choose the documents table that we just created in our vector database And the name for this tool let's set it as F1 knowledge base so that the agent knows what it is that it's calling And we'll then give this tool a name and description so that the agent knows how and when to call it So it's an F1 knowledge base Retrieve relevant details on formula 1 with this tool And then limit We saw this before with the simple vector store We can set this to 20 as well And we will include metadata in our response because then we can provide that to the agent to help ground the response Okay let's leave it at that and save And then let's move this down here We'll get our simple vector store out of the way and we will hook up the same embedding model And this is the dimensions that I mentioned This text embeddings three small model defaults to 1536 dimensions And that's what we have on the superbase side So it's important that they align So right now we don't have anything on our database as you can see here So let's quickly test the agent just to make sure that we're not getting anything back So we'll ask what arrow elements are on the wing and it has gone to superbase It won't have returned anything and it says sorry I don't know And if you open up here and click superbase you can see that there is nothing in the response So that's perfect So let's now adjust our ingestion workflow to tie in superbase So we come up here and let's unhook our simple vector store and get it out of the way And now we're going to go to our superbase vector store which is there and we're going to add documents to it So our credential will be auto selected We are inserting documents We can select our documents table And we can leave our batch size again at 200 And we now need to provide our embedding model as well as the document itself So move those over here That way we'll still have our text splitter set as well Now if we run this flow it's going to download the F1 regulations and it's now upserting the chunks to that superbase vector store And as you can see it's going in batches to the embedding model to create those vectors And if we drop in here to Superbase that has auto refreshed and you can now see the various embeddings and the content in the vector database And that has finished So 683 chunks for this document And this is really what's happening behind the scenes So you have content here So this is a chunk of around a,000 characters And that's essentially from the first page of the PDF We have metadata a lot of which is actually set from the PDF itself A lot of it is actually irrelevant really but we can tidy that up again And here we have our embeddings So these are the numerical representations of this content that you see here The concept of metadata is super powerful in rag because it allows you to actually prefilter your rag collections before sending in queries So it means that you can get much more accurate results if you use metadata filters correctly As I mentioned a lot of this is irrelevant So let's make a change to this flow now And instead of sending in all of the metadata that's associated with the PDF let's be more descriptive about what metadata we want to send So if you double click on this if you go to JSON you'll see it's empty but that is because if you go to add option and then response you just need to set include response headers So if you click that and click test step you'll see now that we're still getting the binary data but if you click JSON these are all of the document headers that are associated with that document that are then being sent to the vector store as the metadata for these vectors and a lot of it is just totally irrelevant So instead of passing the binary of the file which is what we're doing here if you look at default data loader we're loading the binary of the PDF Let's now make some space and we'll extract the text from the PDF So we'll click on plus and if you just type in extract you'll see extract from file and then there's extract from PDF and then this will automatically select data as the binary field which you can see there and that's it So if you click test step you'll see now that it's outputting all of the metadata again but we are getting this text parameter So now we can just pass the text into the vector store and not all of the noise of the metadata that's coming from the PDFs themselves So here what we might do is if we just add a new node we'll use the edit fields or the set fields and we'll just create a manual mapping We'll just call this one content and we can now drag in that text parameter So that'll be loaded now into this field and it'll just be this field that's indexed Okay so let's now run it again And while that actually does look successful I forgot to change the type of data So we're not getting binary now We're actually specifically grabbing this content field So we can leave it as load all input data So let's try that again Yep And it's generating the embeddings in batches And if we sort by ID descending we'll be able to see the latest ones And here we have our new content which you can see there And you can also see that the metadata is totally cleaned up now We're not getting any of that PDF noise Great So the issue we now have is that we have upserted this PDF document twice to this vector store and the actual document chunks have been duplicated So you can see with the ID we've 1300 records but there's actually 670 odd chunks in the document So this same document was uploaded twice in the vector store and has duplicated it So this is a big problem We need to make sure that we only have one version of a document in a vector store Otherwise when it comes to retrieve it we might get very poor results because we're just returning back the same chunk of a document 10 times Nevertheless if we test this out now with our AI agent so let's ask the question again What arrow elements are on the wing now that we do have information in our superbase vector store we should be getting chunks back that we can base an answer off And there we go We are getting information back And if we drill down into what was returned from the superbase vector store you can see the various chunks And to that point about duplicate chunks you can see here that this chunk is the same as this chunk So in the next lesson we're going to create a data ingestion pipeline that handles this problem of duplicate documents to keep your vector store as clean as possible In this lesson we'll be building out a data ingestion pipeline so that we can load multiple documents into our vector store And we'll also be building a record manager to keep track of these documents to avoid situations where we are upserting duplicate documents And what we went through up here is this ingestion flow that we're going to build out further Whereas this is the inference flow where we're actually querying the database through an AI agent chat interface To begin I've created a Google Drive folder called rag files So the idea here is we can drop files into this folder and they'll automatically be processed by the automation And here I've dropped in a snippet of those F1 regulations So this document is eight pages long instead of 170 That way we can test it out with a smaller data set And once it's up and running we can load lots of documents into this So here then let's remove this manual trigger and instead let's click on plus We'll go to Google Drive and at the bottom where it says triggers we'll click on the on changes involving a specific folder From here you'll need to set your credentials So for that you need to click on create new credential and then click open docs I won't go through it in this video but I do recommend that you carefully follow this step by step because there are five stages to it and if you skip any substep within this then you might run into problems Now with my credentials set up I can choose the folder rag files that I created for this And then you can watch for different events So I'm going to watch for when a file is created And as you can see it says changes within subfolders won't trigger the node That's worth noting Okay So now let's click fetch test event And you can see that the F1 regulations have been returned So that's great So we now have a file that we can play with And as you can see we're not getting the binary We're getting the information about the file So the next thing we need to do is actually download this file So back to Google Drive download file And where it says file here we can set expression and we can drop in the ID from the trigger which is this one here And then if we click test step we should now get binary which we do So there's our PDF And now we're essentially in the same position we were here So let's remove now this manual download of the file cuz we have it here And we can hook it up to our extraction So now we're extracting text from the PDF and then we're mapping the text from the PDF to this content field which is then upserted to the Superbase vector store So let's test this workflow So we've downloaded the file extracted the text and it's going up to superbase and this time we have 30 items because this is a smaller version of the document So we still have our problem of duplicate documents If I run this again I'm just going to add another 30 vectors to this vector database So we need now to implement a record manager where we can track what's in the vector database and see if anything has changed and if it has we can delete the vectors that were related to that document previously and upsert new ones So if we come into superbase then if you go to database and click on tables let's create a new table and for this record manager let's create two fields The first one is going to be Google Drive file ID and we can set that as text And then the second one is going to be our hash which we'll also set as text And that has now created our record manager And actually we want both of these columns to be required So if we click edit fields let's set the allowed nullable to false All right So back into our workflow So if we make some space here what we want to do now is search this record manager on superbase to see has the file already been upserted to the vector database We also need to implement a mechanism that will help us determine whether something has changed in the file or not if it already exists in the database So let's carry out the search first So if we click add and type in superbase we want to get many rows and we can choose our table name which is our record manager We can set the limit to one because a document should not appear more than once in this record manager And then for filters if you click on add condition we want to say if the Google Drive file ID equals the ID of the file from the trigger itself So that's that one there Okay So if we save that and just run this one in isolation you can see no output data returned This happens sometimes in N8N And what you need to do is specify always output data otherwise the flow can stop at this point And if we click test step again you'll see now that we're getting an empty array which is fine So firstly let's put in a condition that if the document doesn't exist in the record manager we'll just add it to it So if we click on plus again back to superbase we're going to create a row and we can specify our record manager and let's define the columns So the ID is autogenerated as is the created date So we just need to drop in the Google Drive file ID which is this one and the hash which we need to actually generate So we get out of that and if we drop in the hash here So let's click on plus and N has a crypto node So if we just type in crypto you'll see this one and we can essentially create a hash or a unique kind of fingerprint of the actual file itself So under type let's use SHA 256 as an algorithm We can select binary as yes and it's going to basically pick up the file from Google Drive and it's going to create a hex hash So if we click test step and then scroll to the bottom you can see where it says data this is the hash So actually let's just rename this hash and test it again now So there we go So this is what we're going to save to superbase So let's rename some things here So this is generate the unique hash This is get file So that's download file This is search record manager and this is create row in record manager Okay So back into our create row and we now can drop in this unique hash this unique fingerprint which is this one here and we've already set the file ID So if we click test step here So we're getting an error here on the search record manager and it's that it doesn't know which file ID to actually inject into this field to actually search superbase And the reason for that is this trigger can actually return multiple files because you know five or six different files may be uploaded at the same time to the folder So this would return five or six different items So what we need to do is we need to use a loop over items so that we can process each file one by one So if you click on add and just type in loop you'll get your loop over items And we'll set the batch size to one because we want to process one file at a time So that's there And we need to tidy this up a little bit because we want these as part of the loop So we'll just delete this replace me Delete this because this is the done channel as opposed to the loop channel And then we can tie that in there So for each file that has been provided by this trigger we're going to download it And now once we get to the end of the processing of this one file we can loop back to the next file So that's what we'll do there And then back into the search record manager we can now pick up the ID from the loop node and it'll know which one we're talking about So scroll down to there You can see that's where we're getting the error But now you can see the loop node is there And we just need to click execute previous nodes to actually load that Let's just click test workflow and we'll get it going So back into this So now we can pick up the ID from that loop node which is there And there's our ID And now you can see it's green because it now knows which file ID to actually inject into that And also now we have our record created in our record manager So let's open up Superbase again and let's jump into our record manager And you can see there's our file ID and we have our hash So we can now start putting in some conditions because just to look at what we've done so far we have a Google Drive trigger which could provide us a number of files We then loop through each file We download it We generate a unique hash off that file and then we search our record manager to see has that file already been upserted And if you click test step here you'll see that it has We're getting a file ID back and we're getting a hash So now we can put in the conditions and build out some more logic So let's use a switch because there's a few scenarios we need to take account of here So we'll click add and we'll go to switch So the first scenario is if the document doesn't exist in the record manager It means we've never upserted it before We've never seen it before So for that one if you go to expression you can see here we're just getting nothing So open up some curly brackets and it's just JSON And as you can see we're getting an empty object So then if you come down here to object so if this is empty it basically means we've never seen it before and we can go and create the row and upsert the document The second route is if the document is in our record manager but the hashes are the same That means there's nothing to do The document hasn't changed So for that one let's come back in here If we click test step and load up a record where the hash actually appears as you can see there and back into switch and for value Then what we're saying is if the hash from the record manager is equal to the hash that we generated as part of this workflow which is that one then in this scenario nothing has changed So we get back out of that here We can just loop this back We can move on to the next file There's nothing else to do So the third scenario then is if the hash from the record manager is not equal to the hash that we generated it means the file has changed and in this instance we then need to go to our vector store delete out all the old vectors and insert new ones From experience I find this is probably the safest approach because if you try to overwrite vectors in the database you may end up with orphan chunks which are then difficult to clean out And before we build out this logic we're missing one key aspect which is within the metadata of the document itself As you can see here we need to store the file ID So currently the file ID is not there We need to be able to search for all the vectors based on that file ID and bulk delete them So to do that then let's just trigger the workflow again And we are getting an error actually because we need binary to extract the text from the files So we'll just duplicate that node and drop it back there And we need to update the ID as well So we'll get that from the loop Okay that's worked now So then we need to come into the default data loader And as you can see it's just loading the content into the vector store So we need to add metadata So go to metadata We'll add a property And this will be file ID And then we can grab the value again from the loop So mapping loop over items And there's our ID And now we run it That's upserting again to superbase And here's our new vectors And if we click on metadata you can now see the file ID So now we can come into this route where we're actually able to search for those vectors based off that file ID So here we're not using the superbase vector store node We're using the superbase node and get many rows And we can choose our documents vector store We can set return all based on this filter And then we can add the condition here So what we're saying is if the metadata is and we'll use this I like operator where you actually use these asterisks and then we can drop in the file ID here So we'll pick that up from the loop again which is that one So this will now return all vectors in the database that are associated to this file ID So we now need to just simulate the data so it follows this route So if I just go into the record manager I'll delete them all except one and let's just make a change to the hash So that way it'll think that it's actually different Great We now only have a single record where the hash is a bit different Now if we click test workflow it'll now flow down this track And we're getting an error but that's fine So let's double click it And this error that we're getting is because this condition the I like operator can't be used for a JSON data structure And you can kind of see it there under metadata It says undefined So instead of building manually we might just use string and we can just build up our own filter Okay And there we have it So when the metadata file ID so that's the JSON syntax is equal to and then it's a like operator two wild cards and then it's the file ID So let's test this one And there we go We're getting back 30 items So we now need to bulk delete everything here And actually we could probably do it um just by changing this to delete So it's delete our row but I think we can pass the same filter So it should delete everything Let's try it There you go I think it deleted 30 records Yep the record count is actually reduced Excellent So we can rename this one to delete previous vectors And in this scenario we don't want to create the row in the record manager We want to update the row so that it has the new hash value We click on plus and go superbase This time we're going to update a row And this is the ID from the search record manager that we got here So we can choose record manager as the table And then the select condition is where id equals id Okay And the data to send then is we are just sending the new hash So hash is there and we can pick it up from our crypto node here All right So we'll save that So this is update our record manager And now we can come back in here because then we're just going back through the process of downloading the file extracting the text and then upserting the the data So let's save this And what we'll do is let's delete out everything from our documents now because there's so many duplicate documents here And we'll also delete out everything from our record manager Our vector store and record manager is empty Let's now test this end to end So it's gone up here cuz it doesn't exist in the record manager yet It's downloaded the file and let's see what it's done So click on record manager and there's our line and if we click on documents we have 30 records Excellent And now if we click this again to process the same file without any changes Brilliant It's going down this loop and it's just moving on to the next file And then if we simulate a change by going to the record manager and just changing the hash value a little bit and press test workflow we should come down this track which we do We delete out the old vectors and we upsert the new vectors And if we keep refreshing it'll fill back up again Oh I better stop this Actually there is one flaw in the logic which is we had 30 items here that we deleted but we're actually done with them at this point Interesting to see that they have been returned I'm surprised we're actually getting all of the vectors that have just been deleted but anyhow yeah let's just aggregate off the back of this So use an aggregator We'll aggregate all of the items into a single list We're going to put it into a data array for example So if we click test step and yeah we're back to a single item then That's exactly what we wanted So now if we look at our record manager we should have a single row with the hash That's correct And if we look at the documents we have 30 records That's also correct And just one final check Just test it again and it should loop back Brilliant So that is how you handle the ingestion of multiple PDF files from a Google Drive folder while also ensuring that you're not duplicating documents and chunks in your Superbase vector store Let's test this out now So let's remove the records that we have in our vector database And if we go to our Google Drive folder let's get rid of this test file and let's upload the three F1 regulation files that we tested earlier in the course So they are now here Okay So now let's activate the scenario cuz what that does is it'll then pull the Google Drive folder every minute and look for files to process So we've activated the scenario and then if we come into executions we'll see the actual run when the polling happens Once the workflow is executed based off that polling it'll show up here as running Okay So that's now running as you can see there Now it's not possible to actually see the progress of the workflow until it has finished And if we click into documents you can see the vectors have started to flow through There's our second document We're still running and the total at the bottom here We have 383 records so far up to 700 And now the third document has appeared as well Okay so it's finished It's processed those three documents in 44 seconds And they are large documents And you can see the full workflow from start to finish And now if we test this out with our chat interface we'll just ask again what is the plank assembly and we quickly get our responses back And it's a pretty decent answer And by clicking on the superbase vector store in the logs you can see the specific chunks and the metadata associated with those chunks Now there's not a huge amount of metadata so far The main one really is the file ID It is also providing the lines So where the chunk was actually created So let's now jump into metadata in a bit more detail because it is hugely powerful in providing more accurate results Let's go back to the FIA's website So Formula 1 is just one of the sports that it governs It also governs other sports like the World Rally Championship the World Endurance Championship So we take the World Rally Championship for example these also have sporting regulations technical regulations safety regulations etc So if we were to index all of these documents alongside the Formula 1 documents if you were to ask a question like "What is the plank assembly?" Well then are you asking it about a Formula 1 car or a rally car and this is really where metadata can be very powerful So let's grab the sporting regulations and the technical regulations for the World Endurance Championship And before we drop these into the folder to process let's have a think about what type of metadata we should save here So if we come back into NAN back into our vector store and let's look at the default data loader because this is where we specify the metadata So I think what might be of interest here is motorsport category So for each document that we're uploading let's define what actual category it is Is it rally is it F1 is it endurance etc The other thing is we're setting a file ID but we haven't set the file name yet So we'll put file name there And another trick that we could do is we could save a very short file summary And this can help the LLM when it's generating answers to questions rule in or rule out certain chunks even if they are returned by the vector retrieval system So we have three additional metadata fields there So let's now populate them So I could click test workflow and that way I can load up all of the variables for all of the nodes so that I can build this out But a really neat trick in Naden is if you click on executions and if you go to a previous execution like the one that imported those three files you can then click on this copy to editor button And now if I double click this we have all of the execution data from that run And you can see three items were processed Okay Okay so back to our default data loader If we wanted to bring in the file name we just need to go to the loop which is here loop over items and there is the file name So obviously a lot of clues in that as to what sport this is in relation to for motorsport category and file summary We'll go to an LLM and actually ask it to generate that data And we do only need to do this once per document You don't need to do it for every chunk So if we push this out here and let's click on plus and we can use a basic LLM chain here and the source for the prompt we're going to actually define because this is going to be the file name as well as the text of the file So we can already drop this in here We'll set this to expression markdown is always useful here So we'll say file name file contents and there is the file contents and there is the name So that's what we're passing to the LLM Now we do need to be careful because the actual file contents might be too large for the context window length of an LLM and it might produce an error So instead of sending the entire document you could send the first 100 words or 500 words So I've just asked chatbt to generate a JavaScript expression for me that just sends the first 500 words So if we drop that in there and you can see now that it's cut after the 500 words So that should speed things up as well Particularly when we're processing a lot of files we want to keep the bills down on the LLM calls So the system message here then is based on the provided file name and file contents extract out a one-s sentence description of what the document is about and classify the document based on motorsport category only output JSON in the following format We're going to set this both in the system message as well as an output parser attached to the chain and I'll show you that in a second So we want a JSON response motorsport category and document summary And we'll just put ads there for the LLM to actually add the data And as always with an LLM when you're extracting data you need to give it an out so it doesn't hallucinate things if the data is not in context So if you are unsure of either of these just output NA in the field Allowable values for motorsport category include F1 rally world endurance and let's leave it at that So usually that is enough to actually convince the LLM to output in that format But within N8N you do have other options and this one is the require specific output format which I have found is pretty useful So if you click on that and then you get this pop-up to say connect an output parser So if you click on that then you have this structured output parser and that's essentially what we've already included in the system message We probably don't need it in both but it can't do any harm So if we just paste that in there and then that's all we need So for model let's use GPT4.1 cuz that's what we've been using to date Now you could use the likes of GBT 4.1 mini or even 4.1 Nano if you were looking to save on budget For the moment I'll just choose the main model And then response format we'll set as JSON Okay So we'll save that And to test this we're just going to pin some of the data so that we're able to force it down this flow Otherwise we'll have to keep messing with the hashes or keep removing things from the record manager So we'll get all the way to here and now let's see can we test this Okay So that is running There we go So we have our motorsport category F1 The document then is outlines the financial regulations including the cost cap And that is correct This these are the financial regulations Okay So we now have some metadata that we can now associate with each of the chunks in the vector store So we bring this over here Jump into your default data loader And then for motorsport category you can just grab it now from the output of the basic LLM chain There's F1 And here is our file summary Great So now let's save that Now we are going to need to reprocess all of our chunks because we've created this new metadata but it's not associated with everything that's already loaded into the vector database So we need to clean everything down and reimpport everything again So I've manually deleted everything again from our vector store and our record manager And now if we go back into Google Drive and now let's add in our regulations from the World Endurance Championship which we downloaded And that's running and it's finished We have hit an error actually because this isn't outputting any data for some reason Of course this edit field is probably pulling text from the node behind it And since we've added this node it's actually broken it So back into there So for this we just need to drag in the text from this node and it'll have a hard reference then to the name of the node as opposed to just dollar sign JSON That's just how NN works Dollar sign JSON refers to the previous node A hard reference to the actual node name can reference any node Our files have now been imported We have 1,800 chunks across five documents So now if we come into our AI agent and let's ask it a question where there's a commonality between both categories of sport What are the rules around a safety car so there will be different rules for the safety car in F1 versus the endurance championship So here are the main rules around a safety car in Formula 1 and similar FIA regulated championships And if we go into superbase you can see that we have the file name now associated with the chunk as well as the file summary along with the motorsport category as well So now if we come down to our AI agent I've entered motorsport category as the filter name And then ideally we're going to get the AI to dynamically inject either F1 or world endurance here Now the issue with that is if I use this syntax which is used regularly for AI agents So it's from AI and you're getting it to actually populate this out Unfortunately that's not working for me at the moment So if I refresh that I'm getting an error that I can't get around That does look like a bug So I'll report it to the NAN team But just to get around it for the moment I'll remove that from here And I'll just put in an LLM chain before it But once that bug is fixed it should be able to dynamically generate it within the AI agent itself So we're using the chat input as the user message And here we're just saying based on the context reply with either F1 or World Endurance And as usual only output in JSON And let's attach an output parser similar to what we did before Motorsport category Add And we'll copy in the model as well GPG 4.1 And you could definitely use a smaller model for something like that So let's just use 4.1 mini Okay And then if we just run that one on its own we should get an output which is F1 as you can see there And let's now just drop that in here There we go And now let's test this end to end Okay so we've gone to the vector store This time we're passing a metadata filter So we should only be getting chunks back where motorsport category is set to F1 And now that's a much better response And to verify that we can click on the agent click on the vector store and you can just work through and see the various chunks They all have F1 So let's ask the question when is the safety car deployed in F1 and then this node is extracting the category that's gone to the vector store to fetch relevant chunks And then we build out an answer off the back of that based on the information available Safety card is deployed during the race or formation lap lists the exact article from the regulations So that's a great example of metadata filtering You can add multiple filters and build up complex logic But this interface only supports kind of basic metadata filtering So if you wanted to get quite complex with it you probably would need to hit the super base vector to store via API and build out a really dynamic body for that call But what you can see there is pretty useful And to give you an example of the types of metadata filters that you could set you could for example categorize documents based off a department within a business Maybe you could filter or categorize by document type So are they reports are they invoices are they emails are they web pages you could categorize by author So if you wanted to search a particular person's body of work you could do it that way You could get an LLM to generate keywords or tags So it's not a structured category list but it could just be a cloud of tags that you could filter off So there's a lot of flexibility here with metadata filters And then the other one would be date created or date modified Now vector stores aren't structured databases You can't sort by the most recent but there are some parameters that you can use to help you filter out old results for example Next up let's expand our rag pipeline to handle different types of files and formats So if we come into our folder here on Google Drive you can see we currently only have PDFs And when you click fetch test event within N8N for this Google Drive trigger it explains it at the bottom here while building your flow When you click fetch it fetches a single file So to handle lots of different file formats let's remove all of these from this folder So I'll drop it up to the folder above And now let's create a new type of file So let's try a Google doc for example So test Google doc And let's keep going with our FIA example So we'll name this organization and let's just copy out everything that we have here Okay So now if we come back in here and click test workflow So it has followed down this track but we need to now handle this different type of file format because if you look at the download file you'll see the file extension is DOCX but we're getting a different MIME type The other thing is if I test this a few different times we're getting different hash values So this one starts with E96 This one starts with 645 even though nothing's actually changing in the file So I think we're going to have to modify the logic here and generate the hash off the text that's in the file And that way then we can base the logic on if the text has changed within this document Okay So let's zoom in and we'll make some modifications So let's move our download file over a little bit and let's put in our switch now So we can base the logic off the file formats So here's switch So here we can say if the mime type equals this then it's a Google doc So this is our Google doc branch Whereas if we trigger the PDF instead so let's get that one out of there Drop in a PDF and trigger it And now if we click it Yeah but it's the same It's mime type as well And that's application-p Okay So the second route here is going to be our PDF route And this route is going to be our Google Docs route And the other thing that's interesting about these switch nodes is that they actually pass the binary through So you can see the binary is coming through from the download file in the previous node and then it's been passed through to the next node here Not all nodes in it work like that But that's quite useful now because what we can do is we can grab our extract from file and maybe just change around the logic a little bit here and bring that closer to the front So the first thing that happens after this switch is we extract the data and then we can generate hashes off the back of the data consistently for different file formats So there's extract from PDF Now we also need an extract from Google Docs So for this we need to add our Google Docs get document and here we just need to pass in the doc ID or the URL which is what we're getting in the previous node So you just need to find the ID which is that one and then if you click test step on the right hand side you now have the text from that document Now there is this simplify button So actually if you turn that off and test it again yeah you're getting a lot more information So yeah we might leave it as simplify So that's the text from there and from here now we can actually generate the hash So we can hook both of these up to the generate hash So in the last lesson we were generating that unique fingerprint or that unique hash off the binary of the PDF file Whereas now we're just going to generate it off the text from the PDF file And with the Google Docs it's returning a content field Whereas with extract from file I believe it extracts a data field So we just need to change this generate hash So it's no longer taking a binary file And then we can just put in an or operator So it's either going to be this JSON.content and then doublepipe or it'll be JSON.data or it'll be JSON.data which will be coming from this So the rest can stay as is We're searching for the file ID If it doesn't exist we create the row So this logic around the record manager can stay as is because it's only adding the file ID and the hash that's generated here However when we get to here we no longer need to download the file So maybe what we'll do is if you click plus let's add an edit fields or a set fields and let's just call this our text and then this can be the long expression that can pull the text from the different types of files So that's there So this text will be of type string and then that will be that expression and we'll rename this so set text and then this can feed into there and this can feed into there And now instead of having the expression there we can just get the text from the previous node which will be JSON.ext because that's what's here text So let's save this and we'll just reorganize it So this is now the flow We're setting the text generating the hash then working through the record manager And now we no longer need to download this file So we can just go straight into this LLM chain to generate our metadata And if we pull this over here we now need to just remap these variables in the prompt So to do that let's save it and let's actually just run this so that we can load everything into memory Okay it's gone to Google Docs It's now set the text It's going through the record manager It's actually ending here And I think that's a bug and it's probably because it's not outputting anything Yeah no output data returned So generally you you always need a node to output data even if it is nothing because then you can handle that otherwise it just stops it dead as it's done there So you can handle that by saying always output data Now if we test it again There we go We've gone all the way through And that's just an empty array then at this point Okay So we're hitting an error there but that's fine If we come into here yeah this is what we need to remap because this is now what we're going to get from that new set text node we created So there it is Set text So instead of JSON.ext we can delete that and drop it in there So it needs to be updated there and it needs to be updated there Brilliant What you'll find when you're testing this is you'll need to keep coming in and just deleting stuff out of the record manager because this safety net is actually stopping you getting further in the chain if it's already been created So I've just deleted it there again so we can finally test and verify this one Okay And then back in Yeah now that is working It's all green and we are getting a good summary So this node is essentially priming the text that's to be sent into the vector store So let's just rename that setup text for embedding And then if we click into it now we can just pull that from the set text node that we created earlier That's this one And you can see it's green And that's the full text that's coming from the file that's to be chunked and embedded Great So save it And again one last test Yeah And that's working Brilliant And there's our record And that's this document now embedded effectively And in the data loader you can see the various chunks So with this new structure now you can drop in any type of file format So let's say it was a HTML page for example So if we just save this as a HTML and now let's drop this HTML into this folder just so that we can actually simulate this file format And you can see that's the full HTML data and this would be very useful for web scraping for example which we'll get into in a little bit more detail later in the course So if we come in here and if we now test this we're going to be picking up that HTML file as you can see there And now we're hitting a bottleneck here because we haven't actually a root set for a HTML file So let's add that So now mime type is text HTML That's what this additional route now is So now let's extract from this file So extract from file and we have an extract from HTML option If we test that step we are getting an invalid HTML Let's try it with extract from text file That way we don't need to worry about whether the HTML is valid or not And there we go That's everything that we're getting And the issue with what we're getting there is there's just so much noise None of that is really relevant for a kind of a retrieval system So best practice here is to actually convert this to markdown So if you click on plus and just type in markdown there's the HTML And we're going to output data And actually let's change it now to content And then we can hook this up here And that is looking for JSON.content If we click on run here it's now saved in this text field and the rest of the flow will work as expected Let's try another common use case here So we have the option of extracting from a PDF and that's what you saw here The difference with these PDFs is the text is actually editable or it's selectable So by its very nature it's machine readable and that's what we've been indexing in the vector store But what about the scenario of scanned pages like this one so this is a Formula 1 document which is a steward's decision on a race and it's all just being scanned up So none of it is selectable There's bits of handwriting in here So how do we actually process this and index it in the vector store well let's take this document and let's drop it into our rag folder which is what we've done here And now let's ingest this file and let's see how much data we can extract from it So click test workflow So if we come down this track now it is saying that it's extracted the text But if we look at the set text we're just getting lots of new lines There's no actual information here So to actually read this document we need to use some sort of OCR system and MRL OCR which was released in March of this year is one of the world's best OCRs as it claims here So let's use this API to actually extract the text from this document So once you have your Mistl account set up you go to API keys and you create a key and then you copy that key into the Mistral connection So let's just unhook this for the moment and let's create a HTTP request because there is no native node in N8N to hit this Mistral OCR So we just need to use the standard request nodes So we're going to post to this endpoint This is all documented in Mistl's API documentation For authentication click on predefined credential type and just type in Mistl and then create a new credential and you can paste in your API key and you'll get a green success Now you do need to add a minimum of $10 to your MR account for the API to be active So that's set here And the first thing we need to do is we need to upload our PDF file for it to be actually OCR So we'll set this as upload file And then we're going to send the body And the body is of type form data And the first thing we're going to do is set the binary that we're getting from the switch node So the name here is file And the input data field is data from the switch node And the second thing we need to set is the purpose of this upload which is OCR And that's it So if we save that this is our node So hook it up here And now let's test it Great So that's gone And it's uploaded the file As you can see there that's the data that was sent And we're getting back a file ID So next up we need to get a signed URL so that we can trigger the actual OCR So it's another HTTP node and we're going to post to this endpoint where we need to add in the file ID that we just received from the previous node So that goes in here and essentially we're retrieving a URL that we can then pass to Mistl in the next node to trigger the OCR And again we'll select our Mistral credentials and actually sorry this is a get not a post where we send in query parameters So we send in the expiration We'll set that as 24 and we'll send in the header that we want to receive JSON back So we'll accept application-json Okay So if we click test step we should now receive our signed URL which is what we did here So now in the next node we can pass in that URL for the file to be OCR So we're going to post to their OCR endpoint as you can see there Again we're passing in our credentials and we'll send this body and this was pulled straight from the mist documentation So change it to expression and let's open it up So what we're saying is we're going to hit the model OCR latest We're passing in a document URL and this is where we need to get our signed URL So we just drop that in there Okay So let's save that and test it again And now it's working through those 11 pages And if we double click there we go Every page has markdown So let's pin this data so that we don't need to keep triggering this OCR function So just select the node and press P So those three are now pinned So we now need to aggregate all of these markdown fields into a single field that we can just drop into the set text So NN does have a number of aggregation nodes that you can use There's the summarize where you can concatenate there's the aggregate there's the merge But here we have a relatively complex data structure So we have an array which has a number of objects and within that there are fields So essentially we just need to map this So instead of using the native modules what I'll do is if you press on the edit icon I'm just going to copy this out and bring it into chat to generate a JavaScript expression for me So here you go Can you give me a JavaScript expression to aggregate all of the markdown fields into a single text string and here's the response So this is the expression So now let's bring this into a new set fields And we'll call this one content because again that'll be picked up in the next node and we'll set the expression So open up a couple of curly brackets and we'll paste it in Now we are getting undefined but that's because data doesn't exist So within edit end everything starts with dollar sign JSON So there's that And then if we just remove all of this There we go So that is all of the markdown fields aggregated into a single field Okay So now that would then be connected up to the set text and away you go Now the only thing is here we're OCRing the PDFs Here we're extracting data from the PDFs So we need some logic to determine when do we need to OCR versus when can we just extract the text So maybe the first thing we'll do is we'll extract the text from the file Let's cut that link And if we test it we are getting lots of new lines back but there's no text So let's strip out all of the new lines or carriage returns or whites space and then count the length of this text and if it's say less than 10 characters then we can go and OCR it So we'll add a note again This can just be another set fields and we'll take in the text and actually it's already removed all of the fields So we might not really need to do much here If we just type in length yeah we are getting 20 So now we do definitely need to remove the new lines and the white space So we can use the replace function which essentially then replaces these new lines or these white spaces with nothing So it just deletes them So by adding that in we can now see we're getting empty back which is ideal So let's rename this strip out new lines and whites space We'll make a bit of space here ourselves And now we can have an if node Now let's just execute the previous node And then if we take this text so if the text or more importantly if the length of the text and let's say if that's less than maybe 10 characters just in case there's other non-printable characters that we didn't actually catch in the replace function before We'll say yeah if that's less than 10 or more so if that's greater than 10 then we can just keep going in this direction Whereas if it's less than 10 we need to come down and actually OCR the file and then bring that up to there So this is looking a bit confusing So let's actually make some space and reorganize this So there's our Google Doc branch This is the PDF branch and this is our OCR branch That all of that is combined into there So that's effectively what we need Yeah And that's going to pass the text That's ideal because that's what's needed here JSON.ext Okay So I think we're at a point where we can now test this So let's give it a go Test workflow Extracting the file Stripping out the white space And because there's nothing in it it then goes and gets OCR The markdown is aggregated It's set in this node here as you can see there And then it works its way through here and it's indexed into the vector store Excellent So let's unpin these now And now let's test this with a PDF that is machine readable So we get that out of there and bring back in this one And if we click test workflow it should float through the true route here which it does Excellent and it goes up to the vector store So that's how you OCR documents that are not machine readable and then you're able to actually send that text into the vector store so that it's searchable Okay and there's our little explainer and then that can be built out further as you add more file formats But the important thing is that however you're extracting or processing the text you always need to output either a field called content or a field called text for it to be picked up here So our data ingestion flow is definitely getting more complex but it's more featurerich now that it can handle multiple types of files And then with our record manager router we can manage the tracking of the files to make sure we're not duplicating data in our vector store Which then actually brings me to this trigger because this trigger is for new files But actually we have another trigger which is updated files Google Docs can actually be updated in line Whereas if it was a PDF it would end up with a different ID So it essentially would be a different file So we drag this in here and this can be our trigger for updated files And then the same thing happens It's just passing the file IDs to be processed Now the only thing we can't handle here is deleted files Obviously I've been moving files out of this folder but I don't believe there's an event for that here Let me see Yeah So under the watch 4 there is no file deletion So that is a shortcoming of this design so far We need to figure out a method of a soft delete that you could then pass into the system Perhaps we could have a deleted folder So we create a new folder here called a recycling bin And then if we copy this trigger and we can rename it delete files And now we're going to watch for files that are added to this folder or recycling bin And then let's say that this file that we just embedded if we drop that into our recycling bin So then what do we want to do with it again multiple files could be dropped in So we need a loop So we can connect that up We don't need to download the file because we can search for the vectors which is what we're doing over here So let's grab these and we'll drop them up here So for each file that's been deleted we need to delete the vectors So the file ID let's just delete this loop for a second and let's just test this So there's the data Then if we drop this down here we can now select the file ID from the loop which is that one Yeah And we're still aggregating And then we're still updating our record manager Except this time now we're not updating We're actually going to delete a row And of course actually we need to search for the record manager because this file might not have been embedded in the first place So let's grab that node which is this one So paste that in there Yeah So for each file we search for the file in the record manager If it exists we then go through the process of deleting the vectors So we'll update that reference loop over items one cuz that's the name of this one So let's run this node and we'll get something back So it does exist So if ID exists and it's a number and we delete the previous vectors which is what's happening there 1,165 chunks deleted Aggregate all of that down to a single field or a single item And now we can change this update our record manager to delete record manager There's delete a row And there's the ID Excellent And then last but not least we can just delete the file from the folder So into Google Drive delete a file And then we can do it based on ID And it's what we're getting in the loop This one And then after the file is deleted we can then go back to the loop and move on to the next file And so any files that are added to this recycling bin if they're not already in the record manager and they're not upserted to the vector store we can probably just delete them from the Google Drive as well So maybe actually here we just bring that down to there And that's it So let's look at our recycling bin We have this one file If we just test that workflow now and exactly yeah it's not in the record manager because we deleted it a minute ago and we've deleted the file from the bin Excellent So we now have create update and delete from this folder structure So we're looking at now a pretty resilient kind of rag pipeline where we can properly manage our vector store So that's great Dropping files into a folder like what we've done here with Google Drive is one of the main ways of loading documents into a vector store The other way is through web scraping So in this lesson we're going to use firecrawl.dev to crawl and scrape the formula 1 website We'll adapt the existing ingestion workflow here so that it works for both web scraping as well as document uploads So for this you will need an account on firecrawl.dev And I said formula 1 it's actually the FIA's website we need to crawl which is here So within the playground of firecrawl you can simply just drop in a URL like that And you have certain options For the moment we'll just limit this to five just to see how it works And what's great is you can get it to output markdown format As you can see there markdown is ideal for both chunking in a rag system and for LLMs when generating content And then also we're specifying extract only main content So we can really focus on what's actually on each page So if we click run here you'll see that it's crawling the website It's going to pick five pages from their menu no doubt and then it'll return back the markdown content for those pages So this is all great content now that we can ingest and save in our vector store If you click on get code it'll give you the actual curl request that you can then bring into nad So we'll copy that and then up here let's just add a new node HTTP request We can import this curl Now it has brought in my API key here I generally find it's better to set this up as a generic credential type using header off So we'll just create a new credential I've just pasted in the value and it's authorization So this is fire crawl We save that So now we can remove our headers and we're just sending the body And if we click test step we have a success true and we're getting a URL back which allows us to check the status of the job So that is if we were polling this I generally prefer to register a web hook when dealing with crawling That way when each page is actually crawled it can hit the web hook and we can save it And this is the format we need So it's just web hook and then we send in the web hook So back into here we'll add that Then we need to add in our web hook here So this one is trigger fire crawl And then if we click on the plus we need a web hook So that is our web hook What we'll do is we'll just copy out the test web hook for the moment and drop that in there I'll just limit this to two as we actually work through this So now if I click test step there and then quickly come into the web hook and click test workflow that's now listening out for fire crawl to ping that web hook with the data from the crawl And it will do that page by page So this thing will get flooded with requests but that's fine So as you can see it's listening now Okay so we've got a response I had to just make one change there which was to change the HTTP method to post And you can see this is what we got back and it's crawl.page I also made a small change to the request as well And I added in the events that I want to listen out for which is specifically page This is all mentioned in their API documentation here where the different events can be started completed failed or paged Cool So we now have a page that you can see here This is the main website and we have the markdown of that page So with this web hook we're only ever going to get a single page at a time or essentially a single item It can still feed into this loop over items though So let's do that Now we need to abstract things a little bit here So let's just copy this And now that we have data here let's pin this so we don't have to keep triggering the crawl So we'll press P to pin it And now let's just see what happens We're going to need to fix things along the way here So let's just test the workflow And clearly it's not going to download a file from Google Drive So if we make some space here we need an if Let's have a look at what we're getting And this is a good feature in NAN which is if you pick the node that's actually triggered this So that is the web hook which is there And then if you press dot is executed So if this web hook is executed and then if that is true then here we can handle the situation where this is triggered by firecrawl Whereas if the web hook wasn't executed then we can just handle this as standard Clearly this download file is providing binary So maybe the easiest thing to do is actually convert this text to binary so that then all of this logic will still work Let's try that So we click on plus and type in binary So let's click convert to file and we'll use the move base 64 string to file Now we will need to actually base 64 this string So that's something we'll do in a minute But let's just get the markdown first which is that one Now let's click test step And it has created binary I don't know how it worked though because it's not B 64 So if we were to quickly just convert this back to text I assume it wouldn't work Yeah So that didn't work That that is as expected So then let's move this across And there's this code that we need to use to actually generate the B 64 So we're using the buffer class as you can see there So let's get back into here We'll add a code node And we'll just make some space here and just paste this in A couple of small changes to this So item.json.enccoded message and then equals buffer That's throwing an error but I think that's just TypeScript Then we're creating the buffer from So here we need to throw in the markdown which is there And then that can be out So we can remove return item because it's been added to this Let's test it And yeah that seems to have worked Yeah there's our encoded message at the bottom So this is the base 64 of that markdown that you see there Let's rename this B 64 the markdown Convert to binary And here we can choose it's encoded message Okay click test step That has generated this So now actually let's run this and see can we extracted properly which we can That's excellent And now we don't really need that because we can just hook it up here and then all of this logic can take over If we let it flow through test workflow it'll get to the switch So the mime type here is text-plane The issue I'm having here is that actually I can't see that within the JSON because it's only outputting the binary from this field So what I might do is I might just create an additional route here And we'll do the same thing where we just check has the web hook being executed So there's our web hook is executed and if that's true then you can take this track and that can take the same track as track two which essentially just extracts the text and generates markdown It is already marked down anyway so it's not going to do a huge amount but that's fine So let's test it now Okay so that has flowed through And if we look at set text excellent We have all of that We've generated the hash which is brilliant And let's test the hash now So it's A7305 So if we test that again are we getting the same hash yes we are indeed So that's a good sanity check just to make sure everything is working So we just need to go through our logic now and see where we might have issues So that's okay That's producing what is expected The generate hash is clearly working search record manager is showing an error here which makes sense because the loop over items hasn't actually been executed uh because we have a new trigger So maybe then within our search record manager we just need to make it a little bit more general So instead of having a Google Drive file ID we just need an ID It could be a web address or a URL or it could be a Google Drive file ID So let's come into Superbase We'll go into our record manager Let's edit the column here We'll just call this a doc ID because documents could be anything And we'll save And then back in here to our search record manager And if we just reload this table we now can choose doc ID And then here we need to put in a document ID So there's two ways to do this We could use a tenorary operator Or what we could do is actually just set data at the start of this flow depending on what's coming into the loop So maybe let's do it that way It's probably the cleanest We'll set fields And here we're going to call this set data So we'll have a generic doc ID And then this can be a JavaScript expression which is either the URL of the web page that was scraped which I believe is that one there So it's either that or so double pipe And then we need to get the correct variable for the Google Drive file ID So if we save that and if we drag a file into the folder and just click on this one and now we can put in our double pipe and we can find the ID is there Cool Okay that's working now I just had to swap the parameters We click on new files again We now have our URL Excellent So let's keep going down this web hook track So yeah we're hitting an error here and that's actually because I've added this node in between So the data is not flowing through all the way It's stopping here So if we just double click into this I just need to find my markdown again and we'll just drag it back in So this is the correct term Okay let me test that again That now works again All right So there's always a little bit of tidy up when you're adding additional triggers into N workflows because you might need to abstract some of the variables and you'll hit errors like that and there's remappings required So let's test it again And it's now correctly going through here all the way down to here And we're hitting the same error again So that's fine And now if we come back into our search record manager we can now reference this new item that we created at the very start which is our set data or doc ID So then let's go into the switch So let's save and run it again now So our switch is empty and that's why it's coming down here and it's trying to create a record But of course this now has changed So we'll change that to doc ID And we need to update this now to the doc ID that we created So there's doc ID and the hash is still fine Test it again So now it has correctly created that record And let's see what this one is So yeah the file name is referenced from this download file node which is no longer being triggered So file name we can abstract here in this one as well Set data Let's call it maybe doc name And I suppose that could either be the title of the web page or the name of the PDF file or the Google Docs file Okay So there is the title of the web page So it's going to be either that or if we test this one it'll be this file name here So again it's just using that or operator the double pipe Okay test the web hook again back down to here Of course now our record manager is actually forcing a problem So we'll need to start deleting records from this or start pinning data to kind of get around that So yeah let's pin the data here and here So we have created the row and now back into this one And now we can map the new doc name So get out of that And here's our doc name And let's have a quick look forward So set text is okay and that's going to be sent into there Loop over items we'll need to change actually this file ID is now going to be the doc ID So let's change that to doc ID which we have here I believe Yep So we can paste that in there Motorport category is still fine The file name also needs to change So that's going to be doc name as the variable File summary is okay as well Okay I think we are good to go So tested end to end Excellent It's created the metadata now and it's embedding everything Now what we should do is also create a new metadata field which is document type And just in case we end up creating other triggers let's set this here as a variable So we'll do dock type And for this one we'll use a tenorary operator So we'll say if web hook is executed then we're going to set this to web page Otherwise we're going to set this to file So this will be useful information for both our agent or a reranking algorithm that we use to determine which chunks are actually the best to use Okay Okay so we now have that doc type Let's bring it into our default data loader down here which is there Okay so let's save it run it again and we should be in business Yep that worked perfectly Okay so we can now unpin everything Wouldn't mind testing the Google Drive file again just because we have made a lot of changes So we might have missed one or two variables to remap So if we test that there and yeah actually there we go It's already missed one So yeah just need to delete that and grab it from the loop over items directly Brilliant Try it again And we are hitting a blockage here that we're not getting the mime type So I just need to actually set include other input fields here So that way all of the information that's in the loop will flow through this node and append these fields as opposed to starting fresh I probably should have done this 10 minutes ago but anyway better late than never Yeah there we go It's now working Excellent Yeah it's created the row and we are in business So let's clean down both the documents and the record manager and let's import files and let's crawl the website and get lots of data now into our vector store So we'll activate our workflow and now let's drop in our documents So we have various documents from testing earlier Drop all of those into this folder and at the same time let's go and trigger fire crawl now and let's get it to crawl let's say 20 different pages on the FIA's website Actually I need to update that That's the test web hook We need to get the production web hook cuz the workflow is now active So there's the production one We'll drop that in there I probably need to add authentication to this too just so that everyone isn't hitting this web hook So I'll do that in a couple of minutes So we'll save And now let's go Let's trigger it Excellent So we have our URL to check the status And if we go to executions brilliant We can now see that there are a number of web pages streaming through from Firecrawl And if you look at the record manager these are all of the URLs That's great Now a couple of encountered errors So let's have a look at one of those And this one received a null value So it wasn't able to generate a hash off nothing So yeah the markdown was empty But maybe we just need to add a check at this point We'll put it in right here because if the page is empty there's nothing to do So we'll just do an if And actually let's come back into that error and debug this one in the editor So if the length of the markdown is greater than or equal to zero so that would be true Otherwise it can just do nothing because it's only a single web page coming through the web hook Okay all of that has imported So let's have a look at our database We have 19 records in our database So the files haven't been ingested yet Also what we should build here is a processed folder So after these files have been processed we can drop them into this folder here So we'll quickly build logic for that And then let's rerun everything And we can do it right at the end here So Google Drive move a file is that one And of course this should only run if this is actually a file and not a web page So we'll put an if Now we have no data here So let's load up a previous execution And we can now base all of this off those variables that we set at the start of the flow Doc ID doc name and doc type So if doc type is equal to file then it'll come down this track Otherwise it can just go straight back to the start of the loop And this flow is getting pretty big as you can see We'll turn this off for a second So for the delete file then we can just pull in the ID and then we just need to specify where we're moving it And we're moving it into that processed folder So if we now test it end to end it should now process the file and then actually archive it into that processed folder And similar to the issue we had with the record manager we need to actually we need to actually aggregate all of these chunks because we're getting back 520 chunks from the vector store cuz this is a big file So just click on the plus aggregate the chunks into a single item So aggregate all the fields We just need a single item at this stage We don't need to archive this file 520 times And that should be that And it's actually moved it already So we do know it works Let's push it back in Brilliant Archive the file And we are good to go File has been added to the record manager And the chunks are then available in the document store And I'm just testing the OCR again And actually I need to duplicate this node cuz we need the binary again to upload it to Mistral We'll just do that There we go So we'll tighten all this up Yeah that's working now So this is definitely becoming a big workflow We're probably getting to the point where we might need to break this into multiple workflows Let's see Can we tidy it up a little bit here yeah So this might look a little bit better Okay So let's enable our scenario again We'll trigger the crawling of those 20 pages on the FIA's website And at the same time let's drop in our files for processing And from looking at the executions we're up and running again processing the web pages from Firecrawl If we look at the record manager we have our 20 web pages there Brilliant While those files import let's trigger firecrawl again because it should not duplicate those web pages as nothing will have changed So let's dive into one of these now and let's see what's happening Yep So it's come in via the web hook It's converted it to binary It's gone down this track and excellent Yeah it's following the track around So it's searching the record It's seeing that the hash hasn't changed and it's just moving on to the next file So that logic is working perfectly now Excellent So that's how you build web scraping into your rag ingestion flows and abstract them so that they also work for files uploaded into a folder And for that web scraper then all you need to do is just run it on a schedule So you can have another trigger which could be a schedule trigger and you can set this to run every day at midnight for example and it could scrape the entire website and it'll only bring in pages that have actually changed or new pages Now fire crawl can get expensive if you're scraping a very large website on a regular basis In my agentic rag video on our channel I use a service called spidercloud Spidercloud is incredibly cheap So you could scrape a thousand pages for a fraction of what it would cost elsewhere And as you can see the average crawl cost per 100 pages is just over 1 cent by using their smart option So it really is super cheap The other option is crawl for AI And you can actually deploy this locally or you can self-host it So if you really need to scale up web crawling I would definitely go this route I just did a sanity check of the content in the vector database And the content is actually missing new lines and whites space and that was because of the node I had there stripping out all of that to do this OCR check So I've just moved that logic into here We're doing the check there And if it passes we're not actually changing the text So that should solve that problem So our text is perfectly fine now to be indexed by the ingesttor So let's now ask a question of our agent Any news on the Safari rally cuz that was one of the news items that was just crawled and imported And yeah there we go It's referencing the news items that were just ingested along with actually links to the content too And there we go There's our news item That's brilliant And our documents are now being ingested as well We're now entering the advanced section of our rag course And up until this point we have been primarily focused on ingesting data into our vector database through this now extensive workflow here But let's now start focusing on the generation side of our rag system because there are a number of improvements that we can make here to dramatically increase the accuracy and quality of outputs from the agent And all we've really done so far is prep some metadata that the AI agent can use when querying the vector store But let's actually remove this from the flow for the moment and let's tackle two of the best ways to increase accuracy in your retrieval system And that is by using hybrid search and re-ranking So let's connect this up here And in this part of the course we are moving away from no code into vibe coding because a lot of these advanced techniques aren't natively supported in NAN So we need to actually build up the building blocks ourselves So we'll disconnect our tool here And if you click on add tool we'll be using this call N8N workflow tool The idea here is we're going to trigger a separate workflow that's going to trigger this hybrid search And based on what comes back we're going to use a reranking model coher 3.5 which is one of the best in the industry to sort those chunks based on relevance So we'll name this query vector store call this to fetch data from our vector store knowledge base And then the workflow that we're going to trigger is actually the same workflow we're currently in which is this one Okay so we have query vector store Let's get this out of the way Then we need to add another trigger and it's when executed by another workflow So we can drop that in here So the idea then is when the agent hits this tool it's going to trigger this flow Now you could set this up in a separate workflow on NAN but it's nice to have it all on the same canvas here particularly when building it out And the key question here is what data are we going to pass in to actually send to the vector store So if you click on this and over on the left hand side here you can define fields So this will just be query And now that we've set that here if we come back into query vector store that field shows up at the bottom here And then we can just click this button and let the model define how to inject data into that parameter So in other words the AI agent is going to formulate the query and send it into the sub workflow So this is pretty much what was happening here with the Superbase vector store because there could be quite a long natural language sentence being sent in by the user and the AI agent was rewriting that query and sending it into Superbase So it's kind of the same idea And while we're building this up let's just load some dummy data into this So if we click test workflow for example you'll see that the query comes back as null So if you click the edit icon we can just edit this data What is the plank assembly and if we save that then that data is pinned So if we click test workflow again you now have that custom text in that query Otherwise while building this we would need to keep engaging the agent to trigger the tool to trigger the workflow So it's just a lot easier by pinning the data there So we're going to start off with hybrid search Now within edit end if you click plus and then go to AI other AI nodes at the bottom and then miscellaneous there is actually a lang chain code node that you can use So Nad's AI features are built on Langchain but not all of the Langchain features are actually supported by the native nodes in NAN So if you wanted to interact with other elements of the lang chain framework you could use this code node for this hybrid search functionality I actually went a different route I didn't use this code node because superbase actually have documentation on how to implement hybrid search and it looks pretty straightforward with most of the logic actually happening on the superbase side So our workflow here should be pretty clean in the sense that it should just be triggering different superbase functions to actually trigger the hybrid search And if you don't know what hybrid search is it's all documented here on this page So essentially hybrid search is combining full text search which is searching by keywords with semantic search which is what we've been doing to date with vectors And he uses the example here of how sometimes a single search method doesn't quite capture what a user is really looking for So the example is if someone searches for Italian recipes with tomato sauce a keyword search would return you results that mention the word Italian or recipes or tomato sauce but it might miss out on dishes that are semantically relevant like pasta sauce or marinara And while a vector search will provide you those results because they are similar in meaning to the actual query they might also give you other results that are contextually related but they're not exactly what the user is looking for like Mexican salsa recipes So in a lot of use cases it's actually worth running both keyword search and semantic search and then combining the results of both of those searches to provide a better result set that you can send to your agent So the act of combining the results from your keyword search and your semantic search is called reciprocal ranked fusion And this is a simple algorithm that provides waitings to the different results to create a combined list So this document provides step-by-step guidance on how to actually set this up in Superbase So you created your documents table similar to what we did before and then you create this hybrid search function which carries out these independent searches and then combines those lists using that reciprocal rank function And then with that function set up in superbase we then just need to hit it via API and to do that we're going to use a superbase edge function and it gives a sample of the edge function here as well Now the catch here is that when we initially set up our vector database on superbase we used this quick start for lankchain You can see that it's providing us a template where it actually creates the tables and this match documents function And this template includes things like the metadata field which is obviously something we want to use with our vector store That metadata field isn't actually available in the create table here So if we were to just blindly follow the instructions on this page we wouldn't be able to use metadata And this is where the vibe coding aspect kicks in So I used Gemini 2.5 Pro and I passed it my current superb base setup script which was the lang chain one I showed you a minute ago that does contain metadata and the match document function uses that metadata as a filter And I asked it to add hybrid search to this query and then I provided it the link to that documentation And then I had an extended back and forth with Gemini 2.5 Pro where it actually outputed various versions of the code for me to use And this is the SQL script that we landed on So you can do the same thing yourself I didn't handw write this We will make this available within the course in our community So if I copy this out then if I go to the SQL editor and just delete what's there and paste it in and click run you should get a success No rows returned And if you click on the database you should then see your additional table Here I've called it documents v2 so that I'm not replacing the vector store I've already created earlier in the course And let's also duplicate our record manager so that this can be our hybrid search system Whereas what we built to date is just our pure vector search system So if I just open up the record manager you click definition on the bottom right Then you can open that in SQL and we'll just say create table public.record record manager_v2 and we'll just click run And under constraint as well you need to put in_v2 and then click run and you should get a success So now if we go back to our database there's our additional table Great So we have our version two record manager and document store along with what we've already built to date And the only difference now in this new document store is we have this FTS field which is our full text search field And you can see the type is TS vector So back to the documentation then we have our table now created Next up we need to create indexes on those fields so that we get a really fast response back Now I think that was already done in the script that Gemini wrote So if we come back into superbase I'm in database here but if I click on indexes on the left yeah you can see full text search index embeddings index So that's effectively what was set up there So we can now move on to the next section Next we need to create a function that actually carries out this hybrid search So this function is very similar to the match documents function that we were using earlier that was in that quick start template The only real difference is that we're passing the actual query because that's what's going to be used by the full text search to actually find exact matches And then there are other optional parameters that can be changed around the ranking fusion process but we'll leave all of that as the default values that you see there So instead of copying this in back to Gemini and let's let's fetch our updated hybrid search function that takes metadata into account And after a bit of a back and forth this was the hybrid search function that I landed on This one includes visibility on the scores from both the keyword search and the vector search so that I'm able to sanity check that the system is working So I'll copy that out into the SQL editor and then you paste in the function and you click run then you get a success and then you can scroll down and this is my function hybrid search v2 with details So that's now a database function sitting in superbase but we need a way to actually hit it from n So we need to wrap this in an API and we do that using an edge function So back to the documentation and it gives example code for an edge function So I brought that into Gemini and I had a back and forth on it to take account of the metadata but also to update for example the dimensions because we're using a different embeddings model and while Gemini did produce example code for this edge function I did run into a number of errors when I actually ran it and I found much more joy in actually just going to edge functions in Superbase and then from the dropown click on via AI assistant and then you can essentially just ask this AI assistant to create an edge function for And that's what I did here And that AI system created this hybrid search v2 edge function with this data and it's hitting the database function here So there is a difference between edge functions and database functions Database functions run within the database to carry out operations whereas edge functions are deployed in the cloud to the edge and they allow the one-off running of code So this function acts as a secure gateway into your Superbase account And if you click on details then you get your endpoint URL that you can hit and it even provides an example curl that you can send So while this part of it is actually accurate the data isn't because we have defined that in the function itself But Gemini was able to provide an example request body that I could send into the function So we have everything we need now to start building this in N8N I'd encourage you to go through that process of having that back and forth with the AI If you're not a developer you definitely can vibe code this like I did today So I would encourage you to go through that process But if you would like to get access to the code files that I use in this they are available in the rag course within our community So let's copy out this endpoint We'll come back into N8 and then if we click on plus we're going to use the HTTP request node And actually we can just copy in the curl that it sent us here So we'll grab that You can import the curl So it's pre-populated the URL as well as our bearer token Now that of course could be saved within the authentication section of NAN I'll leave it there for the moment And then this is the area that needs a little bit of work So we need to remove this using fields below because this is not the format for this function So we'll click using JSON Now if we come into Gemini this is the format that it's actually expecting So we'll copy that out and let's drop that in here And there are a few changes we need to make So obviously we need to now drop in the actual array of vectors So we do need to embed our query So we'll leave that for a second and move it out of the way We now need to hit OpenAI's vector embedding service And if you click on OpenAI NN doesn't have a standalone embedding endpoint It's only available as part of the vector store that you see here So we will need to hit this directly And this is the endpoint that we need to hit So vector embeddings So it's the same story as before Let's just copy out the curl that you see there And we'll create another HTTP node Click import curl And now we can drop that in So we're hitting our embeddings endpoint on OpenAI It is looking for a bearer key We definitely have this set up here though So if we just turn off the headers and then in authentication click on predefined credential type and then just type in OpenAI and it's that account there And then within the body parameters you can see this is the text string that will be embedded and that's the model that will be used So if we click test step just on this one alone and it worked So these are the vector embeddings for this text So we need to delete this out now and drag in the query field that we set at the start of this workflow and that's what will be sent in from the AI agent So what is the plank assembly as you can see there and if you test that this is the vector representation of that question What is the plank assembly and this is what will be sent into our vector store to get semantically relevant results back So let's rename this generate embedding from query And here this is our trigger hybrid search And if you come down into here let's just open this up in a popup We can see now we have query embedding So if you delete out the numbers that you see there we can now just drop in our array And there you go So that's our full array And because this is a hybrid search we also need to pass in the query because that's what's going to be used for the full text search So you can get rid of the search query text and then drag in the query there Brilliant So we'll be sending in what is the plank assembly and that'll go to the full text search We'll be sending in the embedding that'll go to the vector search They will both produce five matches based on similarity And then both of those lists will be fused together using that reciprocal rank fusion process And then for filter these are the metadata filters So what did we have it was uh motorsport category And actually let's add that as a field now that can be sent in by the AI agent So you could have motorsport category and the AI would need to provide an option for that or as I mentioned it could be department it could be author content type It also doesn't need to be a single string for each metadata item You can have this as an array and then the AI agent passes you the array to filter your results collection in the vector store So yeah let's leave this as string for the moment Now if we go back to our AI agent let's just drag back in our prep metadata because we've made an effort in Superbase now to align the actual functions to work with metadata So we may as well use it So we'll drag that in there and we'll save Okay And back to our trigger hybrid search Let's now add in our motorsport category variable So remove value And it hasn't updated there actually because it's pinned of course So if we unpin and let's just run that we'll get two nulls And now we can add test data So what is the plank and for motorsport category we'll just add in F1 and save that and that's now pinned Okay So we're at a point where we can test this So let's click test workflow We have generated our embedding as you see there If we click on the hybrid search now it is now it is saying no output data returned which makes sense because the database is empty Now it is always worth selecting this always output data because the actual workflow would just die at that point Whereas I always feel you're better off handling a scenario where no output data was returned So we'll save that run it again and now you can see you're getting an empty array So that's perfect Let's delete this And what we'll do is now if we go into our database we have nothing in our version two documents and we have nothing in our record manager So let's now actually ingest some documents and let's test this out So if we go up to our rag data ingestion section up here there's a few changes we need to make So for search record manager we need to target our new table So that's version two And then we'll be creating the row in that table as well as updating the row in that table And for this delete previous vectors we'll be targeting our version two documents table which is our vector table as well as here in the vector store we need to set our new table Now let's drop a document into our folder And we'll come back in here and let's test the workflow now And that should now ingest that document into our new tables And there we go The record manager is up to date and our vectors have started to be created And now you can see the full text search vectors So this is really interesting Previously it was just these dense vectors that we had in this embeddings column whereas now we're getting these sparse vectors here And if you copy one out and open it up you can see what it looks like So you have the keywords that are in the chunk It's stripped out all of the padding words like the or ah or there and it's only bringing in keywords that might be useful in a direct search This is the real power of direct search is that if there was a very technical term in the chunk it would actually be listed here and you're more likely to get a much better result which we're going to see in a couple of minutes So now with that document uploaded let's unpin this and then let's ask a question and then yeah let's see what should we ask it Let me open up the document So we've got this term here PUTB with no ignition It's really specific term Let's try this one So let's just put that into the chat PUTB with no ignition So it goes to the agent which goes to the vector store We're actually getting back Sorry I don't know So let's dive in and see what's going on So if we double click the vector store yeah we're definitely not getting anything and we're not passing the metadata we're just passing the query So let's go back into this and you'll see this warning sign that the workflow inputs need to be refreshed So we click on that now There we go So now that we have a field that we can populate let's pick it up from our mapping and we can drag in that motorsport category variable Excellent And let's ask it again F1P UTB with no ignition Still not getting anything And that's because we didn't actually set it here as well And we just need to drag in that motorsport category filter into here That is that one there So I think probably having the AI agent dynamically building out this entire data structure probably makes more sense than doing it parameter by parameter But that's probably fine for the moment So last test Hopefully it works at this point Excellent We now have results So let's have a look at it Put with no ignition It talks about occupancy hours and operation hours But more importantly let's dive into the chunks that we received so that we can actually see the scores from the vector search as well as the keyword search So you can see the text of the chunk And if you scroll down here we're getting vector scores and keyword scores Now they're not really comparable That looks like a lot lower than that but they're completely different algorithms But this is what's important which is vector rank versus keyword rank And interestingly this chunk was returned fourth highest by the semantic search but it was actually the top chunk from the keyword search And then when those ranks refused it received a final score of 038 Then if you look at the next chunk this was the top vector result but it didn't even appear in the top five of the keyword results So when those ranks refused it got a 019 So then it was second and so on The third chunk was actually second in the vector results The fourth chunk wasn't in the vector results but it was in the keyword results So this kind of fusion of two different lists is really interesting I think And actually let's compare this now to a traditional vector search that doesn't use keyword matching So off camera I just created this node This is our superbase vector search And for this I just deployed a different edge function which is only using our embeddings So now actually if we line this up here save that and let's just load up this execution So we'll copy this into the editor So the query is PUTB with no ignition F1 is the category And if we get rid of our chat and just run this it's going to generate embeddings for the query and it's going to hit both searches And it is fascinating So if you jump into the superbase vector search for this exact term PUTB with no ignition if we search the output only one of the five chunks has that term Whereas with the hybrid search two of the five have that term So the hybrid search has definitely returned more accurate results which can only produce a more accurate answer from the AI So I hope that wasn't too confusing It's kind of complex to go through it particularly when you're vibe coding the superbase edge functions and database functions But next up let's use another popular technique which is called reranking And what re-ranking does is it takes these chunks that are returned from a search system be it a hybrid search or just a vector search and it sends it into a model to order them by relevance And this is interesting because with a typical search it's not actually ordered by relevance It's ordered by similarity And in this case here cosine similarity is the algorithm used Whereas when you're ordering by relevance you're actually feeding the information into a neural net and you get a much smarter ordering of the items based on the query that was asked And the most popular re-ranking cross- encoder model out there is from Coher who have their 3.5 re-ranker that we're going to be integrating with now So you go to Cohir and you create an account and once you sign in you'll land in their dashboard And two things you'll need to do You'll need to generate API keys and you'll need to go to the docs and click on rerank at the bottom We're going to be hitting this endpoint So as we've done before you can copy out this curl And now let's bring it in here So we'll click on the plus So we're going to add another HTTP request node And we'll import the curl again So that's the endpoint that we're hitting We need to pass authorization So this is where you need the API key Let's paste it in there I've dropped in the key And now if we look at the body you can see what's going on So we're indicating which model we want to integrate with We need to pass the query because that's the context that it needs to make its determination on what's relevant or not We need to pass in the number of results we want Let's say five for the moment And then you need to pass in an array of strings or an array of chunks So to do that then let's move this aside And we need to create a new node because actually if we load up the data here you'll see that we're getting five items back which isn't going to work We need a single item number one and then also we need it to be an array of chunks So we could add in an aggregator here and then we could add in another node to strip out a lot of what's in these items so that they're just strings It might just be easier though just to do it as part of a code node So let's do that We're now vibe coding So let's go all in on our vibe coding So Gemini has created this JavaScript code that I've tested and all it's doing is it's grabbing the content from each of the items that come back from the hybrid search and it's dropping them into an array So we click test step You can see that's what we're getting there That's the second chunk That's the third chunk Brilliant So now into this node and let's rename this rerank with coher 3.5 If we double click this and we'll set the JSON to expression because we now need to dynamically populate this So firstly the query is coming from the first node which is there and actually we are getting an er and actually we're getting an error here but I think that's okay that'll be solved once the code node runs but for documents let's delete out all these documents and we can just drop in our array here and you can see we don't have quotation marks so you need to use the JSON stringify function and you can wrap JSON dod documents and there we go and actually we can get rid of these square brackets as Okay so let's test that out and let's pin these because we don't need to keep generating embeddings and triggering the superb base hybrid search and we can unhook our vector search as well Okay let's test it We've hit an error but that's okay And it hasn't actually fixed this So let's just put in first Excellent So there is our results And wow it really is actually shuffling that list So what you're looking at here the index here refers to the chunk that we sent in So it's saying that the first chunk we sent in is actually the most relevant So the hybrid search got that right but it's actually saying the second chunk should be fourth and the third chunk should be fifth And it's provided relevant scores on all of those So it really has analyze and modified the ordering here So this is really useful if you're actually returning lots of chunks Let's say you're returning 20 or 30 chunks You could send all of those into cohhere and only get it to output the top five And that way then you're not sending lots of noise and irrelevant text into the LLM which may then go off track and hallucinate an answer So you can keep it on track with the most relevant chunks using this type of system So now we just need to reorder our documents array here on the left based off this new ordering that we have on the right And again let's vibe code it And here's the script And if we click test step and here we go These are our sorted documents So now the fourth chunk is second The fifth chunk is third And you can verify that by what you see here And now we can simply just end the workflow here because this will then return to the AI agent to actually generate the text So this is return reordered items And now let's remove all of these pins We can save it And then back to our agent and we'll ask about the PTUB with no ignition And there is our answer which definitely looks like a more comprehensive answer than anything that we've got to date Now of course you could play around with the chunk sizes You could get 10 chunks or 20 chunks and then rerank them to a smaller amount So this is really the subjective part of evaluating rag agents You need to use your own documents that you're quite familiar with because you'll be able to spot when it is kind of hallucinating or where it's not giving a fully comprehensive answer Okay so that's our hybrid search and our reranking model Next up let's look at improving our rag system even further with contextual embeddings So if we look at the chunks that we have saved in our vector database I think this is a good example of a chunk which kind of lacks context It's starting what appears to be in the middle of a sentence It's got an array of random enough numbers There's a couple of terms thrown in but it really isn't that obvious what this chunk is about And this is the real limitation of rag systems The chunks are created independently of each other So if the previous chunk to this was actually setting the scene or defining some of these terms or these features that chunk might not be retrieved and then the agent wouldn't have visibility of it So that's definitely going to hurt the actual accuracy of what the rag agent is generating So by contextualizing the chunks what we're doing is we're adding in a section at the top of each chunk to set the scene for the chunk So this was popularized by Anthropic in their contextual retrieval paper from September last year I've created an entire video on this on our channel and this workflow that you see here is available in our community which is contextual retrieval with context caching And here we're using Gemini to actually generate these chunks or these snippets that are added to the start of each segment I've also created a workflow and a video on cache augmented generation or prompt caching because these two concepts go hand in hand You're going to the LLM for every single chunk and you're providing the entire document every single time and that way it can situate the chunk within the document The problem with that is it's really expensive and it's slow and LLM providers know that which is why they've brought out prompt caching and prompt caching is a lot cheaper which makes contextual retrieval more affordable for users So when looking at our rag ingestion pipeline the area that we're going to need to change is down here where we're setting up the text for embedding and sending it into the superbase vector store And similar to reranking and hybrid search N8N doesn't have any contextual retrieval nodes we can use So we need to build up the constituent parts And for this we'll use OpenAI's prompt caching Of the three main providers OpenAI's prompt caching is the easiest to actually use The problem with it is there's only a 50% saving on tokens in cash whereas with Gemini it's 75% and with Anthropic it's upwards of 90% So let's make some space We have our text for embedding here which is essentially what we're getting from this node So what we need to do is we need to chunk this manually because if you click on plus and if you go to AI and then other AI nodes you'll see that there are text splitters but these text splitters only work as sub nodes of a main vector store as you can see there So they actually have to be joined like that So that's not going to work for this use case So if we create a code node again I've got AI to generate a chunking script for me which you can see here So this is a recursive character text splitter It's chunking based on a size of a,000 characters with an overlap at the start of 200 And then we've set separators and boundaries So this should now take our text and provide the chunks the same as what's actually happening here So let's rename this chunking and let's drop a file into the folder here so we can test our chunker Now that's uploaded and let's click test workflow And then what we'll do is let's actually just pin all of this data or pin as much of it as possible If we test that again now it should flow through it all which it does and we are hitting this node Excellent Oh yeah we have the wrong reference to the text here So that is content and we test it again Yes So that has worked and we're getting 611 items So actually that's quite a large document to be testing with Let's get a different document up and running Yeah this one is 16 pages That'll be better Now I'll just need to unpin and repin everything Yeah that's better 58 chunks So that's a bit more manageable So let's move all of this along And now with these chunks we now need to loop through them to generate embeddings So we're going to use a loop over items And the other thing is we don't need to do this in batches of one chunk at a time This very much depends on what rate limits you have with your LLM provider I'll set this at 10 for the moment So there's my loop and let's name this loop over chunks And now what we can do is we can go to OpenAI and we can use the message and model option and we can choose a model to use here Now we're going to be hammering this API every single time a chunk needs to be processed We're going to be sending a trigger to this LLM So I wouldn't use anything too expensive So maybe 4.1 mini maybe 4.1 nano Let's try mini for the moment And if we go to Antropics contextual retrieval document they actually include the prompt to be used in this And this is it here So let's copy out this prompt And we'll drag it in here And let's just set it as an expression because we are going to be loading dynamic variables And we have undefined And where it says whole document you can delete that and just drag in the content of the entire document That's that there And then for the chunk you can drag in the text from the loop And we actually need to execute previous nodes to see that So we'll just delete that for a second Okay So let's run this now just so that we can load up the data into memory And we can stop it straight away Now so we have our chunk at the loop So we can drag it in there And if we zoom in let's have a look at what we're saying here So first of all we're passing the entire document and then we're saying here is the chunk we want to situate within the whole document and we pass the chunk and then we're saying please give a short succinct context to situate this chunk within the document for the purposes of improving search retrieval of the chunk Okay And let me just check the rate limits So 4.1 mini has a 10 million tokens per minute limit which we could hit So that's something to look at We're not going to probably send 10,000 requests per minute but definitely the token limit we may hit So that's something that we'll just need to be conscious of I suppose it depends how fast this returns as well And maybe for the moment let's even just set the loop size to one just to get this up and running with a single chunk And then we can increase the batches and increase the scale So we'll test it out again And it's finished We haven't looped it back So it's just ended there And if we double click now you can see this chunk contains a title page publication table of contents for the 2026 Formula 1 power unit sporting regulations Excellent So let's rename this one generate chunk description And now what we can do is just line up the text that can then go to the vector store So we do edit fields And that's essentially what we've done here with setup text for embedding But this is actually setup chunk for embedding So we'll call it content And here what we're doing is we set the value as an expression We're firstly going to provide the chunk description and then we can provide the chunk and then that's what's going to be embedded for this chunk and it's grounding and situating the chunk within the context of the document So then that can go to superbasis's vector store for a default data loader We can just set a really high chunk size This will be a th00and um there's only a single chunk but this is a required node You can't just delete it So we'll just set this to 100,000 Okay we're going to use the same embedding model That's fine And then after that chunk has been embedded and upserted to the vector store We can then go and process the next chunk So we'll drag it back to the loop there And then this replace me branch let's get rid of that And then when all of the chunks have their descriptions created and have been upserted to the vector store we can then drag this down here to continue to the next file Okay So I think we are ready to test this out So let's just make sure the actual loop works correctly That looks good Yeah it's just incrementing one by one We can stop that And actually out of interest what's the size of the tokens that we're sending here do we have any logs or any information on that i'll need to dive deeper into the monitoring to actually figure that out cuz when I built this out before we needed to actually build out a weight function to get around some of these rate limits Anyway let's scale this up now and let's go to batch size Let's say 10 So now it's hitting that 4.1 mini API 10 times per loop So you can see why you can get rate limited with this approach particularly because we're hitting an LLM It's not an embedding model like it is here But that was fine there And an approach here actually is to just set retry on fail and actually provide a high enough weight So the max is 5 seconds and the maximum of times you can try is five So we'll set it as that Now we have 58 chunks in this document So we're halfway through already Contextual retrieval and the generation of these blurbs or these descriptions really slows down the actual ingestion process As you can see here now if we were using GPT 4.1 Nano it might be faster and we might end up with the same results or the same quality So you could play around with that and test it and see what works and what doesn't work Okay so it's finished processing that document So if we just go into this now and if you scroll down to the usage you can see the prompt token details So this inference to 4.1 mini took 10,000 tokens of which the vast vast majority of them were cached But this is where the 10 million token limit per minute kicks in because with that size document if you trigger that a thousand times let's say in a minute you would hit the max which you might struggle with in this flow But if that document was 10 times bigger you would only need to trigger it a 100 times to actually hit the limit which is definitely possible in this loop So that is something that would need to be designed around maybe some sort of counter of tokens And once you hit that 10 million tokens within a certain time frame you pause for a period of time and that way you'll be able to make it a lot more resilient And now if we jump into our vector database and if you have a look now at the chunks it's really interesting So every single chunk has this blurb at the start This chunk is from article 4.2.2 of appendix 1 in the power unit regulations and it talks about the specific area of that subsection and what it's all about So this is incredible grounding of the chunks and it just means that you're not really losing much context now by chunking And based on the benchmarking that Entropic carried out by using this approach along with using reranking which is what we're doing here and with using hybrid search which is what we're doing here it reduced the error rate or the failed retrievalss down to 1.9% compared to standard embeddings which had an error rate closer to 6% So I'm now running the full file import end to end I've increased the batch size to 50 just to see how it handles it Yeah that seemed to perform okay These are obviously pretty hardened APIs with OpenAI I do find Anthropics APIs can be a little bit hit and miss They tend to go overloaded quite a lot Okay it's archiving the file There's only one file to import And if we come into our vector store we have our 58 chunks We have our document with its hash And it has archived the file And that file is upserted Brilliant There are other advanced rag techniques that you can use to either improve the ingestion process or the retrieval process I've covered a lot of these on our channel so I won't go into much detail on this course but I do recommend that you check these out if you're looking to dive deeper into any specific technique For example in this video here I go through a gentic rag in detail And the use case I walk through in that video is our Agentic Rag blogging system that we have available in our community And a key part of that blogging system is the role of the researcher And this agent rag system that's powered by claude 3.7 sonnet with thinking mode enabled So it's a reasoning model It has a pine cone vector store with a number of different namespaces It has structured data sets on no code DB as well as external grounding with perplexity and withina What's quite incredible how this agentic rag system is able to use its internal reasoning to actually rewrite and expand on queries sent into a vector store as well as analyzing and making decisions on the data that comes back from those vector stores to decide what to do next You can limit the recursion at the rag level here Max iterations is a setting and it's set to 20 there but that could be set much higher if needed So I definitely encourage you to check out that agentic rag system and that's that video here Another interesting pattern with rag is cash augmented generation I have a full video on that here as well And within this blueprint that I walk through in that video I built out cash augmented generation with Gemini Anthropic and OpenAI And what you saw today with the contextual retrieval system we built that was using CAG with OpenAI which is prompt caching However with Gemini it's done very differently You actually upload your file to Google Cloud and they give you a cache ID that you then pass in with every request So you're not actually uploading the file every time You're just passing in the ID So it's quite a nice system And because they have very large context models it can be quite useful if you have a lot of data that you need to go through And then Antropics version is a little bit different as well Using cache augmented generation with entropic requires you to actually specify a cache control parameter for each prompt that you send in That way you can actually cache system prompts or grounding documents that you might include So I think this technique will only become more popular as the context window length of models increases and the costs of inference come down And finally another advanced technique I've gone through on this channel is late chunking which you can watch in this video here So late chunking is a strategy that was introduced by GINA.AI and the whole purpose of lay chunking is that instead of chunking the document independently before you create the embeddings with late chunking you actually embed the entire document up front and then you chunk it So it's a really interesting paradigm shift and that's what this flow does It creates an embedding of the entire thing and then it splits it all up and it upserts it to quadrants vector store As you can see there similar to anthropic GINA have created benchmarks to see how much of an improvement you get with this technique And what they found is as the document length grows and grows the improvement dramatically increases because obviously with longer documents there's more context to lose There are just so many techniques that you can use to improve the output of a rag system I wasn't able to go through everyone in this course but I will be diving deeper into individual techniques on our channel in future So make sure to subscribe to the channel to get notified when I publish those videos I hope you enjoyed this course and you learned a thing or two about RAG and how it works in NAD If you'd like to get access to any of these blueprints they're all available in our community the AI automators And make sure to give the video a like and subscribe to our channel It really helps us out Thanks so much for watching and I'll see you in the next