hello everyone and welcome to another section of our deep dive Series today we're going to be talking about elastic search which is one of the most popular and Powerful open-source distributed search engines reminder that our Deep dive series is intended to give you depth that's going to be relevant for your interviews and so what we're going to be trying to cover here are both how Technologies can be used in an interview and an assistant design setting as well as how they work so that you can understand the internals and answer questions uh that may come up I'm Stefan I am one of the co-founders of hello interview I've worked as a senior manager at both meta and Amazon and over the course of my career conducted north of 2,000 interviews very familiar with how interviewers tend to think about the process and hoping to share some of those insights along the way this session is going to be divided into two parts in the first part we're going to talk about how to use elastic search this is particularly relevant for interviews where search is actually not the primary functionality or in certain types of interview like the product design interview at meta or interviews for full stack engineers at startups where your interviewer wants to see you solve problems not necessarily you have deep knowledge of how distributed systems work in the second half of the session what we're going to do is peel back the layers of elastic search and talk about some of the design decisions under the covers and how clusters function the goal there is both to give you some insights into how it actually functions but also some inspiration for your own designs especially as they pertain to more infra style interviews before we get too deep just a reminder that all of this information is available on our website hello interview.com there we've collected a bunch of deep Dives on technologies that we really think you should know just like elastic search as well as common problems that you might encounter in a system design interview in these instances we've broken it down walk through some of the different choices that you might need to make and then giving you an indication as to which ones are great and which ones are probably not the ones that you might want to use in an interview we've also recently launched our guided practice product guided practice is solving this problem that a lot of people watch videos they read blogs and they kind of fail to integrate the information that they've consumed with guided practice you're actually going to work through a problem you'll submit your answers on a whiteboard word by narrating your response and then you'll get personalized feedback on how you did what things went well what didn't go well and how to improve this is a great way to make sure that you're actually retaining the information that you're taking in and there actually is a guided practice for post search which is a perfect complement to this video in the first half of this video we're going to talk about how to use elastic search in a system design setting for that you first need to talk about search so what is search well typical search experiences are comprised of a few different pieces first I'm going to give the search some criteria let's pretend that I'm searching for books I'm actually going to use a bookstore as a motivating example throughout this video so maybe I want to search my bookstore for books that have the title that contains great and for a price that's less than $13 that's potentially describing a lot of books so I might want to sort those books in my search results in that case I might specify that I want to see the most recently published books that contain the word great in my search result page I might also have an option to break down the search and refine what I've described these are typically called facets or faceted search and one example is I might want a refine by the publisher maybe hash it or Simon and Schuster boo hash it for suing the internet archive anyway the net of a search experience is a set of results basically a set of documents in this casee the documents represent books that I want to return but they could really be anything we can think of this as arbitrary Json blobs that we might wearn in return while elastic search does support other use cases like analytics use cases as part of the elk stack I'm not going to talk about this too intensively in this session just because search is really rich and I want to go into more detail now how does elastic search think about search well there are some key Concepts that we have to identify and name before we can go into more detail about that the first and most important concept is the document like I mentioned documents aren't necessarily PDFs or things like that they're just Json blobs and they can potentially contain anything the next primitive that we're working with in elastic surch is called an index and index is kind of an overloaded term in computer science index is usually referring to how we structure data to make retrieval really fast in this case an index is just a collection of documents that we want to make searchable within an index we have two relevant Concepts one is called a mapping and one is called a field the mapping is going to specify in some sense the schema that I want to be searchable on my documents so my documents might have dozens of fields but my users may not need to search all of those my mapping in some sense is telling the search engine elastic search which of those fields are pertinent and then what are their types so in this case I can tell elastic search that the price is always going to be a floating Point number and as a result elastic search can organize that information so that I can have range queries and I can find Price Less Than $13 so with our terms in place we can now talk about how we actually bring docs into elastic search and how we make them searchable elastic search provides a restful API and it actually actually has a pretty good one so if you haven't designed a restful API before it's actually a useful one to look at both so you can get some ideas about what a solid production level API looks like and how they handle some complicated Corner cases the very first call that we're going to make is this put to slash whatever our index name this is creating a books index and I can specify some configuration parameters but all of them are optional I'll get into what shards and replicas are a little bit later important thing here is this creates me an index I don't have to specify a mapping but I can and the reason that I might want to is because if elastic search can't infer the types of the fields which ones I want to search particularly complex configurations it may make some mistakes there's also a lot of overhead to having elastic search guess at what your mapping might look like because it's going to be indexing a bunch of fields that you may not care about now note the field types here have a ton of implications we talked about how the price we wanted to be able to do range search on so that's available here note also that we've got these text and keyword types text allow us to do rich full text search so if I want to search for Great in Great Gatsby I can do that in a text field a keyword on the other hand is opaque each of the tokens is kind of a unit and so this is really appropriate for places where for instance my set of categories is fixed maybe in the UI and the users can click or select from the available categories this guarantees that I can do those lookups Lightning Fast think a hash lookup but it won't allow me to for instance search for all categories that contain the word fiction when that might include non-fiction note also here that I am including this nested reviews this if you're famili with normalization and denormalization might be a little bit clear but what we'd like to be able to do is answer our search queries with as few requests as possible and by nesting the reviews in the document we can basically construct queries where we can say I want specific types of books that have certain reviews now this is not without its downsides if reviews are changing a lot or if my queries rarely touch them this is actually going to be really EXP expensive for me to maintain that way and I might actually be better served by creating a separate index for them but in this case I'm showing you what this might look like and assuming that reviews aren't going to be shifting a lot but that customers care about them next let's talk about adding documents to our Index this is dead simple I make a post request to this undor doc endpoint inside my index and I pass all of the documents contents when I do this elastics search is going to respond with a basically success message that tells me a number of details for instance what was this document's identifier what was the index it was created into and this version number the important thing here is that elastic search is a distributed system we could be making updates from any node and they might take some time to percolate through the system so it's really important that we handle collisions and concurrence this is particularly relevant when we start to talk about updates for updates the simplest possible API is to make a put request to this doc endpoint with the ID that I'd seen earlier I pass in the entire document body and I can update it now this is probably Overkill if I just want to change the rating on one of the reviews for instance but it also has some downsides in that if two updates are submitted simultaneously the last one wins this is where that version identifier comes in what I can do is in my update I can pass in the version number and what elastic search will do is it will reject that update if that version number doesn't match this gives me a chance to handle the Collision or the conflict on the client my client can then go in query what the latest version is take a diff decide what to do and resubmit it to request this is actually a pretty common pattern in distributed systems and it's just a way to ensure that the update happening is the one that we would anticipate another strategy here which actually doesn't solve the raise conditions but solves part of the problem is that I can make field Byfield updates by post posting to this underscore update endpoint and in this case I'm simply bumping the price from $33.99 to $4.99 Okay so we've talked a lot about how to insert and update documents as if elastic search is a document database and it kind of is but the key thing here that we want to get into is search well fortunately search is really straightforward too there is this nice API endpoint underscore search which sits on the index and allows you to search using this query language that's basically Json and very SQL like in nature so if for instance we wanted to allow searches by title where it contains a specific word then we can just simply do a match query with great in the body if we wanted to extend that by saying that we also want it to be less than $15 we can do that and then if we wanted to have a query of books where the reviews had this comment excellent and their rating is greater than or equal to four out of five we can do that too now note these are get end points you can't actually put this in the body of a get request it doesn't actually meet standards or you can actually stuff this into a query parameter but you're limited by the size of a URL which I think is 2048 someone in the comments please correct me it's also possible to post here although that's a little bit out of Canon in that most of the time when you're doing search requests you're expecting them to get they're not actually mutating anything but the important thing here is this API is really nice and convenient and allows you to specify arbitrary search criteria the first part of the search product that we talked about earlier now once I've got search results s they're going to come back to me in an API that looks like this elastic search is going to conveniently report to me how long that search took whether it timed out how many shards it hit again we're going to talk about this a little later and then it's going to tell me how many hits and it's going to return the contents of those documents to me now this is configurable and in some sense you may not want to return the entire document in your search results but elastic search has this underscore source which is basically that Source document that you originally uploaded it's also possible to return only specific fields from the document I may only want the ID or the title I'm always going to get the document ID inside of elastic search there's also this field called score which is trying to calculate the relevance of the document to the query that I've input and that brings us to the second part of our search product which is the Sorting just like searching and filtering elastic search has a pretty extensive API to allow you to specify whatever sort order you want so in this first instance I'm sorting exclusively by Price whereas in the second instance I'm sorting first by price and then by publish date remember that I need one field to be the first and most important sort order the remaining fields are basically TI Breakers I can also specify scripts elastic search has a language they call pain L which allows me to specify basically formulas that I can use to calculate so this would be the price with a 10% discount doesn't make any sense as a sort order to be frank because the sort doesn't actually change but you get the general idea that you can pass in a formula here I can also sort by those nested objects and Fields so if I wanted to sort by the highest review I could do that as well elastic search will allow me to specify the sort order very rigidly if I want to I can tell it exactly which fields to order on or I can use that underscore score field that we talked about in the result set earlier which basically orders by relevance now relevance is a really deep topic there's some machine learning and statistics aspects to this but some of the simplest algorithms are actually pretty intuitive and easy to understand so the most simplest of them all is one called tfidf term frequency inverse document frequency and there's two fundamental observations here first the most relevant document probably contains my term more frequently if I'm searching for elastic search I want it to mention elastic search a lot the second thing is I'm going to favor those words that don't appear very frequently across all my documents so as an example if I'm searching for elastic search guide and I've got two documents one is a short document which is functional and elastic search guide and then the second is a summary of a court case about some weapon being handed to a guide and the prosecutor using the technology which one should I prefer they both contain elastic search and guide well the first term frequency is going to tell me that elastic search appears very commonly in this document so it's going to get a score boost just by the number of times that elastic search appears the next thing we're going to note is that guide actually appears in a lot of documents it not only appears in my elastic search guide it appears in court cases it might appear in a bird watching guide and as a result it's a relatively uninformative term so what tfidf is going to do is it's going to score those items which contain the keywords that are densest and don't appear in many other places higher than those keywords that maybe don't appear very often or appear in almost all the rest of the documents in my Corpus the default scoring algorithm is not actually tfidf but it's a close variant and it's a good enough approximation you can assume that elastic search is basically making some statistical inferences based on your documents if you don't specify a specific sort order now that we filtered and sorted the results we're almost ready to send them to the user but we've got one last step which is we might have millions of results that we could potentially return and most people aren't going to scroll to actually see all of them it be madly inefficient so we need to Institute some sort of pagination there are two types of pagination stateful and stateless stateful pagination requires that the server keep track of something so that way it knows what the next page is usually you're talking about cursors here in stateless pagination we're going to pass in some sort of parameter that allows us to localize approximately where in a result set we'd like to look so as an example this from and size query is simply going to take results starting with the index of from zero and pages of size 10 it's going to take the first 10 results and return them to me when I query again for the next page I'm going to increment that from to 10 I'm going to keep the size the same and that will give me the next 10 for a total of 20 very straightforward I can also use this search after so in this case I'm sorting by date and ID when I get to the bottom of my results I am going to have a date after which there are no more results in subsequent pages and an ID after which there's no more results what this is is telling elastic surch is it saying don't return me any results that might be before these two quantities these are both comparable but search after is preferable particularly in the cases is where you want to go deep into the pagination because this requires that me get all the results and then we're going to slice into that result set based on the inputs whereas this actually allows me to restrict the documents that I'm querying based on this search after after I've gone deep into the result set I may actually have a much smaller Corpus of things that I can search over and so elastic search has to do less work it's a kind of clever idea now both of those those search types have a problem which is the index is constantly changing new documents are being added some are being deleted in the worst case scenario you actually have a document which would have been on the next page but changed maybe due to a deletion and bumped up to the page that you were just on if that's the case then your pagination whether you choose search after or the from and size is going to miss a document so how do we deal with this elastic search supports a point in time search and the idea here is I can create a point in time where elastic search is going to snapshot what the index looks like at that period this is going to return me an ID in this case 460 and I can use that point in time while I'm searching note I need to specify a keep alive this is actually pretty expensive to maintain and so I need a signal to the server that I still want it for one more minute every time I search but what this does is it searches based on that snapshot and will guarantee that each page of results is consistent when I'm done I need to delete that point in time or let it expire so that I'm not using excess memory on the server but this eliminates at least a small class of problems that you might have with respect to pagination how this point in time is implemented is an interesting point of discussion for the next section of our video before we move on I want to Briefly summarize how you can use elastic search in your system design interview typically elastic search is going to be for complex search scenarios if you need to do geospatial indexes if you need to do Vector search if you need to do simple full Tech search like we've described elastic search is probably your friend it doesn't necess NE necessarily mean that elastic search applies in every scenario and there are some potential potholes that you need to be aware of first of all elastic search is probably not going to be your primary database while the API that I described looks a lot like a typical document database elastic search's durability and availability guarantees aren't actually that good and over time most people have relied on setting up a primary data source be it post grass or Dynamo DB and then using change data capture CDC to basically push updates from that database into elastic search the idea here is elastic search is going to be a little bit behind your database but it will have all the updates the next thing is elastic search is going to work best with read heavy workloads if you've got a lot of Rights happening or if your rights dominate your reads you may want to consider whether elastic search is appropriate there's a lot of ways that you can change the nature of your workload maybe limit the number of Rights or batch them out but generally speaking elastic search is going to perform best when we're reading a lot of data if you're using elastic search you have to be able to tolerate eventual consistency it's going to have a delayed version of the data eventual consistency is basically endemic to the design of elastic search if you need strong consistency if you need to check for double bookings or things along those lines you're going to need to use a different database but if you're not using it as your primary database you already know that denormalization is really key to making elastic search functional if you've got a particular search page make sure that you flatten all of your data and that you're not trying to make nested relational queries you basically shouldn't need to do joins in an elastic search search if you do you're going to be in trouble first of all because joins aren't supported and secondly the way that you might bolt on joins by doing lookups outside of elastic search is really going to compromise performance and the last thing to consider before you use elastic search is whether you actually need it elastic search clusters are not a trivial matter to operate and maintain it's a new technology in a lot of cases a simple search capability in your existing database might cover your needs if you need full teex search if you need to index billions of documents elastic search is your best friend but if you've got simplistic needs aim for Simple Solutions you don't necessarily need to dig in so far we've talked about elastic search primarily from the angle of a user or a client and from an interview perspective this is still incredibly useful in a lot of interviews what you need to be able to do is to point out when you might use elastic search and how it might fit into your overall design if you're designing Twitter search might be a peripheral concern for your overall design and delegating that to elastic search is a completely sensible design choice in other interviews either because your interviewer explicitly tells you don't use elastic surch I'd love to see how you implement some of the Core Concepts or because they tell you to implement a system which seems a lot like elastic search being able to understand what goes on beneath the covers of elastic search is going to be helpful and that's really what the second section is about now the first thing that you need to understand about elastic search is that it is in some sense an orchestration layer for Apache Lucine which is a low level search functionality elastic search handles the distributed systems aspects the API how to manage nodes how to manage backups how to keep availability and replication whereas Lucine is trying to make sure that the data is organized and that it's fast Lucine operates on a single node elastic search operates at a cluster level and so as we're peeling back the onion of elastic search we're going to be progressively talking about the different layers inside of elastic search before we talk about some of the fundamentals of Lucine the first thing to understand about elastic search as we kind of peel back the layers of the onion is the various nodes that make up a cluster now an elastic search cluster can have a single host or server node is not necessarily a physical machine and a physical machine can take on the responsibilities of multiple nodes but at a conceptual level these are the respon responsibilities that exist inside the elastic search cluster that need to be performed by some Hardware somewhere the first not type is our Master node or our Master eligible node and what happens here is when we spin up our cluster we are actually specifying seed nodes or seed hosts which elastic search will try to connect to and then we'll elect a master of the cluster we only want one master node so it can make administrative decisions like when we create indexes or which nodes are actually part of the cluster but the important thing about master eligible nodes is that there is only one master that is elected at any point in time the next node type is the coordinating node and you can think of this as kind of the API layer to the cluster the vast majority of requests that are coming into my elastic search cluster are going to be search requests and what cordinating nodes are doing are they're taking those requests parsing them and then passing them to the relevant nodes that have the data that we need in order to return the search results users are for the most part interacting with these coordinating nodes data nodes are where our indexes are stored they are functionally where we keep our documents and they're probably the most fundamental of all the node types we're going to have a lot of them data nodes need to have really heavy IO either lots of memory or plenty of fast dis because the coordinating nodes are going to be sending search requests and they expect the data nodes to be able to respond the ingest nodes are like the backend for our service it's how the documents make their way into the cluster and finally the machine learning nodes are not always used but are for machine learning intensive tasks and the idea with the machine learning node is these nodes might require access to a GPU and are going to have very different access patterns now each of these Noe types by virtue of their responsibilities actually imply different Hardware my master nodes need to be incredibly robust and reliable it's very important to me that they're not going down my coordinating nodes are going to do a lot of network with network traffic with the outside world my data nodes as I mentioned are actually going to have a lot of uh dis IO or need lots of memory and my ingest nodes are going to do a bunch of analysis of documents they typically are more CPU bound and so what this implies is when I set up a cluster I might actually choose different server or VM type to take on these node responsibilities in order to optimize my cluster remember that the fundamental grouping of elastic search documents is an index this might be my books index or my reviews index inside of an index I actually have shards and replicas and these shards are actually going to be responsible for both containing all of my documents but also building up index data structures to make searching fast the shards are mutually exclusive if I've got a document it will be assigned to one Shard or another and the replicas are basically going to be carbon copies of those shards I might have for instance a replica of a Shard across multiple machines and what this allows me to do is when I'm searching my coordinating node can choose whether it wants to use Shard 1B or Shard 1 a and spread the load across them so by replicating these shards I actually gain throughput particularly on the read Side by sharding I can spread out very large indexes across many different machines I believe A Shard has a hard limit of two billion documents which is a whole lot but even if I've got a smaller number than that I still may want to divide those documents either because the aggregate size is too large for a single machine or because I want to spread the load across my cluster to make my queries faster this this is the central idea of elastic search is that these shards are breaking down my data and inside of each Shard is this Lucine index the Lucine indexes are one to one with elastic search shards basically A Shard is just encapsulating a Lucine index inside Lucine index we have segments and the way that Lucine works is basically by accumulating these immutable segments what happens when I create a new document is Lucine will first try to batch that document into as many rights as it can contain before flushing it out to create a segment with multiple documents when I add more documents later I might create a new segment and then at some point in the future Lucine may decide that it will merge these segments into a third segment and then delete the old ones the important thing is that these segments are immutable they're not changing now we support updates in elastic search how does that actually work from a Lucine perspective well with updates what we're doing is a soft delete so Lucine inside these segments maintains a map of deleted documents so if segment two contain document one two and three and I deleted document two this segment will actually store that ident ifier two internally and then when I query that segment I will remove any results that contain two it's been deleted when I do an update what I'm going to do is soft delete I'm going to mark that document as deleted in the old segment and then create the updated document in a new segment this maintains this immutability aside from that map of deleted documents these segments aren't changing and it's one of lucin's more powerful features because it enables us to exploit a bunch of caching and concurrency benefits since segments aren't necessarily changing when I load them from memory I can do that in a compressed format and I can load them all at once I can guarantee that they aren't changing underneath me I also can have multiple readers touching the data in the segment in a concurrent way without worrying about race conditions or contamination the segment data won't change it's readon in some respects and so this Lucine idea of maintaining these immutable segments is really key to both the performance of elastic search it allows it to be very fast but it also puts some limitations that we discussed earlier on being more read heavy and not being able to deal with updates or insertions as quickly as maybe other systems that might allow mutable data inside the Lucine segments we need to store all of our documents conceptually you can think of a segment like containing documents but the important thing is that those searches need to be fast it's not enough to just dumbly store our data in these segments we need to make sure that we can search quickly Lucine employs several techniques in order to pull this off one of them is a reverse or inverted index let's talk about that briefly so when you want to make things fast with computers you've kind of got two options first you can consider how you arrange the data in memory if you store an array randomly you may need to scan over the entire array in order to get the thing that you're looking for if you sort it you can find an element in log and time if you know exactly what you're looking for if you want to do a lookup you can do that in O of one time with a hash table so changing how you organize the data is one key the next key is being able to use copies if we can make a copy of our data and store it in a slightly different way then we can potentially make access to all of the data really fast now of course we trade off storage there's a bunch of costs that are associated with it but those are really the two options for how we can make systems fast so the very first data structure this inverted index is basically making both those trade-offs what we're doing here is we've got a set of documents the first one's says Lazy Dog it's ID number 12 and then the second is ID 53 and it says lazy days of the best day in my index what I'm going to do is store a dictionary of the words that are in the documents to the identifiers of each document that contain them and what this will allow me to do is I can very quickly in this index look at all documents or all yeah I can look at all documents that contain the word lazy I simply go to the lazy uh index index I can do that with a hash lookup and then I can grab all of the documents that it contains now there's a bunch of complexity that comes into these indexes for instance I might have variants of lazy that I want to be able to include I might have other ways of spelling dog that I want to collapse together and certain words like R may appear in almost all documents and so maybe we don't want to include them this is fundamentally a question of tokenization or stemming or litiz and we won't go into too much detail about this but this is an important consideration in order to make these indexes quick now my inverted index gives me a series of document IDs but my search query might actually have a sort order maybe I want to sort by Price how can I actually make sure that my results match that query the next step for us is to think about how the data is actually arranged in memory there's fundamentally two different ways that you can think about how to organize a table of information like what we have here the first is what you would do in a SQL database each of these rows would be stored basically contiguously and you would have a bunch of structs or or fields that are going to represent each of these inputs so as an example this is a piece this is a piece and this is a piece my table is an array of these rows now this makes it really easy for me to get the entirety of a row given an identifier so if I know what my ID is and I can look up the the row uh index then I can very quickly pull all of the data for that record and in SQL databases this is a very common operation but what we want to do in this case with our Lucine segment is we want to take a bunch of Doc IDs and resolve one of the fields we for instance want just the price we may not even care about the title and the author in our query and so in this case what we want to consider is a columnar store in this case we're actually going to store the data as a bunch of columns that we then concatenate together so instead of thinking of the table as rows that are sitting on top of each other we'll think of it like fixed length columns and what this allows us to do is it allows us to get a single or a small number of field values across all of the documents very quickly if I have a large number of Doc IDs I can grab all of their prices without iterating and crawling over the entire table grabbing each row even though I don't care about title and author in Lucine this is an index called doc values and Doc values contains these columnar fields that allow it to do very quick sorting and filtering based on those doc IDs that are identified by the inverted index now while Lucine has some very low-level optimizations for how data is organized to make the queries as fast as possible there are also other optimization opportunities that exist inside of elastic Sur one example of this happens in the coordination nodes when a query is received by a coordination node it is first parsed and then there's an opportunity to optimize that query what happens is the system will basically plan what data it needs to access in order to get the results the user wants and try to optimize that plan in order to make it as efficient as possible now what might be efficient or inefficient let's let's pretend that we've got a query that has two keywords Bill and NY and in our index we have 1 million documents that contain the word Bill and 1,000 documents that contain the word NY there's a lot of different ways that we could execute this well first of all what we could do is we could grab all of the documents that match Bill all of the documents that that match nigh Union them together and then do a string search through all of their contents this is clearly very slow and it would give us the correct answer of all the documents that contain Bill and NY together but it would require that we transfer the entire content of the document across the network which is just a non-starter another option is for us to use these IDs so we could go in query all of the documents that contain the ID for Bill and we could intersect that with all of the IDS for NY now this number is necessarily less than a thousand so a lot of these IDs aren't going to match but the IDS that have both Bill and nigh are going to be matches to our query that sounds pretty good but we can do even better than this for instance if we start with n we don't have to transfer a million document IDs over the network we only have to transfer a thousand document IDs over the network and we might be able to do even better by going finding all of the documents that contain n and then searching through their Contex to see if any of them contain bill this is a much bigger number and if the contents are small enough we can simply do an aggressive search across a very small number of documents so this is the essence of query planning that there are a lot of different options that give the same result but have very different performance characteristic that we get to choose from when we try to execute a query and on the coordination nodes we have this option to generate query plans and then using St statistics that we have on our index like how many documents match a particular keyword we can then go and decide which shards we want to access which Lucine segments we need to go in query and how we can get the results back to our users as fast as possible all right so we've gone super deep to understand how Lucine works at a low level to answer search queries as well as optimizations that can happen on coordinating nodes to do query planning and optimize the ex ution plans let's take a step back to walk through the entire layer of the onion to talk about how documents are ingested into an elastic search cluster and how they're made searchable so the very first step is to create a document in a typical setting a client is going to send a document to an ingest node which is going to run an ingestion Pipeline and process that document before passing it along to a data node once that data node has received and index the node it's going to send an acknowledgement back to the injust node which can be confirmed back to the client so they know hey my document's been received on that data node we are going to add that document to a Lucan index in most cases that's going to result in the creation of a new segment if we've been adding a lot of other documents there may be a buffer and a segment might be created with multiple documents now now when it comes time to search our clients are going to contact the coordinating node the coordinating nodes are going to parse the query that the user has submitted they're going to make a plan for how they can access the data basically which shards it resides on and then they're going to submit those to the relevant data nodes the data nodes are then going to pass that query to their Lucine indexes which are going to run in parallel across the relevant set segments and utilize features like the inverted index and the doc values to both filter and sort the results before returning it back up the stack the data nodes are then going to return those partial results to the coordinating node which merges those results and returns them back to the user so we've got parallelism at multiple levels you could potentially have multiple shards s that are being queried we could potentially be sending requests to different replicas and inside the index we can parallelize the operations across multiple CPUs thank you for sticking with me through this video we've covered a ton of ground if you've got a highlevel interview coming up either full stack or product architecture I hope you can use elastic search in those instances where you need high performance search for complex scenarios that really require immense scale if you've got an infrastructure style interview coming up I'm hoping that learning about elastic searches internals and how Lucine works is informative and build some inspiration for some future designs that you might have I'm happy to answer any questions that you all might have in the comments and would love uh any reactions or ways that we can improve this um in general we put out lots of great content on this channel so subscribe to us if you're interested in more system design or more interview specific content and until next time I'm Stefan talk to you later