and so I think a lot of this is you know the game that we're playing here if we get back to like what we talked about in fundamental software architecture these katas you know that's exactly what the system design interviews are it's this hypothetical scenario and can you figure out the like Jiu-Jitsu way of cutting through the ambiguity getting to the core of the problem you're trying to solve and then come out the other side something reasonable out out of a proposal and collaboration [Music] hey there welcome to book overflow this the podcast for software Engineers by software Engineers where every week we read one of the best technical books in the world in an effort to improve our craft I am Carter Morgan and I'm joined here as always by my co-host Nathan tupes how you doing Nathan doing great hey everybody well we are excited uh for the episode today you got to change a scenery Nathan for any of our video listeners yet I'm at a conference in Denver so that's what it's not too far from where you you live right I get to hold my mic phone today instead of uh using a fancy stand for it no it's not it's uh it's only like an hour from where I live but I was like the commute back and forth traffic's terrible and yeah company provided a hotel and I was like I will take it I'm I'm jealous of uh I go to Toronto for business sometimes we got a couple Engineers that live there and like they just drive to Toronto it's really nice you know um well we are uh we got a new book for you this week um you know as always like comment subscribe wherever you're at share share the podcast helps us grow um but the book this week we're very excited it's a system design interview by Alex zuu uh you might know him more as the creator of the wildly popular newsletter bite bite go we're gonna talk a bit more about bite bite go uh we have lots of nice things to say about itth so we'll just give the introduction here um but a little bit of background on Alex Zoo he's an experienced software engineer and entrepreneur and he's previously worked at Twitter apple and zanga he was Master Zinga is that it yeah that's one of those words you only ever like read I knew a guy who worked at Zinga that's why I knew how it was pronounced um he received his masters from Carnegie melon University he has a passion for Designing and implementing complex systems and that's great because that's what this book is all about uh the book introduction we get says system design interviews are the most difficult to tackle of all technical interview questions the book is volume one of the system design interview an insiders Guide Series that provides a reliable strategy and knowledge base for approaching a broad range of system design questions this book provides a step-by-step framework for how to tackle a system design question it includes many real world examples to illustrate the systematic approach with detailed steps that you can follow um very very interesting book a very dense book we've been enjoying it uh we read the first half for through chapter nine we'll talk maybe later on the podcast but next week we're we're going to take a quick break from it uh I have an on-site and just my uh my my schedule's kind of crazy so we're gonna do an essay to save some time but then we'll cover the next half after that all that to say just the first half this week um Nathan what are your thoughts after reading the first half of a system design interview yeah I thought this was a it's a great format if you're preparing for a system design interviews and I think when you when you are in that point in your career if you're looking to switch it's a slog you got to do a lot of prep work oh yeah um this is the kind of density that you'll get a lot of value out of um I didn't think it was easiest to recover to cover of course we talked about that before it's our fault because it's the format of this but I I felt myself nodding my head and going oh yeah I hadn't thought about bringing up that point you know like I I felt like I would do pretty well in a lot of these system interview questions but I was like oh yeah that there's like that 90% And then like the 95% good answer and at last 5% can be the deal breaker you know when you're interviewing so yeah I thought I thought it was a great but what did you think about the format I've been recognizing that too where like I'll be with him like 90% of the way like oh yeah like I I know how to do this and then there's there's like a little bit of stuff he brings in at the end I'm like oh you would look so smart if you brought that up so that's it's been really cool for for that um very good mix of depth and breadth we've talked about that on the podcast we are kind of just by Nature a breadth podcast that's what happens when you uh read a book every two weeks or so um I mean I wish I had more time to deeply study certain chapters I you know we just have to keep up a pretty fast pace here and there were ones where I'm like oh I got to come back to this um I mean it is what it is it it it's uh you know the guide for prepping yourself for system design interviews and I find it essential I mean until this point uh I've always recommended cracking the coding interview for system design interview questions and it's a great book too and and it's great for a lot of reasons um this is like that on steroids um I also like this book because it congratulates you after chapter that and has a nice set of references like that again I appreciate lots of references um he's like hey you made it this far good job I'm like I did make it this far Alex thanks um so yeah this this has been a fun one uh we mentioned the podcast before we're trying to find the right balance between Technical and career focused um and even non-technical right we've kind of run the gamut so far this year we have software engineer guide book that's very career focused we have team topologies which is like obviously team focused uh and then this is very very technical along with Hyper media systems so um yeah it's it's fun to to get another one of these very technical books and we're excited about all of it uh we're not going to spend too much of this podcast talking about the nitty-gritty details of all the different system designs you'd be better served by reading it but we're going to talk about a lot of the uh system design stuff and just system design interviews in general and and um bite bite go so we've got a lot to talk about this episode so stay tuned we'll be back with all that and more and we are back thank you so much for listening so let's talk just a bit about uh Alex zoo and his newsletter bite bite go now Nathan you are more familiar with this than I am I know of Alex I know of bite BYO but I am I'm not a subscriber I should be yeah no this is great so where you know there's a lot of great like growth engineering type uh substacks and newsletters that are out there especially for career development this one is this one's very focused on like how does Netflix scale to you know a trillion transactions and gets really into the architectural designs and I'll say that by by go is an amazing resource you've maybe even seen he's gotten kind of famous with these sort of like animated gif data flow things that if you look at link his stuff goes viral all the time um what's nice is that his viral content that's really consumable in newsletter is not just a regurgitation in the systems design book oh okay um it's so they're they're like complimentary and I think it's it's nice that if you need to go into these deep deeper understandings of like how do I actually Ace this systems interview this book's fantastic but byebye go I I highly recommend giving taking a look at it because again I I love to know how things work or how does a how does a tech company like solve certain types of problems and they they pull no punches it is I would say it's the systems design equivalent of Lenny's newsletter which Lenny's newsletter is a um product oriented newsletter the guy who came from Airbnb um fantastic resources as well especially if you're like doing product design or product engineering um byby goes up there with that and I think they're huge like last time I checked they have like over a million substack subscribers that's crazy that's just like a that is insane yeah that's it's um I think as we're doing all of this you wonder kind of is it all worth it right like this continued education you like I was like where you taking this you're getting get existential for a second Carter think about life lately no um I think about that with the podcast I think it's no secret with the podcast right that this is It's a non it's a non in substantial uh reading commitment right and there's definitely been nights before we record where I've had to read 90 or 100 Pages you know because I fell behind that week um and and you know you sometimes wonder like oh man we're reading so much are we doing are we even getting anything out of this and I think the reason I bring this up is because um I think that applies to a lot of continued Computer Science Education and by B Go is is a great example here um and you kind of wonder how much of a Competitive Edge of am I really getting I just want to give my take on it which is like I think we are gaining a substantial Competitive Edge and if you don't like to think of it as a Competitive Edge I think we're just learning a lot I've actually been um kind of working on like a Greenfield project lately uh on my own and I have been shocked at how much kind of latent Knowledge from all the books we've read is immediately becoming useful in designing and building this thing um y so I think uh you know if you're not willing to be sickos like us and read a book you know or 200 Pages a week or whatever seek out good resources like bite bite Go I mean that it's going to be a big boost in in your career and how useful you are as an engineer and I would say it's a this is an excellent resource that's a compliment to the pragmatic engineer uh content as well where you you do need a balance of career strategy and engineering Excellence learning strategy and they both you can't just be the systems expert you really do need to know how to like also incrementally build this in a way that can deliver value to your team right you have to do both yeah I think the systems interview is a sort of dance right that you have to you have to go through with the interview process that you might do a coding interview increasingly I've had amongst my friends especially as you move up in seniority in the interview process they expect a solid showing and a system design interview absolutely um you you have to understand why you're building something what you're building how you're building how do you get to the requirements of it um and this book does a really good job I think there's a there's a whole chapter dedicated to like just how to gain Clarity you know you like you have a one hour window typically 45 minutes to an hour they're going to give you some technical design that you're going to have to come up with and you basically have to go from I have no idea how to implement this to getting nailing down some of the requirements and what are not requirements because you can assume all kinds of complexity in your brain and the game really is how much complexity can I strip out and then show that I understand how to build a system like this in a 1 hour window right that's not real life you're not going to be expected to do that but if you know how to play this game and you know these patterns it's not different than knowing the time complexity of an algorithm that you're doing for some technical interview right um it's just now you're dealing with like distributed systems or hashing algorithms or databases or caching or you know any one of these topics that might come up and um so yeah if you if you take a positive view to like okay this is a game I'm playing well understand the rules of the game you're playing you may not love that you have to play this game but know the rules and you can do the best version of what your knowledge has to offer right and um yeah most of our listeners probably understand what a design interview is but I'm always surprised breaking out of the demographics we have a decent chunk of people who are like 18 to 24 years old so lots of got college students and just in case you're not familiar syst interview is different from a coding interview a coding interview right you're presented with one coding question right you know the famous one is invert a binary tree or reverse a string or whatever um system interview much broader right and that's where you'll get questions like design Instagram um the examples we have here in this book are like design a rate limiter design consistent hashing design a key Value Store design a unique ID generator and distributed systems design URL shortener all sorts of things like that um I I actually really like system design interviews I like giving them and I like doing them I mean no one likes doing interviews but out of all the interview types I think it's my favorite um me too you get a lot of points and system design interviews in the how you approach them and not not just the what you're building this book is all about the what you're building right it has lots of really great patterns for building these specific types of systems um but one thing I think is cool about system interviews is you get a lot of credit just if you can explain your ideas well you get a lot of credit if you can ask the right questions you get a lot of credit if you're engaging your interviewer throughout the process um you can't be a total Bumpkin and like not know anything but you can really plus up your performance by um having some good bedside manner so I I like as someone who uh who hosts a podcast and can ramble for an hour every week no problem um this interview plays my strengths yeah and and I I thought the book did a good job of I mean again these are sayane defaults these are not way the way you have to approach this um one of the things I liked about the structure of the book is they kind of go instantly into a design interview question which is like scale from zero to a million users without giving you a real structure of how to answer this question they kind of walk you through like things you need to reason through and right and so from a scale from zero to a million users they you basically have to design a very simple system and you go okay well if I hit the scaling threshold I'll you know I'll add a load balancer and get to this one I might need read replicas on my database um they also have this idea of a a framework for approaching the system sign and they get to that by chapter 3 and actually they use this every time they build on something they use it for the rest of the book right and I thought this this the four-step process I thought was useful and again this is where the value you get out of this where it says you know understand the problem and establish the design scope um propose step two would be how propose a high level design and get by in I can't tell you how important that step two is where you're basically fishing right you're saying hey you know I could take this approach is this the direction that you kind of are thinking and they may go oh yeah you could take that approach and um go further or they might go oh no no no no you don't have to worry about uh consistent hashing in a distributed database or something you know like you that you can assume that that's done for you right well that takes a huge design component off your plate right you don't have to go off in some rabbit hole of talking about you know um hashing algorithms that will rebalance themselves after you know you take something out of a pool or you know some really technical thing that you could spend the whole interview on um they go oh no no no we have that solved focus on this part right if you could do that earlier in the interview just you're not going to go because what I've done this before right I've done a systems interview earlier in my career where I realized I was just barking up the wrong tree at the end of it right I I I I probably gave a very technically correct thing to I fixated on as opposed to solving the deeper problem that's something that comes with maturity of saying what business problem are we really solving here like what are they really trying to get to um and then you have to do your deep design right like most of the rest of it is you're going to be expected to dive in deep to what the two of you have kind of agreed on in the interview and if you picked the right direction the interviews are awesome they're fun you get to collaborate you get to like kind of show your knowledge um and then you have to wrap it up right at the end of it you go okay well um you know with this this and this and this I think you know and they talk about back of the envelope they're like go you do some back of the envelope calculations I think this would meet their criteria that we talked about at the beginning um with this design that we just did right that's a I mean it's also like the same format of how to write a good essay right but I thought this book gave us a good framework of like how to think through these things which is nice when you're nervous in an interview you need a framework um well and that skill you talk about about like um get byy in early like that is that's skill you need at work oh my gosh yes absolutely that you know that's something I've been working on uh we we are preparing to take our service from internally know what's known from tier three to tier two which just means that more people rely on it and so with that comes a higher expectation of um up you know of availability and reliability um and one thing you need to do to show that you're doing that is do uh they're called game days um I think that's like a standard industry term but right you need to simulating outages with the team right and then uh and working to fix them um you know I'm in charge of kind of planning and executing these game days and I was figuring out I'm like okay well in order to do this we either need to be able to run game days in production which is possible but requires a lot of tooling and ways to not break everything or we need to have a enough of a production like environment that running a game day in there will actually accurately simulate what it's like to do you know to to debug something in production and there's a lot of hurdles there neither insurmountable um between those two options but I was just uh I was talking with um one of the other uh staff engineers and just saying like what do you think between these two options he proposed a third one he's like what if we just like Dungeons and Dragons it right like just have like a simulation of what's going wrong and then we just have people you know like they say like oh I you know I'm I got to look at the graphs to figure out what's going on and like you're looking at graphs that are like normal they're not showing anything but as they look through them maybe you say okay you found the right graph and then you show them an image of what it is supposed to look like or whatever right um and and and we liked this option just because it's going to be on Tenth of the amount of work as either pursuing either the other two options but you know this is something I floated with the other tech leads and with my manager and was like hey what do you guys think about doing it this way instead and got buying from everyone like oh that's great you know you can imagine a scenario where instead even before talking with that original staff engineer maybe I embarked on a really big project to duplicate all of our production data and set up like a you know another production server or whatever and do all these things and then like okay great we can run our game day and people been like why' you spend three weeks on that we could have just done it like Dungeons and dragon style or you can imagine another scenario where I proposed where I'm like okay I got a game we're doing a Dungeons and Dragons style my manager is like what are you talking about we can't do Dungeons and Dragons game day um so getting that buying it's important that's cool I like that and exactly like get getting um we we recently had this I have I have a Greenfield project at work as well I have a team that I'm a small team that I'm managing and we initially interpreted some requirements in a way that was we were kind of getting down this Rabbit Hole of like oh oh man these are going to be really hard like I I just identified like these are there's lots of unknown unknowns like we just don't know exactly what we're going to we back to get clarity it's actually a project that has the attention of our CEO and we were kind of ideating through things and realized we could do this in a much simpler way for like a vzero you know like dential prototype and it drisk our project tremendously it took something that I was worried was going to take maybe three months or something thing to work on something I think we could turn around about a month and a half right nice um with some incremental stuff we shipping in between but it was really important because I like something again I was like worried about I'm like oh this is going to be really complex and how do we make it simple for the user we realized we just didn't need it especially early on like we were like no we'll just do a very simple version of this remove some of the optionality for the user which opens up optionality for us and we've talked about optionality um with k back uh you know we can put little placeholders in our code where we can add complexity later but the initial implementation can be very simple right we don't really expose this in a way that um makes it so that because we don't know how the user is going to use the tool all the way and if we add a bunch of features the users will inevitably start using features that we've given them and maybe we didn't want to give them those features right maybe we realize it's was kind of a halfway thought of idea and it's way harder to take that away from somebody than to just like be like here's a really stripped down version Oh and here's a new feature like add these out incrementally um and here's a negotiation right we couldn't just strip those features out if there was an expectation from our product team that something was there day one U we would have to accommodate that with the timelines and everything else but if you can get permission to say okay I'll do a naive approach on purpose right we'll do this so that we can incrementally add these things and then product people love it because they get to go oh here's on our feature road map and they can like roll things out and show that they're making Improvement so that's it is it is interesting um to figure out how how to go about an incremental or an evolutionary architecture design in the in the right way and that's a fun Dimension too yeah I'll say I think there's a good argument that the interviewing process for software engineering jobs is roughly broken I've talked about the podcast before but I defend it at the high levels like if you were literally Apple or Google or Netflix or Amazon or whatever and you have thousands of applications and you know you can afford to reject good Engineers because like for you the much riskier Al outcome is accidentally hiring a bad engineer um then you can do the whole leak code song and dance because you lose a lot of good Engineers by doing leak code but you rarely hire bad Engineers from it um system design interviews are I think the least broken part of the process you're right Nathan that there's obviously like you're not going to sit down and design Netflix in one hour um but these are I I think it's the most applicable mapping of skills um you know from the interview to the job and I I find myself engaged in these sort of system designs interview questions every day where I am never doing like leak code style coding on the job right yeah I agree it also gives you an opportunity as an interviewer to see how collaborative this person is and to me that's a huge one it's can I can this person own up on their weaknesses do they ask good questions are they able to navigate uncertainty these are like all parts of the job those are all real world things and if you can simulate this through a system design interview you get a glimpse of it it's not perfect right but you can be like oh this person jumps to conclusions and makes lots of assumptions and has already like tingled themselves up in this complicated thing and this might actually be their approach to life and that's going to be a hazard right and so I think a lot of this is you know the game that we're playing here if we get back to like what we talked about in fundamental software architecture these katas you know that's exactly what these system design interviews are it's this hypothetical scenario and can you figure out the like jiujitsu way of cutting through the ambiguity getting to the core of the problem you're trying to solve and then come out the other side something reasonable out of out of a proposal and collaboration um that's the art and that's what people are trying to see with your output from this right they're like oh yeah they actually they actually they asked some really good questions they really got down to the core you know typically you have on the interviewer side I'm just this is a good thing to bring up if you're going to read a book about how to Ace this on the interviewer side we typically have a rubric that says they should get to this point or they should hopefully ask these types of questions or ideally they ask clarifying questions to get to this design direction right at least the rubrics I've used in the past and we try to use rubric because we're trying to take something that's inherently creative and give a framework that people can succeed and we can compare candidates against each other sometimes it's not easy to compare apples to but um have empathy for the interviewer as well right they have they remember this is a game that they have on their side if you're asking to design a rate limiter well you should understand the expectations of rate limiting you should understand maybe three or four prominent designs and approaches that big tech companies have taken to doing rate limiting um the tradeoffs that are there right uh these are the kind of things that if you want to be prepared for your interview not just completely derailed you just you need to know and that's you can't make that up you you don't know necessarily like if you don't know how Google does rate limiting or something or an innovation that flicker came up with right like um and that's become the prominent pattern for something um you'll you won't be as prepared you might come up with a kind of creative answer but if you don't know like oh that there's this cool you know hashing algorithm that servers can use that don't have to coordinate or something right um that just shows your knowledge versus having to make something up on the fight if yeah go ahead no you go ahead I was G to say and if you don't know please don't lie like oh y see oh gosh I I hate it when I have like a candid I'm like resonating with and I'm like oh they just bsed me like yep because I would much rather you go you know what I haven't worked with deep down rate limiters in the past but I think that this might be how I would approach it um I would much rather that even if I might be in the moment a little disappointed that you don't know that much about rate limiters I would then at least go okay well you don't let's see how you navigate this like okay that's that's interesting like tell me tell me how would you approach this and um that's how you set expectations low folks I do that with with h coding questions I say now granted I've never used Java before but I'm just going to take a stab at it okay okay um I I want to talk about some of the specifics in the book but just before we move on to that I just want to give us a chance H any system design interviews you've done in the past that have stuck with you either ones you've conducted or ones you've done yourself o yeah I had to [Music] do I was interviewing for like a staff level software engineering position actually I think it was a staff level Sr position site reliability enging and we were looking at it was a it was a high availability design on dealing with a bunch of like a high throughput API server that they gave a bunch of like observability constraints and some other um operations constraints because you know with with Sr you're really really worried about SLI SLO and slas right as most people have heard of that's like the that's like the sort of business agreement of like how much uptime something has so maybe 99.99% uptime might be an SLA um but internally you have to Tool out your service level objectives and your service level indicators and so for an SRE you have to have a very good knowledge of how SLI slos and slas relate to each other because there's like basically you have like the legal requirement of what you're promising a customer but internally it's like well here's how we want to behave on a day-to-day basis and here's how I'm actually going to measure it right and so there the system design interview was like well how would you measure this how would you build this in and um it was a lot of fun it was actually like a collaborative whiteboarding session remotely which I is the only time I'd ever actually had to use a whiteboard um sometimes you'll ask to be to prepare stuff beforehand sometimes you you kind of collaborate through I I remember that one being like a lot of fun I also did awesome on it so like that that one stay Stayed at top of mine what about you yeah I'm glad you had one you did awesome on cuz the one that res REM that I remember is one I did badly on um he those are both those are both valuable lessons I know right so I'll say one of my current company we do a real deal systems interview we also do a great kind of interview that we call the technical Deep dive which is almost like a system design interview that you that you prepare for like you come with prepared materials like they'll say like we just want you to walk us through a project you've worked on in the past right and so that's really cool cuz I I showed up with like a whole architecture diagram something i' had built and then that I aced that one my current company I kind of aced our systems design and and that technical Deep dive and uh um yeah did really well there I think I was interviewing for it was a software engineer position I want to say it stripe um and it was just it was dumb it's a good example of how you kind of want to stick with what you know yeah um because I I I've been up front in the past that like sequel is not my strong suit um I've done a lot of work with like no SQL databases in the past just now trying to really throw myself in the deep in with SQL to understand it one of the things that always scares me about squl is um like no squl like you're using Amazon Dynamo DB lots of downsides with it but it's always going to work pretty much you can throw almost infinite amount of requests at it and it's just going to handle it no problem um not true with you know a SQL database and even if you're using something like AWS RDS right at the end of the day it's still a server node with your database on it um and I get really really nervous about uh bringing the database down right it it it winds up being a single point of failure for the whole application and so I'd gone to weirdly bizarre links in this interview to really minimize reads and writs on the database and I just yeah like I think it it was very much like the uh like the Frozen the Frozen caveman uh pattern or anti-pattern that uh we read about in fundamentals of sofware architecture like I'm sure they must have thought like wow this dude you know brought down a SQL database at one point now he is obsessed with it it was the opposite I've never brought down a SQL database because I never really worked with SQL that much um yeah I just went to really really Great Lengths to design this kind of convoluted architecture um really what I should have done and and if I just known more right um I I should have said uh I like I should have done some back of the envelope map we're going to talk about that in a bit but been like okay so how many reads per second how many writes per second what's the length of a record how much is going to be stored in there um and even after all that back of the envelope map if I had said like listen SQL not my strong suit I'm going to assume that we can have a sufficiently large sufficiently powerful SQL database that the amount of reads and writes we're doing per second aren't going to be a problem and that we have and and that uh and and then in this particular application I think I could have said like and will never exceed more than 64 terabytes of data um and said is that an okay assumption I guess in this particular example they would have said yeah yeah that's fine to assume and even they had said well we would like to talk more about you know what if there did get to be more data even though I put that kind of on the back burner and said okay that's fine let's assume that this one database can handle it for now I'm going to build out the rest of the architecture right and then come back to that um yeah I I was just bizarrely paranoid and um I think it didn't go well I didn't get any feedback that was like oh that went really really badly but I think it it for my opinion certainly not the stronger showing yeah no I I I I'm trying to think of where where maybe I and those are the ones where like yeah you're trying to fish early on and you're like okay is the hint that they want me to come up with a sharding mechanism or is the hint that they um that you know what I'm actually looking at this and even though you told me to do this in Sequel I actually think no SQL would be better and they actually see if I push back right like and you don't know because it could be no no no no this has to be sequel right like that's fine um but sometimes when you're fishing it's like can you find the the secret golden ticket that you're actually shooting for and In the Heat of the Moment sometimes you're just you know it's like you know going up and doing karaoke or something it's like you pick an awesome song and you're like I got this my voice vocal range it's a lot of fun and sometimes you pick a song you're like this this didn't land and I shouldn't have sung this and why did I do this and so um I think the same thing happens here you're like uh oh yeah with the with that system design interview the reason I remembered it because did really well they were really excited um I ended up not taking that job I got it was a Vibe check thing and I'll tell you even after all this stuff like it was a good offer fully remote um but I had mentioned I was in grad school because I was in or I was about to start grad school and they were like concerned that I was going to use some of my spare time like non work hours to work on this they like you might need to be available and I was like what like what are you talking about and then the other one was that I don't know just got a weird vibe that company um went bankrupt like nine months oh really yeah so I was it was like I dodged a bullet but I but just from a work culture standpoint I was like there was a something weird even though I even though technically I think I would have done been a great fit for the role it was like a really kind of interesting problem to work on it was in like i' done some vintch stuff before this is like a fintech company fintechs are fun because um they went Bank because it was like a they were in the crypto space and some one of their crypto Investments went belly up and they became insolvent oh no yeah so it was it was like they were making a bunch of money it's just you know if you lose a billion dollars or something it's like you can't recover from that so yeah it was interesting well let's uh take a quick break and then we like we said we're not going to talk about a lot of the you know finer details uh from all the different chapters please read the book yourself you know there's tons of great stuff in here but we're going to talk about some of the techniques he proposes and um some of the concepts we found interesting throughout uh Alex H's book so we'll be right back and we are back thank you so much for tuning in so let's talk about um some of the strategies he proposes uh these are things that pop up in all of the different chapters because every chapter is associated with one specific type of uh system design um we we talked about it in the first half of the podcast uh asking questions you want to make sure you're on the same uh you know you're on the same track as your interviewer you got the same Vibes going you want to make sure you're not barking up the wrong tree um so we at the beginning of each chapter he has a really great uh you know he just does like a little interview style like the candidate and interviewer can ask a question the interview or respond so I really like chapter seven which was design a unique ID generator and distributed systems I'll read just the uh The Prompt he says in this chapter you were asked to design a unique ID generator and distributed system your first thought might be to use a primary key with the autoincrement attribute in a traditional database however Auto increment does not work in a tributed environment because a single database server is not large enough and generating unique IDs across multiple databases minimal delay is challenging um so then he gives the kind of this back and forth between the candidate and interviewer like these are the types of questions you might want to ask so candidate what are the characteristics of unique IDs interviewer IDs must be unique and sortable for each new record is ID increment by one the ID increments by time but not necessarily only increments by one IDs created in the evening are larger than those created in the morning on the same day do IDs only contain numerical values yes that's correct what is the ID length requirement ID should fit into 64 bit uh what is the scale of the system the system should be able to generate 10,000 IDs per second so these are just some of the example question you know that you could ask but there's really really valuable stuff in here in particular this chapter because you might read this and like my first thought when it was like designing unique ID generator I'm like yeah it's a uyu ID like that's all you need to do but then there are you know yep there's yeah he says like well the requirement is 64bit uu ID is 128 bit the requirement is that IDs generated in the morning are you know that they can sort them right you can't sort you you IDs bya by time or even if you were to do a time stamp and then slap a uu ID onto it right now you're larger than you're already 128 is already too large but now you're larger than that um so just lots of really great great clar questions in this particular example and the type of questions you should be asking and and um I'll tell you I personally got obsessed with this for a little while so I was like oh yeah I would have done because I there's there's several like three there's three I could just on top of my head I can think of that are common he mentions one of them in this book so there's like snowflake ID which came out of Twitter there's also UL ID and X ID um and these are interesting because like they're sortable and a lot of them have these kind of cool features right you can name space by server so if you have unique identifiers for your servers you get in a unique name space they're sortable over time because they typically have some sub chunk of sortable time piece and it typically have some sort of like entropy guarantee so that if you have concurrent processes on the same server that happen to have similar timestamps um you can still like if there's a a race on your smallest increment of time right because you at some point you have to decide what's the granularity is it Nan seconds is it you know whatever um you can still have a little bit of entropy which is like some random number generator that kind of appends to the end um these it's a juicy problem and there's a coordination problem to it you always try to figure out and like this is where like this this section of the book explores all those things and these are fun like hopefully you find that fun otherwise you might be in the wrong field but um why you listening to this podcast yeah exactly so like I looked at I'm like oh this is awesome I like this uh but it's true if you're having to do one for yourself and you're not just like regurgitating snowflake or xid or ulid um or they give you something that's like inherently A variation on those uh just to see if you have enough under familiarity but you can because a lot of these two are you're like what are the pitfalls what are the tradeoffs right like we always want to hear these things too uh in the interviews let's talk a bit about back of the envelope estimation yeah I love that um this is good stuff right and I think this is something I'd like to be getting better at and I think this is something people Miss in system design interviews um you know talking about by back of the envelope estimation you can't know what system you need to design without knowing the amount of stuff it needs to do right designing a system you know a web server that handles 10 requests a day is Trivial throw that on your raspberry pie and plug it into your wall right um designing something that handles 10 million requests a day is way harder um same when we're talking about databases right you know if you're like me and you're really concerned about scalability um you need to you know no SQL can be a really powerful tool but it also just kind of depends on how big your data is I mean if you have a billion records but each one is one bite long that's very different from having a billion records where each one is 20 megabytes long right um and I think uh like reading that that's something I want to be better at like when we talk about like okay so I think the system interview I whiffed on was about a it it was about designing like a hotel system like a just you know for managing reservations I wish I had asked that like how much data is in a reservation you know how you know what's uh and then how many reservations do we expect to have how long do we need to retain the data um how burst is the data that's that's one will catch you up right even if you say 10 million in a day so like quick back of the envelope right if I want to figure out request per second I would do divided by 24 divided by 60 divided by 60 right that gives me down a request per second that actually comes out to like 115 115 requests second which is non-trivial it's a decent amount of requests per second but very rarely is that just like a consistant data stream of 150 requests per second you might have a burst of 600 requests per second for some period of time and then down to 10 requests per second and that changes your design right you need to understand what your upper bound is you need is there scaling stuff that has to go with this uh you is there if it's bursty this also might trigger oh maybe we can use a queuing system because that's a really great way of evening out um bursty traffic and lots of great stuff that comes out of it yeah um what about you Nathan any thoughts on back of the envelope math any of the I love it yeah yeah this is one where I think we all can probably do better like I think what ends up happening is if it's in a domain you've done a lot of work in you get pretty good at doing back of the envelope with oh request per second or something right um but one of there was like a couple of other areas that like you need to be really familiar with this um I don't have this memorized I I do understand that like you know if I can do it in L1 cach versus having it go to um memory references to going on to dis right like that's the order of like speed can I keep it in a um you know heaps versus Stacks there's all these like kind of cool basic algorithm things um I I would probably if I was going to go to like a thing interview I'd probably memorize these yeah ballpark time because they have a whole table here it's like oh 0.5 Nan is an L1 cache reference you know if you want to do something that's pretty impressive for back of the envelope just memorize these um these times that that go to like dis seek and things like that and just tell them like hey I'm just going to I'm going to say that disc seek takes 10 milliseconds right it doesn't matter Hardware changes nvme versus block storage on you know network storage or whatever those are going to make big changes on that but if you can get them to agree to the assumptions that you're making you're like I'm going to do a back of the envelope calculation this is what I think the unit is per read this is what I think that and they can agree with you or they go oh no no no I think that's too slow that's too fast or you can assume that whatever um you you'll just come off looking better um I think another area that they talked about was slas um like understanding both again they have a great one was like you know 99% availability means you can have 15 minutes of downtime a day right and that's one of those like I just have memorized because I was Sr for a long time uh and so when I'm negotiating 939 rate 99.9 and that's a minute and 44 seconds I don't have that perfectly memorized I knew it was under two minutes but that's two minutes of downtime a day if you ask for four nines that's 8.6 seconds per day right that's a much different conversation of like what's acceptable downtime going from eight seconds and that's a violation to uh a minute and a half you got a lot more wiggle room you can do a lot more it's a lot more forgiving to to to get a minute and a half versus you know eight eight or nine seconds well and uh you know memorizing this sort of stuff and being able to do these you know back of the envelope calculations it's all about what sets you apart from uh your the other candidates you're you know potentially competing against um I've I've made a promise to myself that the next time the market is hot again I'm going to I'm going to make hay while the sun is shining um just because uh you know it's a the Market's is a little Frosty right now I don't think it's as Doom and Gloom as people say like I you know I had to get a job a year ago you know I to switch jobs and I got plenty of interview offers you know um but it's true that uh expectations for interviewers are a lot higher than they were in the past um I think it's funny like Amazon is kind of famous for like they've got the hiring bar and the hiring bar is always moving up it's not always moving up the amazon is not immune to the laws of supply and demand um and uh you know I think a lot of companies are looking for higher caliber Talent these days just because they're able to be pickier this is one of the ways to signal that your higher caliber talent and I will say it this really does squeeze folks that are earlier in their career raate if you're lucky enough the to have got some experience in your especially if it's working on problems that are interesting to other companies you are marketable even in bad environments if you are a recent college graduate with no internships and you've never worked in a startup or you know Enterprise type stuff before well that company knows it's going to be there's a lot of training that they're going to have to do and there's a lot of stuff that you just never have solved these types of problems the best way to maximize this is to actually build personal projects Rel to these systems design interviews and just get good at doing these you can you can't overcome it we just hired a recent college graduate actually mathematics undergrad but who is just a great programmer and helped solve the types of problems that we're looking at it's not common a lot of times we are hiring folks that are not Junior Dev but when we identify unique talent we grab it right like it's it was one of those where we're like we're going to learn something from this guy while he's learning a lot from us like you know it was I was never one of those people so you know I I look at this and I'm like this is a really cool situation to be in but uh yeah and um ass system design building these sorts of things on your own especially if you're a new grad or even if you're you know not uh like you said Nathan it's a great way to set yourself apart I love you know like it there's no doing well in job interviews is kind of similar to like being healthy and that there's not a lot of secrets to it right there there's no tricks it's like I know what I need to do to be healthy and every we all know what we need to do you got to exercise you got to watch what you eat you gotta you know uh you got like there's not a lot of tricks to it right it's just hard it's hard to do those things um but yeah but but yeah I think it's it's same thing of like well if you preport to be a you know nutritionist level knowledge then show me like give me a meal plan for the day what are the considerations like I'm trying to get stronger what what's the protein intake I have if I'm this height and this much weight like that's a technical interview for a nutritionist right like that would verify whether they actually know what they're talking about versus somebody like me I'm like ah me this is what I've been doing you know exactly right I lost 60 pounds eating Taco Bell every day that that was my yeah that's amazing I did not I did not do that I don't recommend it um it's good if if you were literally only concerned about losing weight and if you're concerned about like muscle mass or any other measure of Health it's uh you'll just be sickly and frail um but in terms of like uh kind of the same thing like getting your resume picked up like there's no secrets like you just need you need to set yourself apart you need to look different from the other candidates right as you get more work experience it's going to come as a result of the places you've worked at and the things you worked at at those places um but when you're a new grad like I'm always telling especially people looking at internships like you know what I'd love to see on a resume I'd love to see anyone who did a personal project that used AWS in any meaningful capacity yep right like anyone who's built out even the the the semblance of a system I'm always trying to sell people on a if it were me and I were trying to get something picked up I would do like one of those um Tech based dungeon games you know like uh you're given a a scenario and then you you know you're given like four options or whatever and you just choose one because what love about that project is it can scale up really well like it can start just running in your terminal right but then you could restructure it all as a rest API and I will say since we ride hyper media systems I like should I call it a rest API or is it a Json data API gon call the rest API um Carson will get furious um and uh yeah it's uh like then then it can be arrest API and then after it's arrest you know okay well can you put it up in the cloud can you host it in AWS can you instead of you know communicating with it via the command line can you make a react application that displays the text right can you host that react application AWS like there's a lot of incremental Improvement to be made there and if I saw a you know a candidate with that on their resume and they list all the different AWS Services they used and how they got it up and running and especially if there's a URL that I can click on and go play the game like that would really make a resume stand out to me yeah and and I will say it would be really valuable to read this book and go through these and I mean you could just memorize the stuff that's in there and you'll probably do pretty okay um there's nothing that replaces building it yourself oh yeah so like let's say you really liked the section on designer key value store and it talks about three or four different approaches if you really want to Ace that interview and you've never built something like that or been on a team that built something like that it would really this is a perfect example of like go Tinker With It Go build the thing that's in there you're going to find surprising things you might even find a bug in the book right um you might find that they skipped the step or didn't talk about something uh that would look really good on a technical interview that you understood oh yeah you have to be careful with this key value store to check for this or that right like and do these kind of interesting things um I I like that they get the projects get more practical as they go on right like I actually loved chapter eight and N chapter eight was a design a URL shortener that's one of my favorites that was that was fun because again it wasn't incredibly complex it was really a testament on getting Clarity on your design constraints um because URL shorten is a pattern that people are familiar with right you take a URL to any arbitrary length URL and you give back some consistent thing uh they course gave a bunch of requirements like oh the hash can't be longer than this or yeah yeah you know do do a bunch these other little pieces and that's cool it's like a juicy little problem that you could you know and I think in that one expecting you build a little resting Point little duplicates this way what happens if it's missing values what happens if you know um and there's lots of approaches to this do you do you hash and include all of the URL parameters or do you ignore URL parameters right that those would be like because a lot of URLs have like affiliate marketing tracking links and stuff like this well yeah is that a unique hash or in your system do you really just care about um you know having that one time those are design constraints and things you have to think about right well and like you know uh chapter nine all about designing a WebCrawler you had mentioned that you had designed a we crawler right yeah I'd actually built one in real life I had no idea what I was doing I was actually learning I was learning go like I'd taken go from like a hobby tool to something that I was serious about writing go for um you would like this I actually used lambdas um there we go and a queuing system and it was a distributed web crawler where the this the we had a a whole Locking System that was actually done in reddis um it was really kind of interesting because the lambdas could call themselves it was like a recursive it was a recursive like Lambda web crawler that would run massively parallel we also needed to be polite we need to honor robots.txt there's all these like rules of ways that you do crawling and scraping and it ran for years like I think I think that project ran for like four or five years it was like used by this ftech company that I was I was just making this thing in my spare time outside of my like devops role um it ended up becoming this like my most mature you know software project that had it was kind of cool like a distributed crawler thing and it was like fun to talk about I brought it up interviews that I done in the future and I was able to like talk about the caveats and again I think that a lot of this what WebCrawler TR uh chapter would have been lost on me unless I'd actually build one myself and I'm like web crawlers are hard they really they really are um because you can you can waste a lot of resources easily you can crawl in really dumb ways it's gotten even harder because Spas are so big now right like now your crawler basically has to have a whole browser rendering engine in the crawler so that you can actually like extract you know so it it gets yeah it's wild um it's not easy not easy stuff if only we were all using HTM X right then our our web crawling would be easier and I say like if if you're doing things like crawling news sites and stuff that actually has gotten easier their news sites have gotten really good at server side rendering a lot of the core content maybe a lot of the other Nuance things but um getting the the core of it because some sites want to be web crawled and server side rendering is really nice for web crawlers it you you lose a lot of web crawler capability if you do it you know Spa style um which I think they actually talked about this before that a whole chunk of the internet especially in the early 2000 early 2010s was like lost because crawlers didn't know how to crawl Spas right um and so you know things like the internet archive and like this are unique and that they do really cool crawling stuff um but a lot of other other kind of commercial data Miners and stuff like just didn't adapt for a while well I think that about takes us through the first half of the book um really interesting book uh we're excited to read the next half like we said we're going to take a quick break uh next week just to cover some interesting essays um I have an on-site and a midterm which are consuming way too much of my time so uh we're human too we gota got to manage our the hours in our day um but as far as the first half of this book uh Nathan I mean what are you gonna do differently because you read this book so I'm gonna read my answer but I think your answer is better um mine is that I said I said not much that I'm actually like this is pretty core to what I'm doing especially as a software architect um I I'm not actively interviewing this not like something interesting to me right now um but the you know these are the kind of tools resources that I actually use day-to-day thinking about weird stuff we haven't built yet um so it's this was this is just a fun book but I have I don't have any like oh I'm gonna do something differently from this well I wrote I don't be better at back of the envelope estimation either um when I do it just being more thorough but also like just thinking about it more in general like we our project at work um we recently have we're trying to integrate more with kind of like the core uh company monolith um and so you know that's that changes a that puts a lot more traffic on our service and we need to be more reliable because of it um I I don't know if anyone asked like how much how much more traffic can we expect to get from this how much traffic are we getting currently um can we handle that you know that increase um and even if uh and even if like it's it turned that would be totally arbitrary like we could handle 10 million requests per day and we're currently doing 500,000 and the core product's going to add another million right and we're just not even close to to what we think it is and there're so good questions to ask because like I don't even know who I need to talk to at work on the core side to figure out um to even figure out how much requests per second we should anticipate um I just like to be better at uh that sort of stuff so this book was a good uh a good reminder I think that was a better answer and honestly we all can improve our envelope estimations I think uh yeah maybe I was just feeling a little sassy when I was answering that yeah yeah I got this but yeah but that's true um and I think yeah there's and there's there's this kind of cool things I've seen um folks that I I've worked with that used to work at Google and stuff they a lot of times they'll have service budgets so you actually have to commit to what your upper bound is um and so you're expect to kind of do these back of the envelope things where you say hey look I can look at the data that's coming into grafana or whatever and that's maybe my Baseline but if one of the other services in our multi-service architecture decides to change its Behavior because they get a new product um release and all of a sudden my service is now having to double or triple the amount of requests per second um do we give them a budget for how much they're allowed to go before the rate limiter goes off right um and so these are interesting because that way you actually do have to think about what is unacceptable behavior from another service in our company and those are healthy those are healthy conversations and back of the envelope I mean back of the envelope uh calculations will help have a good conversation with that team when they're negotiating uh those kind of things absolutely and Nathan who would you recommend this book to okay so if you are preparing for a system interview this is the book for you that's I I don't know I think that would be the context upon which I would recommend this book it's a great book but with a very specific purpose I think so I think if you're a junior engineer I I've talked about this at the beginning of my career kind of wondering like wait a minute so I write the code I run it locally I merge it into GitHub what happens after that right this is a good book that can kind of peel back some of those layers and give you an example of maybe the larger systems beside what you are immediately interacting with but I I say the same like if you got system design interviews coming up like this is the book to read um Alex has done an absolutely fantastic job with it and uh we yeah highly recommend for anyone who's got s interviews coming up if you're just if you're mildly just curious how things work and you should be that's a great place to be bite by go is phenomenal not only do they have a huge Archive of stuff that you can read but just new things come up and he's really good about staying on top of like emerging patterns um if you want to it by by go is a good compliment to book overflow in the sense that you know if you're trying to expand your horizons understand Concepts and develop your career I I would absolutely put this in like the you know top five newsletters you should be paying attention to absolutely well thanks everyone for tuning in we'll be back next week some essays and the week after that we'll be back with uh um the second half of this book you can find us on Twitter at book overo pod you can contact us at contact over flow.io go to B overflow.com Morgan and Nathan's work with functionally imperative at function imperative. comom uh thanks for tuning in everyone we will see you around see you [Music]