hello everyone and welcome back to the channel as you can see uh my sun Baron has really not faded at all after a week and a half uh and my face is currently the same shade of red as the Netflix logo but uh today we're going to be doing yet another systems design video uh I recently acquired a copy of the Alex zoo book which has given me some more inspiration for problems ideas So today we're going to be doing a payment processing system for an e-commerce website now uh I had always thought about this problem and a lot of people had asked me to make it before and I always thought it was kind of dumb because I was like well just go to stripe but uh then you know I got some inspiration and realize that the problem is a little bit more complicated than that so today we get to go through all of those failure cases and talk about them in depth okay so as previously mentioned today we're going to be building out the internal backend payment system of an e-commerce site like Amazon or Teemu or rakuen or anything like that so the gist is you know if I'm a user and I'm about to pay for an item of carts we've got my confirmation button over here we've got some confirmation of which card I'm about to pay with and then also an itemized list of the items that I'm paying and perhaps even the sellers that that money should be routed to cool so let's go ahead and formalize some problem requirements so like I mentioned this is forign e-commerce site and two of the main things that we want to do is number one we want to basically account all of the payments for all of our buyers and also all of the profits for all of our Sellers and ideally those should be ordered by time so that we can quickly figure out over a given time range how everyone is doing in terms of how much they're spending and how much they're making additionally it's very very important that for an e-commerce site like this we have a perfect record of all orders and payments we do not want to lose any of that data because otherwise customers are going to get very angry and that's when we get sued so another thing besides not allowing anyone to lose their data once they've submitted an order is that we don't want to allow duplicate orders as well if I'm a user and I click the submit button and then I click it again I shouldn't pay twice the system should stop me from doing that so no double submitting payments AKA we want item potent payments and then finally uh I'm not actually going to talk about building out the actual payment service itself uh because the United States credit card network is a complicated thing it's a bunch of apis I don't want to talk about dealing with bank accounts uh not only because it would be boring but because I don't know how to and if I did I would be a wealthy man so uh we're going to assume basically that for any payments from the users to us Amazon the middleman uh we can use something like stripe and then uh for any batch payments that we make to our vendors we can use something like depal cool so let's now go into some overall design considerations I'm not going to draw out any highle diagram just yet but hopefully this could uh maybe contextualize the problem for some of you guys and give you some hints into where we're going to take this thing so number one is that we really don't care about latency right I mean we do we want to make it as good as possible but at the end of the day we said we can't lose data and so what that means is that even if it takes a few seconds for a user to submit their payment information that is totally okay we care a lot more that they're never going to lose that payment information and so we're going to optimize for that number two is that we can't approve all payments instantly right some credit cards may get flagged as potential scam payment and so maybe stripe is actually going to take a few hours or maybe even a couple of days in order to actually approve a given payment and then send notice of that back to us they might just say hey uh for now this thing is pending uh and you're going to have to deal with it later number three is that as long as we have one source of truth table right with all of our pending payments and our accepted or failed payments everything else that we have can be derived off of that right we basically need this one Ledger or this one payments table and then we can use deriv data that we've used plenty on the channel before as we all know I'm a flank a file and uh you can basically ship all that data and all those changes into Kafka and then make a bunch of other databases that are reading from Kafka and ultimately uh making their own views on top of that data so the first question is well if we need to have one table that's effectively perfect right no lost data it has to be the source of truth how would we actually go ahead and do that I guess the thing that should pop up in your head is using some form of consensus right so we've got a a couple of options one is just to use strong consistency via synchronous replication so let's imagine I have a leader database over here and I have a follower and then I have a client what synchronous replication says as we've covered many times in this channel is a right is not considered uh valid on the leader until it has reached the follower the follower has acknowledged it and then finally the leader can return the status back to the client so we have to go one two three for the acknowledgement and then four for the acknowledgement from the leader back to the client now the problem with something like this is that we have absolutely no F tolerance whatsoever if the leader goes down we can't make any rights if the follower goes down we still can't make any rights so that's going to be a problem sure you could perform failover on the leader but now you don't have another replica to write to and so as a result of that again no fall tolerance synchronous replication is not going to do it for us so now what you might be thinking of is hm how have we talked about achieving strong consistency in the past while actually having some ability to lose nodes or you know achieve some fa tolerance and the answer that we've at least spoken about is using distributed consensus algorithms so one that we've covered in depth on this channel is raft but you could also use something like paxos or Zab any similar type of algorithm and so the way that we would do that is on the replication log so in the past we've mentioned that these types of distributed consensus algorithms are used for multiple nodes to make sure that they can basically write one type of log and and then as long as one right is committed in that log successfully by the algorithm we know that it will never be lost I mean it can be lost if I were to go up to all three computers for example and hit them with a hammer but the point is uh it's very very unlikely especially if we distribute where these computers live and if we're only talking about something like dis failures or networking failures um so how's that going to look well if you recall uh basically to very very quickly sumarize raft you have have some sort of leader node you've got a couple of follower nodes and basically in order for a right to be committed it needs to First be acknowledged by a quorum of nodes in your cluster so if we have three that means that the right has to go through successfully on two of them and so for that right to happen first our leader gets the right it proposes it to all followers and steps 2 a and 2 B this guy is down over here so it's never going to respond and then uh the top leader or rather the top follower is going to respond successfully so the leader says okay go ahead and Commit This thing once it gets an acknowledgement that the right has been committed to the follower and the right has been committed to itself it can go ahead and tell the user hey your right was successful now the thing that you should note here is that even though this is a great way of ensuring that all of your rights are going to be persisted to at least a couple of nodes and that even if the leader goes down that right will still exist and continue to exist in subsequent epochs uh the problem here is that we are sending a lot of messages over the network and so this type of process is very slow if we expanded our cluster size from three nodes to five nodes again we have more fault tolerance but even still our rights become slower and slower so fortunately that's not our main concern for this type of problem but we will talk about how to make that type of right a little bit faster later in this video so the next piece of the puzzle that I want to cover now that we've talked about strong consistency a little bit and how we can make sure that all of our rights are persisted is actually making payments item potent so typically when we want to make something item potent we basically have two options one is to do something like a two-phase commit right where I can say well if you're the user uh and you're trying to submit a payment uh you know when you put it in the database uh locally you also have to put it in Stripes database and uh you know it's got to work at the same time and if it doesn't you're in trouble in reality that's going to be a slow process and in addition we can't just like go access Stripes internals so we have to go along with what ask us to provide which is an item potency key so the item potency key works because you know if we provide the same item potency key for a transaction twice stripe is going to take a look back at us and say hey I've already seen this one this is a duplicate payment I'm going to ignore it cool so how will we actually generate an item potency key this is another piece of the video that although I could go into it for plenty of time I won't because it's effectively the same thing that we did in our first systems design problem on this channel which is the tiny URL problem which really means we basically have two options one is that we can pre materialize all of our item potency keys right we create a bunch of random Keys we split them out onto a bunch of servers and then every single time someone requests a new item potency key uh the server is basically just going to assign that request uh the next unused item potency key so that's one option another option is for all the clients when they request an item potency key to First basically generate some sort of hash maybe it's based on the user ID maybe it's based on the timestamp that the request is being submitted because ideally we do want as few collisions as possible so something like a hash could be nice and then basically it's going to go to the database say hey I'm trying to claim this item potency key and the database assume assuming that we're doing some sort of locking or maybe it's a SQL database that's enforcing primary key constraints is going to tell us whether or not that key has been taken if it has been taken maybe we bump that up uh number up by one and try and take the next key or something along those lines so in that case we would probe for the next available now of course this is only going to work if every single time that a client generates a key if two clients generate the same key their request needs to go to the same database node uh so you know we have to partition in a smart manner uh that's partitioned by the the value of the key itself or maybe a hash of the key otherwise two requests could go to separate database nodes and they might not know about one another and then they're both assigned the same item potency key cool another thing that we can do to make our table reads a little bit faster is we can actually go ahead and index that table based on that item potency key because basically uh you know we want a database full of item potency Keys uh eventually we're going to have to check whether a given item potency key exists when we hear back from stripe and so we want to basically index our table by that key value so that we can quickly find its location so like I mentioned we have this this one table uh this centralized payments table or maybe we can call it a ledger table or even an orders table but the gist is we have one table that is strongly consistent and so if all the rights are going through it that basically means they all need to be uh ordered in one sort of distributed log and that is going to be problematic and hard to achieve so what would actually be better is basically if we could somehow increase both the right and read throughput of our system and the way that we might be able to do this is by using partitioning right so basically uh instead of just having one payment table like this guy we can have a second one and then each of them is effectively running their own raft algorithm to make sure that all of their rights are strongly consistent now the nice thing about this is this could get a little complicated if we had to do any cross partition rights then we would have to introduce some two-phase commit into the equation but the nice thing about payment rights is that you know we're just creating one row at a time and we're just modifying one row at a time so as a result basically all of these rights are independent to one another so in theory I would think that our throughput would scale linearly with the number of partitions we have so of course we'll have to experiment with that but it should should be a nice solution so basically uh like I mentioned what we want to do is probably partition our nodes by the hash range of the item potency key this way they're distributed relatively evenly and also again it means that if two users were to attempt to claim the same item potency key at once for their incoming transaction or for what they're about to submit to stripe they would both be routed to the same database node uh there would be a conflict and only one of them would be able to claim that key cool and then assuming uh we did the Prem materialization strategy where it's not clients generating their key but it's actually the database generating it what you could do is basically just round robin requests from the load balancer to try and evenly distribute load across using one of those databases so again that should help us to scale out our throughput quite a bit okay so let's now finally that we've introduced kind of these opening Concepts go over a pretty simple design for how we're going to try and handle all this payment information so Step One is basically we have a client over here and he finally lands on that payment page he's selected which products he wants and uh you know he's just opened up the confirmation page the second that we open up the confirmation page we want to generate our item potency key the reason being that you know if I'm going to click the payment button multiple times it's going to be from that same confirmation page and so I can't just generate the item potency key uh when we click the button we have to generate it uh like based on the actual payment page load itself self and then store that locally in the browser so we create our pending payment the second we land on this confirmation page and get the item potency key how do we do that well we go over here to our payments database so now what we might have is something that says like key is 1 two 3 4 uh payment is $10 and right now it's pending cool so now in step two we take our payment server and what we're going to do is go to stripe with our item potency key now at some point in the future stripe is going to do a bunch of stuff and ideally it's going to get back to us right we're eventually going to receive some sort of notification from stripe and we'll talk about that in a little bit more with our item potency key attached to it you can go look at the stripe docs this is very much how it works now our payment server receives that we're going to go update the status in the database hey this thing is no longer pending now it's completed and then using change data capture we can basically sync all of this data to our other databases and derive more data from it hopefully that makes sense as a really high level overview however there are a lot of systems involved here and that means we can be prone to failures so let's go ahead and talk about some of those so for starters what I first mentioned in the overview section was that stripe does not process all payments instantly so in step three over here we're not necessarily going to get get back this message instantaneously from stripe after step two it could be an hour it could be a day it could be a week to process that payment so actually what stripe does in their API is you can basically give them uh a URL which is known as a web hook so what a web hook really does is you know uh I expose an API from my server I tell stripe the name of that API and then when stripe uh completes a certain request for a payment it'll actually go ahead and call my API with the necessary information that it exposes so stripe is going to hit that on completion so this way even if we don't receive word of whether a payment was completed or failed instantly our server always stays ready to receive it great so what are the failure scenarios now that uh we've kind of explained how we're actually going to hear back from stripe well for starters number one what could happen is step one goes through right we get our item potency key we put something in the database with the status of pending and now after that maybe our payment server dies and we never actually submit it to stripe so that's failure scenario number one failure scenario number two which is also going to be a problem is that you know because of the fact that you know we submitted step two we said to stripe hey uh go process this payment for us and then stripe has some sort of web hook saying hey we're going to reach out to this particular payment server right here some URL that it exposes however maybe our payment server went down and as a result now there's no web hook that's actually exposed so we never hear back from stripe so what can we do to fix these I think in my opinion the best answer here is pretty naive and it's just going to be polling so on some infrequent interval basically we want to look at all of those pending transactions in our database anything that hasn't yet been completed or failed where we haven't heard back from stripe and then we want to go ahead and ask stripe hey what happened with this you can still pull stripe for sure it has API that allow you to do this so the first thing to note is that a we should probably only do this for pending payments where some amount of time is actually passed the reason being that uh if we go to stripe and we say hey uh what do you think of this payment has it completed or failed and stripe says I've never actually heard of this payment ID or this item potency key then you're going to say oh then you know what we must have actually never made it to stripe for this one let's go ahead and delete it from our database however if you do that after like half a second from when you added it to the database uh what could happen then is that you know stripe just has yet to hear about it it could be a slow network connection and hasn't yet told us uh that the payment is valid so you you may want to wait some amount of time before you actually start asking stripe about a payment maybe 10 minutes maybe an hour you could configure this in practice so what do we do based on what stripe tells us for starters let's say stripe says Hey I've never heard of this payment I don't recognize this item potency key well that means that it probably never reach stripe we should go ahead and delete it from our payments table it's not pending it just never even got there in the first place number two is a completed or failed status from stripe uh then what we would want to do is go back to the payments table and add the completed or failed status seem simple enough number three is stripe says hey I'm still not sure yet we're still processing this thing do nothing we'll pull for it again maybe in another hour that's totally fine so another question is well okay we want to only do this for pending payments where they've lasted uh for some time right they've been pending for maybe an hour already then we want to start polling how do we quickly figure out which payments these are because like we mentioned uh right now we're using uh a database that is strongly consistent and as a result of that raft algorithm uh rights are going to be pretty expensive and in addition if we want to make a good read from our database we would have to grab some locks on the data and now all of a sudden we're interfering with rights so what can we do instead well what I propose doing is creating a pending payments cache the reason being twofold one we can store our data in memory which is going to be faster than on disk uh which is how it's going to be stored in the database only a small percentage of all the payments of all time will be pending so I do think it's fairly reasonable to assume that we can fit it all in memory number two is that like I mentioned when you want to make a consistent read and you don't want to read stale data um basically you would have to lock on your database node to get a good read otherwise uh you know incoming rights could modify or mess with the data that you're reading so you don't want any contention there because it's going to mess up the right throughput of any incoming payments that new customers are making so what we would prefer to do is instead basically isolate all of these things in a separate cache uh so even though we are going to have eventual consistency here right we don't have maybe the most up-to-date view of the payments uh we're not going to mess with any locking and we can read all these things from memory and it's totally okay right like if I see a payment marked is pending but it really has been completed maybe I go to the database and say hey I think this thing is completed now the database is just going to tell me yeah I already know that doesn't matter so what does this uh workflow kind of look like well we've got our strongly consistent payments SL orders table over here because of change data capture everything is going to be flowing through Kafka and then we can just have Flink as a cache where it builds up some stateful cash of all the pending payments and memory the reason I say to use Flink is so that if it goes down we can basically restore all that state from a Snapchat uh sorry a snapshot that we put in S3 and so the gist is what we can do is as payments come in we can add them to this linked list or rather a doubly linked list ordered by their timestamp and we can also have some sort of hashmap pointing to those payments the reason we want to doubly list link lists with hashmaps is that removes from this list are 0 of one if we hear from stripe that a payment has been confirmed and additions to this list are going to be o of one and additionally the list is already sorted by the incoming timestamp so that way we can only poll uh for payments that have you know existed for a certain amount of time and we don't have to sort the list or anything like that it's just one single part of this list is going to be you know beyond our threshold and one is within our threshold and we can just pull for those item potency keys so like I said we're going to pull stripe on some sort of interval who knows if it's a minute who knows if it's an hour you have to test this out in practice and basically once we hear back for any of those item potency Keys we can go ahead and write right back to the payment SL orders database which again flows back through CDC and into Flink so that we can update our internal State okay let's talk about derived data a little bit more in general because like I mentioned besides just having this Ledger table we want to be able to relatively quickly read certain details about payments for both buyers and sellers right so we want both the revenue per seller in one place and also it'd be good to have um either the orders or basically just you know how much a given buyer has spent in a given time period as well so for this typically what you want to do again is use change data capture as long as this guy over here is strongly consistent we do know that eventually eventually is the key word all of that information is going to flow through CDC and into our databases that we care about eventual consistency is okay here because these are just views for the user if they have to wait another couple of minutes to see their upto-date data that's okay we just know that this guy is the source of truth if we ever get audited or sued or anything like that so basically you know payments DB gets a change it goes through Kafka it goes into some sort of stream consumer and then the stream consumer based on both the buyer ID and the seller ID you know goes through a load balancer and populates the correct Shard of a database so for example if we wanted to look at all of the cells or a sales that a given seller has had over a month and let's say that we see one particular order where we've got seller ID two just made $10 we consult the load balancer see which node that goes to and then write to the particular one now because these are inherently all ordered uh or rather because these are all inherently um populated with a time stamp right all of these messages time 10 we can order them in our time series database properly and so as long as we order by that timestamp uh now we can really easily and quickly query the time series database for the revenue of a given person over a particular time frame all righty so let's go ahead and get into the diagram so as you can see on the left we have our client so we're going to go through the same exact workflow that we already did from a high Lev perspective but now ideally we are going to just make it a little bit more specific so as we mentioned step one is that we have to basically ask our payment service for an item potency key the way that the payment service does that is it can go ahead and hit our payments DB over here which is strongly consistent payments DB strong consistency that can act as a key generation service we use round robin to hit one of them let's say this guy in particular apologies for The Accidental flick and then we're going to come back with our item potency key we then use that in step two to go over to stripe and give it our item potency key in step three some amount of time later our payment service should ideally be hearing back from stripe and going to the database to the appropriate one based on the item potency key to either confirm that the payment has gone through or to say that it failed however of course there's always a possibility that this just went down and it never went through in that case we want to register change data capture onto our payments database the reason being that Flink over here can then see that there is actually a payment uh pending payment and so on some sort of polling interval it can then go over to stripe say hey is this guy actually confirmed or is it failed once it finds out the answer to that it can to go to the load balancer to figure out the right place that that item potency key lives and update the status again this all flows back through the change data capture for cka and back into Flink so that it knows that it no longer has to pull for that item potency key the second second piece of this puzzle is going to be doing all of our derived data over here on the right we also have a stream consumer pulling from Kafka change data capture this doesn't have to be stateful it just has to basically forward messages over to our derived database tables one of which will be the user orders where we basically sh it on the buyer ID and order by the time stamp that it was placed number two is going to be the seller Revenue where we sh it on the seller ID and yet again order by the time stamp that it was placed finally these should probably both be time series databases but I'm open to potential other Solutions if someone said they wanted to do something more column oriented uh maybe put them into a snowflake or a data warehouse or something like that I'm open to that as well anyways guys I hope you enjoyed this video and that it was informative uh going to keep looking for more inspiration for problems but if you do have any feel free to jump in in the comments section and ask me for problems I do ask that when you give me them please try and give detailed functional requirements as well cuz a lot of the times people will say something and it's not instantly clear to me what the exact challenge is so you know really try and hone in on the difficult part of the problem and that way I can take a stab at it anyways enjoy the rest of your week I will do the same and I'll see you in the next one