when designing large scale systems one of the biggest challenges is ensuring the system remains responsive even when handling massive volumes of data and concurrent user requests in such scenarios the database is the primary bottleneck and to make sure our systems remain fast and reliable even under heavy load we can leverage two key techniques which are called database replication and database sharding let's start with database replication at its core database replication involves creating multiple copies also called rep us of a database and then Distributing them across multiple servers and this ensures High availability because it provides a safety net against database failures if one of our databases goes down the application can still switch to another replica and this will make sure that they will get uninterrupted service and high availability and it also scales the read capacity of our database because now we have multiple databases that can serve the data to our users but how does this replication work we have to two common replication methods the first one is leader follower replication in this setup we have one database which serves as the leader also called master and we have other databases which act as followers or slaves in this case right operations are directed to the leader and then it propagates the changes to follower databases and read operations are distributed across both leader and followers which will enhance the read capacity and on the other hand we have leader leader application here multiple databases act as leaders and each of them can accept both read and write operations however in this case we may have conflicts between leader databases and we need to set up conflict resolution mechanisms to make sure we are handling these conflicts properly and these replications can be either asynchronous or synchronous replications in case of asynchronous replication changes are propagated to replicas in the background it is also faster but it carries the risk of temporary data consistencies and on the other hand with synchronous replication changes are committed to both leader and master replicas simultaneously and this guarantees consistency but it potentially impacts the right performance if you imagine we are writing something to the leader database we need to also wait for it to be propagated to followers and then we'll get the response back and that's why we have the right performance impact now let's come back to the leader leader application which helps us to scale the rights leader leader application is typically more complex because it comes with the complexity of managing conflicts that may arise when multiple leaders accept right operations because you might have a scenario where the user updates a record in one leader database and then this same user updates the same record in another database and now you have to manage the conflict between these two records because they are out of date and you need to merge them together without losing data and to maintain this data consistency we have conflict resolution mechanisms one example of this is timestamp based resolution in this case the update with the latest timestamp wins which means if you have two records that were updated in two leader databases the one that was updated more recently will overwrite the other one you also have the last right WIS mechanism where the most recent right regardless of its time stamp overd the previous one and this also can be custom you can set up application specific rules that can be applied to result of conflicts based on the nature of the data and the behavior of your system now let's have a look at sharding and how it differs from the replication when a database grows massive even replication might not be enough and a single server might struggle to handle the storage and processing demands and sharding tackles this by Distributing the data across multiple servers which allows us horizontal scaling this means that we are splitting large tables of our main database and we are locating parts of fet in each shart typically tables are sharded based on specific criteria for example in an e-commerce platform customer data might be sharded based on geographical location if you imagine we have a very large customers table which contains all the information about the customers we can shart them either by IDs which means that the first th000 users will be located in the first chart the second th000 users will be located in the second shart and so on but we can also Shard them based on region which makes more sense in this case and then when we need to query some data we can query it based on the region which means for example the US users will be located in the first chart and if you are querying for a customer that is located in the US region then we will querry the first Shard of this database in this case The Shard key will be the region of this database but how do we decide this where each data goes for that we use Shard Keys Shard Keys determine how data is assigned to sh CHS for example we can have the range based sharding which we just saw in this case data is partitioned based on ranges of The Shard key for example customer IDs from 1 to 1,000 might go to Shard 1 the next 1,000 will go to The Shard two and so on but we can also have has base sharding in this case we use has function which is applied to The Shard key to determine The Shard this method leads to a more even distribution of data but it can make the range queries less efficient because if you want to execute a range query you may need to query all shards to find the relevant data which makes it less efficient than range based sharding where you know exactly which data is in which chart and sharding can be very different depending if you're doing it on SQL or nosql database traditional SQL databases typically don't offer sharding out of the box which means you need to implement the sharding logic yourself and this makes sense because that's the nature of SQL databases you have to side of the tables and the shards of which table goes where but on the other hand many nosql databases like mongod DB have built-in sharding support which makes it easier to scale horizontally and you don't have to provide much custom logic because it comes with the sharding out of the box so to summarize replication ensures High availability and it also scales the read capacity while sharding enables horizontal scaling and it distributes data across multiple servers and the choice between these two techniques depends on the requirements of specific system and its data and it's very common to use both of these in combination which means that you can use both replication and also sharding to Shard each of the leader or follower databases in the replicated schema if you're interested to learn more about database scaling and performance then be sure to check out this video next