If you’ve had some exposure to relational databases, you’ve probably come across the term “normalization”. But what is normalization? Why do we do it? How do we do it? And what bad things can happen if we don’t do it? In this video, we’re going to explore database normalization from a practical perspective. We’ll keep the jargon to a minimum, and we’ll use lots of examples as we go. By the end of it, you’ll understand the so-called normal forms from First Normal Form all the way up to Fifth Normal Form – and you’ll have a clear sense of what we gain by doing normalization, and what we lose by failing to do it. This is Decomplexify, bringing a welcome dose of simplicity to complex topics. Data: it’s everywhere. And some of it is wrong. By and large, even a good database design can’t protect against bad data. But there are some cases of bad data that a good database design can protect against. These are cases where the data is telling us something that logically cannot possibly be true: One customer with two dates of birth is logically impossible. It’s what we might call a failure of data integrity. The data can’t be trusted because it disagrees with itself. When data disagrees with itself, that’s more than just a problem of bad data. It’s a problem of bad database design. Specifically, it’s what happens when a database design isn’t properly normalized. So what does normalization mean? When you normalize a database table, you structure it in such a way that can’t express redundant information. So, for example, in a normalized table, you wouldn’t be able to give Customer 1001 two dates of birth even if you wanted to. Very broadly, the table can only express one version of the truth. Normalized database tables are not only protected from contradictory data, they’re also: easier to understand easier to enhance and extend protected from insertion anomalies, update anomalies, and deletion anomalies (more on these later) How do we determine whether a table isn’t normalized enough – in other words, how do we determine if there’s a danger that redundant data could creep into the table? Well, it turns out that there are sets of criteria we can use to assess the level of danger. These sets of criteria have names like “first normal form”, “second normal form”, “third normal form”, and so on. Think of these normal forms by analogy to safety assessments. We might imagine an engineer doing a very basic safety assessment on a bridge. Let’s say the bridge passes the basic assessment, which means it achieves “Safety Level 1: Safe for Pedestrian Traffic”. That gives us some comfort, but suppose we want to know if cars can safely drive across the bridge? To answer that question, we need the engineer to perform an even stricter assessment of the bridge. Let’s imagine that the engineer goes ahead and does this stricter assessment, and again the bridge passes, achieving “Safety Level 2: Safe for Cars”. If even this doesn’t satisfy us, we might ask the engineer to assess the bridge for “Safety Level 3: Safe for Trucks.” And so on. The normal forms of database theory work the same way. If we discover that a table meets the requirements of first normal form, that’s a bare minimum safety guarantee. If we further discover that the table meets the requirements of second normal form, that’s an even greater safety guarantee. And so on. So let’s begin at the beginning, with First Normal Form. Suppose you and I are both confronted by this question: “Who were the members of the Beatles?” You might answer “John, Paul, George, and Ringo”. I might answer “Paul, John, Ringo, and George”. Of course, my answer and your answer are equivalent, despite having the names in a different order. When it comes to relational databases, the same principle applies. Let’s record the names of the Beatles in a table, and then let’s ask the database to return those names back to us. The results will get returned to us in an arbitrary order. For example, they might get returned like this. Or like this. Or in any other order. There is no “right” order. Are there ever situations where there’s a right order? Suppose we write down the members of the Beatles from tallest to shortest, like this. We title our list “Members Of The Beatles From Tallest To Shortest”. In this list, it’s not just the names that convey meaning. The order of the names conveys meaning too. Paul is the tallest, John is the second-tallest, and so on. Lists like this are totally comprehensible to us – but they’re not normalized. Remember, there’s no such thing as row order within a relational database table. So here we have our first violation of First Normal Form. When we use row order to convey information, we’re violating First Normal Form. The solution is very simple. Be explicit – if we want to capture height information, we should devote a separate column to it – like this. Or even better, like this. So far, we’ve seen one way in which a design can fail to achieve First Normal Form. But there are others. A second way of violating First Normal Form involves mixing data types. Suppose our Beatle_Height dataset looked like this. If you’re accustomed to spreadsheets, you’ll be aware that they typically won’t stop you from having more than one datatype within a single column – for example, they won’t stop you from storing both numbers and strings in a column. But in a relational database, you’re not allowed to be cagey or ambiguous about a column’s data type. The values that go in the Height_In_Cm column can’t be a mix of integers and strings. Once you define Height_In_Cm as being an integer column, then every value that goes into that column will be an integer – no strings, no timestamps, no data types of any kind other than integers. So: mixing datatypes within a column is a violation of First Normal Form, and in fact the database platform won’t even let you do it. A third way of violating First Normal Form is by designing a table without a primary key. A primary key is a column, or combination of columns, that uniquely identifies a row in the table. For example, in the table Beatle_Height, our intention is that each row should tell us about one particular Beatle, so we ought to designate “Beatle” as the primary key of the Beatle_Height table. The database platform will need to know about our choice of primary key, so we’ll want to get the primary key into the database by doing something like this. With the primary key in place, the database platform will prevent multiple rows for the same Beatle from ever being inserted. That’s a good thing, because multiple rows for the same Beatle would be nonsensical, and perhaps contradictory. Obviously, a Beatle can’t have two different heights at once. Every table we design should have a primary key. If it doesn’t, it’s not in First Normal Form. The last way of failing to achieve First Normal Form involves the notion of “repeating groups”. Suppose we’re designing a database for an online multiplayer game. At a given time, each player has a number of items of different types, like arrows, shields, and copper coins. We might represent the situation like this. A player’s inventory is what we call a “repeating group”. Each inventory contains potentially many different types of items: arrows, shields, copper coins, and so on; and in fact there may be hundreds of different types of items that a player might have in their inventory. We could design a database table that represents the Inventory as a string of text: But this is a terrible design because there’s no easy way of querying it. For example, if we want to know which players currently have more than 10 copper coins, then having the inventory data lumped together in a text string will make it very impractical to write a query that gives us the answer. We might be tempted to represent the data like this. This lets us record up to 4 items per inventory. But given that a player can have an inventory consisting of hundreds of different types of items, how practical is it going to be to design a table with hundreds of columns? Even if we were to go ahead and create a super-wide table to hold all possible inventory data, querying it would still be extremely awkward. The bottom line is that storing a repeating group of data items on a single row violates First Normal Form. So what sort of alternative design would respect First Normal Form? It would be this. To communicate the fact that trev73 owns 3 shields, we have a row for Player “trev73”, Item_Type “shields”, Item_Quantity 3. To communicate the fact that trev73 also owns 5 arrows, we have a row for Player “trev73”, Item_Type “arrows”, Item_Quantity 5. And so on. And because each row in the table tells us about one unique combination of Player and Item_Type, the primary key is the combination of Player and Item_Type. So let’s review what we know about First Normal Form. 1. using row order to convey information is not permitted 2. mixing data types within the same column is not permitted 3. having a table without a primary key is not permitted 4. repeating groups are not permitted Next up: Second Normal Form. Let’s look again at our Player Inventory table. This table is fully normalized. But suppose we enhance the table slightly. Let’s imagine that every player has a rating: Beginner, Intermediate, or Advanced. We want to record the current rating of each player – and to achieve that, we simply include in our table an extra column called Player_Rating. Notice what’s happening here. Player jdog21 has a Player_Rating of Intermediate, but because jdog21 has two rows in the table, both those rows have to be marked Intermediate. Player trev73 has a Player_Rating of Advanced, but because trev73 has four rows in the table, all four of those rows have to be marked Advanced. This is not a good design. Why not? Well, suppose player gila19 loses all her copper coins, leaving her with nothing in her inventory. The single entry that she did have in the Player_Inventory table is now gone. If we try to query the database to find out what gila19’s Player Rating is, we’re out of luck. We can no longer access gila19’s Player Rating because the database no longer knows it. This problem is known as a deletion anomaly. And that’s not all. Suppose jdog21 improves his rating from Intermediate to Advanced. To capture his new Advanced rating in the Player_Inventory table, we run an update on his two records. But let’s imagine the update goes wrong. By accident, only one of jdog21’s records gets updated, and the other record gets left alone. Now the data looks like this. As far as the database is concerned, jdog21 is somehow both Intermediate and Advanced at the same time. Our table design has left the door open for this type of logical inconsistency. This problem is called an update anomaly. Or suppose a new player called tina42 comes along. She’s a Beginner and she doesn’t have anything in her inventory yet. We want to record the fact that she’s a Beginner, but because she has nothing in her inventory, we can’t insert a tina42 row into the Player_Inventory table. So her rating goes unrecorded. This problem is known as an insertion anomaly. The reason our design is vulnerable to these problems is that it isn’t in Second Normal Form. Why not? What is Second Normal Form? Second Normal Form is about how a table’s non-key columns relate to the primary key. In our table, the non-key columns – or to use slightly different terminology, non-key attributes – are Item_Quantity and Player_Rating. They are columns (also called attributes), that don’t belong to the primary key. As we saw earlier, the primary key is the combination of Player and Item Type. Now we’re in a position to give a definition of Second Normal Form. The definition we’re going to give is an informal one which leaves out some nuances – but for most practical purposes, that shouldn’t matter. Informally, what Second Normal Form says is that each non-key attribute in the table must be dependent on the entire primary key. How does our table measure up to this definition? Let’s examine our non-key attributes, which are the attributes Item_Quantity and Player_Rating. Does Item_Quantity depend on the entire primary key? Yes, because an Item_Quantity is about a specific Item_Type owned by specific Player. We can express it like this. The arrow signifies a dependency – or to give it its proper name, a functional dependency. This simply means that each value of the thing on the left side of the arrow is associated with exactly one value of the thing on the right side of the arrow. Each combination of Player_ID and Item_Type is associated with a specific value of Item_Quantity – for example the combination of Player_ID jdog21 / Item_Type “amulets” is associated with an Item_Quantity of 2. As far as Second Normal Form is concerned, this dependency is fine, because it’s a dependency on the entire primary key. But what about the other dependency? Does Player_Rating depend on the entire primary key? No, it doesn’t. Player_Rating is a property of the Player only. In other words, for any given Player, there’s one Player_Rating. This dependency on Player is the problem. It’s a problem because Player isn’t the primary key – Player is part of the primary key, but it’s not the whole key. That’s why the table isn’t in Second Normal Form, and that’s why it’s vulnerable to problems. At what point did our design go wrong, and how can we fix it? The design went wrong when we chose to add a Player_Rating column to a table where it didn’t really belong. The fact that a Player_Rating is a property of a Player should have helped us to realise that a Player is an important concept in its own right – so surely Player deserves its own table: Nothing could be simpler than that. A Player table will contain one row per Player, and in it we can include as columns the ID of the player, the rating of the player, as well as all sorts of other properties of the player – maybe the player’s date of birth, for example, maybe the player’s email address. Our other table, Player_Inventory, can stay as it was. For both tables, we can say that there are no part-key dependencies. In other words, it’s always the case that every attribute depends on the whole primary key, not just part of it. And so our tables are in Second Normal Form. Now let’s move on to Third Normal Form. Suppose we decide to enhance the Player table. We decide to add a new column called Player_Skill_Level. Imagine that in this particular multiplayer game, there’s a nine-point scale for skill level. At one extreme, a player with skill level 1 is an absolute beginner; at the opposite extreme, a player with skill level 9 is as skilful as it’s possible to be. And let’s say that we’ve defined exactly how Player Skill Levels relate to Player Ratings. “Beginner” means a skill level between 1 and 3. “Intermediate” means a skill level between 4 and 6. And “Advanced” means a skill level between 7 and 9. But now that both the Player_Rating and the Player_Skill_Level exist in the Player table, a problem can arise. Let’s say that tomorrow, player gila19’s skill level increases from 3 to 4. If that happens, we’ll update her row in the Player table to reflect this new skill level. By rights, we should also update her Player_Rating to Intermediate – but suppose something goes wrong, and we fail to update the Player_Rating. Now we’ve got a data inconsistency. gila19’s Player_Rating says she’s a Beginner, but her Player_Skill_Level implies she’s Intermediate. How did the design allow this happen? Second Normal Form didn’t flag up any problems. There’s no attribute here that depends only partially on the primary key – as a matter of fact, the primary key doesn’t have any parts; it’s just a single attribute. And both Player_Rating and Player_Skill_Level are dependent on it. But in what way are they dependent on it? Let’s look more closely. Player_Skill_Level is dependent on Player_ID. Player_Rating is dependent on Player ID too, but only indirectly – like this. A dependency of this kind is called a transitive dependency. Player Rating depends on Player Skill Level which in turn depends on the primary key: Player ID. The problem is located just here – because what Third Normal Form forbids is exactly this type of dependency: the dependency of a non-key attribute on another non-key attribute. Because Player Rating depends on Player Skill Level – which is a non-key attribute – this table is not in Third Normal Form. There’s a very simple way of repairing the design to get it into Third Normal Form. We remove Player Rating from the Player table; so now the Player table looks like this. And we introduce a new table called Player_Skill_Levels. The Player Skill Levels table tells us everything we need to know about how to translate a player skill level into a player rating. Third Normal Form is the culmination of everything we’ve covered about database normalization so far. It can be summarised in this way: Every non-key attribute in a table should depend on the key, the whole key, and nothing but the key. If you commit this to memory, and keep it constantly in mind while you’re designing a database, then 99% of the time you will end up with fully normalized tables. It’s even possible to shorten this guideline slightly by knocking out the phrase “non-key” – giving us the revised guideline: every attribute in a table should depend on the key, the whole key, and nothing but the key. And this new guideline represents a slightly stronger flavor of Third Normal Form known as Boyce-Codd Normal Form. In practice, the difference between Third Normal Form and Boyce-Codd Normal Form is extremely small, and the chances of you ever encountering a real-life Third Normal Form table that doesn’t meet Boyce-Codd Normal Form are almost zero. Any such table would have to have what we call multiple overlapping candidate keys – which gets us into realms of obscurity and theoretical rigor that are a little bit beyond the scope of this video. So as a practical matter, just follow the guideline that every attribute in a table should depend on the key, the whole key, and nothing but the key, and you can be confident that the table will be in both Third Normal Form and Boyce-Codd Normal Form. In almost all cases, once you’ve normalized a table this far, you’ve fully normalized it. There are some instances where this level of normalization isn’t enough. These rare instances are dealt with by Fourth and Fifth Normal Form. So let’s move on to Fourth Normal Form. We’ll look at an example of a situation where Third Normal Form isn’t quite good enough and something a bit stronger is needed. In our example, there’s a website called DesignMyBirdhouse.com – the world’s leading supplier of customized birdhouses. On DesignMyBirdhouse.com, customers can choose from different birdhouse models, and, for the model they’ve selected, they can choose both a custom color and a custom style. Each model has its own range of available colors and styles. One way of capturing this information is to put it all the possible combinations in a single table, like this. This table is in Third Normal Form. The primary key consists of all three columns: {Model, Color, Style}. Everything depends on the key, the whole key, and nothing but the key. And yet this table is still vulnerable to problems. Let’s look at the rows for the birdhouse model “Prairie”: The available colors for the “Prairie” birdhouse model are brown and beige. Now suppose DesignMyBirdhouse.com decides to introduce a third available color for the “Prairie” model: green. This will mean we’ll have to add two extra “Prairie” rows to the table: one for green bungalow, and one for green schoolhouse. If by mistake we only add a row for green bungalow, and fail to add the row for green schoolhouse, then we have a data inconsistency. Available colors are supposed to be completely independent of available styles. But our table is saying that a customer can choose green only for the bungalow style, not for the schoolhouse style. That makes no sense. The prairie birdhouse model is available in green, so all its styles should be available in green. Something about the way the table is designed has allowed us to represent an impossible situation. To see what’s gone wrong, let’s have a closer look at the dependencies among Models, Colors, and styles. Can we say that Color has a functional dependency on Model? Actually no, because a specific Model isn’t associated with just one Color. And yet it does feel as though Color has some relationship to Model. How can we express it? We can say that each Model has a specific set of available Colors. This kind of dependency is called a multivalued dependency, and we express it with a double-headed arrow, like this: And it’s equally true that each Model has a specific set of available Styles. What Fourth Normal Form says is that the only kinds of multivalued dependency we’re allowed to have in a table are multivalued dependencies on the key. Model is not the key; so the table Model_Colors_And_Styles_Available is not in Fourth Normal Form. As always, the fix is to split things out into multiple tables. Now, if DesignMyBirdhouse.com expands the range of Prairie-Model colors to include green, we simply add a row to the Model_Colors_Available table: And no anomalies are possible. We’re now ready for Fifth Normal Form, the last normal form covered in this video. For our Fifth Normal Form example, we imagine that there are three different brands of ice cream available: Frosty’s, Alpine, and Ice Queen. Each of the three brands of ice cream offers a different range of flavors: Frosty’s offers vanilla, chocolate, strawberry, and mint chocolate chip Alpine offers vanilla and rum raisin Ice Queen offers vanilla, strawberry, and mint chocolate chip Now we ask our friend Jason what types of ice cream he likes. Jason says: I only like vanilla and chocolate. And I only like the brands Frosty and Alpine. We ask our other friend, Suzy, what types of ice cream she likes. Suzy says: I only like rum raisin, mint chocolate chip, and strawberry. And I only like the brands Alpine and Ice Queen. So, after a little bit of brainwork, we deduce exactly which ice cream products Jason and Suzy are willing to eat; and we express this in a table: But time passes, tastes change, and at some point Suzy announces that she now likes Frosty’s brand ice cream too. So we need to update our table. It won’t come as any surprise that we might get this update wrong. We might successfully add a row for Person Suzy – Brand Frosty’s – Flavor Strawberry, but fail to add a row for Person Suzy – Brand Frosty’s – Flavor Mint Chocolate Chip. And this outcome wouldn’t just be wrong – it would be logically inconsistent – because we’ve already established that Suzy likes Frosty’s brand, and likes Mint Chocolate Chip flavor, and therefore there’s no way she can dislike Frosty’s Mint Chocolate Chip. In this example, we went wrong right at the beginning. At the beginning, we were given three pieces of information. First, we were told which brands offered which flavors. Second, we were told which people liked which brands. Third, we were told which people liked which flavors. From those three pieces of information, we should have simply created three tables. And that’s all we needed to do. All the facts of the situation have been represented. If we ever want to know what specific products everyone likes, we can simply ask the database platform, expressing our question in the form of a piece of SQL that logically deduces the answer by joining the tables together. To sum things up: if we want to ensure that a table that’s in Fourth Normal Form is also in Fifth Normal Form, we need to ask ourselves whether the table can be logically thought of as being the result of joining some other tables together. If it can be thought of that way, then it’s not in Fifth Normal Form. If it can’t be thought of that way, then it is in Fifth Normal Form. We’ve now covered all the normal forms from First Normal Form to Fifth Normal Form. Let’s review, keeping in mind that for a table to comply with a particular normal form, it must comply with all the lower normal forms as well. The rules for first normal form are: 1. using row order to convey information is not permitted 2. mixing data types within the same column is not permitted 3. having a table without a primary key is not permitted 4. repeating groups are not permitted The rule for second normal form is: Each non-key attribute in the table must be dependent on the entire primary key. The rule for third normal form is: Each non-key attribute in a table must depend on the key, the whole key, and nothing but the key. If we prefer to drop the phrase “non-key”, we end up with an even simpler and even stronger version of third normal form called “Boyce-Codd Normal Form”: Each attribute in a table must depend on the key, the whole key, and nothing but the key. The rule for fourth normal form is that the only kinds of multivalued dependency we’re allowed to have in a table are multivalued dependencies on the key. Finally, the rule for Fifth Normal Form is: it must not be possible to describe the table as being the logical result of joining some other tables together. I hope you’ve found this video helpful. If you have any comments or questions on what you’ve just seen, by all means put them in the comments section below. And if you have any suggestions for other complex topics that you’d like to see explained on Decomplexify, again let me know in the comments. So long, and thanks for watching!