Transcript for:
Comprehensive Guide to Python Data Analysis

Welcome to our data analysis with Python tutorial. My name is Santiago and I will be your instructor. This is a joint initiative between Free Code Camp and remoter. In this tutorial, we'll explore the capabilities of Python on the entire PI Data stack to perform data analysis, we'll learn how to read data from multiple sources such as databases, CSV and Excel files, how to clean and transform it by applying statistical functions and how to create beautiful visualizations will show you all the important tools of the PI Data stack pandas, matplotlib, Seabourn and many others. This tutorial is going to be useful both for Python beginners that want to learn how to manage data with Python, and also traditional data analysts coming from Excel tableau, etc. You learn how programming can power up your day to day analysis. So let's get started. Welcome to our data analysis with Python tutorial My name is Santiago and I am an instructor@remoter.com an online Data Science Academy. This tutorial is a result of a joint effort by remoter and Free Code Camp, and it's totally free. It includes slides, Jupyter, notebooks and coding exercises. Let me tell you a little bit more about remoter were an online hands on Data Science Academy. We specialize in data science, including data analysis, programming and machine learning. We have a complete course catalog and we're adding more content every month. If you're interested in learning data science or data analysis, check us out. As part of this joint effort between Free Code Camp and remoter you can get a 10% discount in your first month by using the following discount coupon. Let's quickly review the contents of this tutorial. In the description of this video, we have included direct links to each section, so you can jump between them. This is the first section and we are going to discuss one is data analysis. We'll also talk about data analysis with Python and why programming tools like Python SQL and pandas are important. In the following section will show you a real example of data analysis using Python. So you can see the power of it will not explain the tools in detail. It's just a quick demonstration for you to understand what this tutorial is about. The following sections will be the ones explaining each tool in detail, there are two more sections that I want to especially point out. The first one is section number three Jupiter tutorial. This is not mandatory, and you can skip it if you already know how to use Jupyter notebooks. Also the last section Python in under 10 minutes. This is just a recap of Python. If you're coming from other languages, you might want to take this first. If that's the case, again, you can use the links in the video description to jump straight to it. All right now let's define what is data analysis. I think the Wikipedia article summarizes perfectly the process of inspecting, cleansing, transforming and modeling data with the goal of discovering useful information, you forming conclusions and support decision making. Let's analyze this definition piece by piece. The first part of the process of data analysis is usually tedious. It starts by gathering the data and cleaning it and transforming it for further analysis. This is where Python and the PI Data Tools Excel. We're going to be using pandas to read, clean and transform our data. Modeling data means adapting real life scenarios to information systems using inferential statistics to see if any pattern or model arise. For this we're going to be using the statistical analysis features panelists and visualizations for matplotlib and Seabourn. Once we have processed the data and created models out of it, we'll try to drive conclusions from it finding interesting patterns or anomalies that might arise. The word information here is key. We're trying to transform data into information. Our data might be a huge list of all the purchases made in Walmart in the last year, the information will be something like pop tarts sell better on Tuesdays. This is the final objective data analysis we need to provide evidence of our findings, create a readable reports and dashboards and aid other departments with the information we've gathered. Multiple actors will use your analysis, marketing sales, accounting executives, etc. They might need to see a different view of the same information. They might all need different reports or level of detail what tools are available today for data analysis. We've broken these down into two main categories, our managed tools, our close products, tools you can buy and start using right out of the box. Excel is a good example. Tableau and luchar are probably the most popular ones for data analysis. In the other extreme, we have what we call programming languages or we Call them open tools. These are not sold by an individual vendor, but they are a combination of languages open source libraries and products. Python R and Giulia are the most popular ones in this category. Let's explore the advantages and disadvantages of them. The main advantage of close tools like Tableau or Excel is that they are generally easy to learn. There is a company writing documentation providing support and driving the creation of the product. The biggest disadvantage is that the scope of the tool is limited, you can cross the boundaries of it. In contrast, using Python and the universe of PI Data Tools gives you amazing flexibility. Do you need to read data from a closed API using secret key authentication for example, you can do it? Do you need to consume data directly from AWS kinases, you can do it a programming language is the most powerful tool you can learn. Another important advantage is a general scope of a programming language. What happens if Tableau for example, goes out of business. Or if you just get bored from it and feel like your career is taught you need a career change? learning how to process data, using a programming language gives you freedom? The main disadvantage of a programming language is that it's not as simple to learn as with a tool, you need to learn the basics of coding first, and it takes time. Why are we choosing Python to do data analysis? Python is the best programming language to learn to code. It's simple, intuitive, and unreadable. It includes 1000s of libraries to do virtually anything from cryptography to IoT. Python is free and open source. That means that there are 1000s of PI's very smart people seeing the internals of the language under libraries. from Google to Bank of America, major institutions rely on Python every day, which means that it's very hard for it to go away. Finally, Python has a great open source spirit. The community is amazing, the documentation, so exhaustive, and there are a lot of free tutorials around checkout for conferences in your area, it's very likely that there is a local group of Python developers in your city. We couldn't be talking about data analysis without mentioning r r is also a great programming language. We prefer Python because it's easier to get started and more general in the libraries and tools it includes. R has a huge library of statistical functions. And if you're in a highly technical discipline, you should check it out. Let's quickly review the data analysis process. The process starts by getting the data where is your data coming from? Usually it's in your own database, but it could also come from files stored in a different format, or a web API. Once you've collected the data, you will need to clean it. If the source of the data is your own database, then it's probably in writing shape. If you're using more extreme sources like web scraping, then the process will be more tedious. With your data clean, you'll now need to rearrange and reshape the data for better analysis, transforming fields merging tables, combining data from multiple sources, etc. The objective of this process to get the data ready for the next step. The process of analysis involves extracting patterns from the data that is now clean and in shape. Capturing trends or anomalies. statistical analysis will be fundamental in this process. Finally, it's time to do something with data analysis. If this was a data science project, we could be ready to implement machine learning models. If we focus strictly on data analysis, we'll probably need to build reports communicate our results, and support decision making. Let's finish by saying that in real life, this process isn't so linear, we're usually jumping back and forth between the step and it looks more like a cycle than a straight line. What is the difference between data analysis and data science? The boundaries between data analysis and data science are not very clear. The main differences are that data scientists usually have more programming and math skills, they can then apply these skills in machine learning on ETL processes. The analysts on the other hand, have a better communication skills creating better reports with stronger storytelling abilities. By the way, these Weiler chart you're seeing right here is available in the notes in case you want to check out the source code. Let's explore the Python and PI Data ecosystem, all the tools and libraries that we will be using. The most important libraries that we will be using are pandas for data analysis, and matplotlib and Seabourn for visualizations. But the ecosystem is large and there are many useful libraries for specific use cases. How do Python data analysts think if you're coming from a traditional data analysis place using tools like Excel and Tableau you're probably used to have a constant visual reference of your data. All these tools are point on Click. This works great for a small amount of data. But it's less useful when the amount of records grow. It's just impossible for humans to visually reference too much data, and the processing gets incredibly slow. In contrast, when we work with Python, we don't have a constant visual reference of the data we're working with. We know it's there. We know how it looks like. We know the main statistical properties of it, but we're not constantly looking at it. These allows us to work with millions of records incredibly fast. This also means you can move your data analysis processes from one computer to the other, and for example, to the cloud without much overhead. And finally, why would you like to add Python to your data analysis skills aside from the advantages of freedom and power theories, another important reason, according to PayScale, data analysts that no Python and SQL are better paid than the ones that don't know how to use programming tools. So that's it. Let's get started in our following section will show you a real world example of data analysis with Python, we want you to see right away what you will be able to do after this tutorial. We're gonna start this tutorial by working with a real example of data analysis and data processing with Python, we're not going to get into the details yet, the following sections will explain what each one of the tools does, and what is the best way to apply them combining and the details of them. In general, this is just for you to have a quick on high level reference of our day to day processes, data analysts, data managers, data scientist using Python. So the first data set that we're going to use is a CSV file that has this form, you can find it right here, under the data directory, the data we're going to be used is this, I have just transformed it into a spreadsheet. So we can pretty much look at it from a more visual perspective. But remember, as we said in the introduction, as data analysts are not constantly looking at the data, right, we don't have a constant visual reference, we are more driven by the understanding of the data right in the back of our head, and we understand how what the data looks like, what's the shape of it. And that's what it's conducting our analysis. So the first thing we're going to do is we're going to read it this CSV into Python, and you can see how simple it is just one line of code gets us the CSV read into byte, then we're going to give a quick reference. And this is what the data frame that we have created looks like data frame is a special word is a special data structure, we use independent tool. And again, we're going to see that in detail in the pan this part of this tutorial. The data frame is pretty much the CSV representation, but it has a few more enforced things like for example, each column has a strict data type. And we will not be able to change it to tetra, it's a better way to conduct our analysis, the shape of our data frame tells us how many rows and how many columns we have. So you can imagine that with these amount of rows, it's not so simple to again, follow a visual representation of it's like, it's pretty much infants crawling, in this point 100,000 rows. But the way we work is by immediately after we load our data we have we want to find some sort of reference in the shape and the the properties of the data we're working with. And for that we're going to do first an info to quickly understand the columns we're working with. In this case, we have date, which is a date time field, we have day, month year on that are just complimentary to date, we have the customer age, which is uninjured, which makes sense right? age group, you can say it's right here. It's age group youth, customer gender, we have an idea again, of the of the entire data set, we know the columns we have, but we also know how large it is. And we don't care what's in between, we will be cleaning it probably, but we don't need to actually start looking row per row, right just with our very limited eyes, we have a better understanding of the structure of our data in this way. And we're going one step further, we will also have a better understanding of the statistical properties of this data frame with a describe method. For all those numeric fields, I can have an idea of the statistical properties of those. So for example, I know that the average age of these data set is 35 years old. I also know that the maximum age in this case if these Or is the sales data is 87 years old, I know the minimum is 17 years old. And again, I can start building right if my understanding of this that physical properties of it. So in this case, the median of my age is very close to the mean. So this is telling me, all is telling me something, and the same thing is going to happen for each one of the columns that we are using. For example, we have a negative profit here, and we have very large values here are these correct, is maybe there's a mistake, again, it's by having a quick statistical view of our data, we're going to be driving the process of analysis without the need of constantly looking at all the rows that we have. It's a, it's a more general holistic overview. So we're gonna start with unit cost, let's, let's see what it looks like. And we're going to do a describe only if you need coast, which is pretty much what we had right here. In the previous in this line, what we did was for the entire data frame for the entire data, in this case, we're just focusing in the unit coast, cost, sorry, column, the mean, the median, all fields, we know already pretty much from this, and we're gonna quickly plot them, we're going to use these tools to visualize them. And it's the same tool, it's paying this that it's using on top, right? It's using matplotlib. So the visualization is created with matplotlib. But we're doing it directly from pandas. And again, don't worry, this is all explained in pandas lessons. So this is unit costs, right is what this is the box, but we have just created, we have the whiskers that mean that shows us the the first and third quartile, the median. And then we see all the outliers that we have right here. So we see that our product study is around $500 is considered to be an outlier. And the same thing if we do a density plot, right. So this is what it looks like. We're going to draw two more charts, right, in which we're going to pretty much point out the mean and the median, right in the distribution charts. And we're going to do a quick histogram of the costs of our products. Moving forward, we're going to talk about age groups with the age of a customer. And at any moment, we can always do something like sales sort here to give a quick reference, we know that the the age of the customer is expressed in actual years old they were but also they have been categorized with three, four, actually four age groups, seniors, youth, young adults and adults, right. So they we have given categories were creative, right to better understand these groups, and we do that with values. Value counts, we can quickly get a pie chart out of it, or we could get a bar chart out of it. As you can see, right here, we're doing an analysis of our data, we see that adults right here are the largest group in our for our data at least. So moving forward, what about a correlation analysis? What is a correlation between some of our properties, we will probably have high correlation for example, between profit and unit cost, for example, or order quantity, that's kind of expected, but that's all something that we can do right here. This is matrix right of correlation showing in red high correlation. So order quantity, and unit cost or where is profit right here. Profit is right here. So we see high correlation with unit with cost with profit. Now with profit, actually, it's the opposite blue is high correlation, I'm sorry, the diagonal, which is blue, is correlation is equals one. So high correlation is blue. And we see that profit has huge correlate has a lot of correlation, positive correlation with unit cost and unit price. And negative correlation is with dark red. So we again can have a quick idea. Let's see, for example, here profit, it has negative correlation with order quantity, which is interesting, right? It's we wouldn't dig deeper into that, of course, the profit has a high correlation positive with revenue, right? And again, it's just a quick correlation analysis. We can also do a quick scatterplot to analyze the customer age and the revenue right to see if there is any, any correlation there. Right? And the same thing for revenue and profit. This is obvious, right? We can we can quickly draw a diagonal here, right. So there is a lot Linear depth and dependency between these variables. So a form a few more box plots, in this case, understanding the profit per age group, right, so we can see how the profit will be, will change depending of the customer's age, and a few more box plots. And we're creating these these grid of year customer age, unit costs, etc, for multiple things. So moving forward, something that we can quickly do when we're working with Python, especially within this is Drew shape or data or derive it from other columns, right. So this is pretty common in Excel, we can create these revenue per age column, if you're here in Google spreadsheets, you're going to do something like revenue, per age, and you're going to do something like equals, right? Equals revenue, divided, I don't remember if this correct formula we're using, but just for, for you to have a reference. And we're going to pretty much extend this whole thing. There we go, Oh, well is processing, and I have 100,000 rows. So you can see how slow it is, I let's compare that just to the way Python works, I'm gonna execute this thing. It was instant, you know, extremely fast. And it was all calculated seems that we have the same results as expected. same results as expected. And we can quickly plot both the in a density plot and in a histogram, as you can see, right there, now that revenue parade is going to be relevant. In any case, it's just to show you the capabilities of what we can do. Let's annual analyze, well, we're gonna create a new column, which is calculated cost is the total, the total orders the total, the quantity of the order, times the cost, right, extremely simple formula, very fast process. And we're gonna get right here, how many rows had a different value than what was provided by cost? So what we're doing right here is like, we're quickly checking if the cost provided by the data set, at some point doesn't align with the actual cost we are calculating. So is there any mistakes that were made by the I don't know the original system, or people doing a data entry, if these new column is different from cost, we want to know about that. And that doesn't happen. So again, quick, quick, regression plot. In this case, it's very obvious that there is some linear dependency between calculate cost and profit. So more formulas, in this case costs part cost plus profit. So we're going to adding a little bit more, there is no difference with the revenue and the calculated revenue that we are having. So that all makes sense, we're going to do a quick histogram of the revenue. We can, for example, on 3%, to all the prices that we are using, we need to increase prices. How are you going to do that? Well, it's very simple with Python, we're just going to do increase everything by point 03. And now all the prices have changed. What else we're going to be able to do quick filtering, let's get all the sales from the state of Kentucky right. So these are all the sales from the state of Kentucky, we can get only the average of the sales by these age group on only revenue, right. So these, all these filtering options, and extremely simple to get with Python. In this case, we say, give me all the sales from these age group, and also from this country, right, and we're gonna get the average revenue from these groups that we are selecting. And again, to modify the data, we can make just a few quick modifications, like in this case, we're going to say, all the sales from country right to revenue, we're going to increase it by 1.1. I don't know why, which is doing it arbitrarily. It's just for me to show you how it works. So far, so good. Again, we've done a couple things, you don't need to know about the details, we will actually go through that in the NumPy independence sections in this tutorial. So just for you to have a quick reference of it. There are exercises associated with these given lectures. So if you want to pause right now and get into the exercises, that's going to be very helpful. We're going to move forward now with the second lecture in which we will be using a database this Akila database and we're going Be erasing data, instead of from a CSV file, as we did before, we're going to read data now from a database. Reading data from a SQL database is as simple as it is from an Excel file or a CSV file, as we were doing with our previous example. And once you've read the data, that's we're going to do now the process is the same. So what we have right here is a query a SQL query, if you don't know about SQL, you can check our courses or other courses online. Basically, we're pulling the data from the database. This is one of the advantages of Python, it's not, there are connectors for pretty much every database provided out there, Oracle, Postgres, MySQL, SQL Server, etc. In this particular example, we're going to be using MySQL. So once you construct the query, and you pull the data from the database, then the process is the same, we have just converted these outside data into a data frame that we can use with our Python skills. The first step, as usual, is to check the shape information description of our data of our data frame. In this case, we want to, again understand the structure of it. So we want to know how many rows we have 16,000, we want to know a little bit more about our rows, we want to know about a little bit more about our columns, and how many rows how many records we have for each one of them and the type of each one of these columns. And we also want to have a better statistical understanding of our data. So we do a quick describe, and we have more details about it. If we want to focus in individual columns, right, we can just do that by in this case, we're gonna focus in film rental rate, right, pretty much how much you pay to rent a film. Um, we're gonna see the kind of distribution we have, we can call it distribution, it's pretty much a categorical field in this case, but basically, the rentals are divided into three main categories are prices, zero 99 299 499. So that's these box plot these pretty much perfect, never seen in real life plot box plot gives you those prices. And move forward, we can also check very quickly a categorical analysis, understanding the distribution of rentals between cities, so we have two cities. And it's pretty much even as you can see right here, creating new columns and reshaping the data for further analysis, etc, is relatively simple. In this case, we're going to analyze their return in rentals, right, which, which films are going to be more profitable for the company div, dividing the rental rate, how much we charge, divided by the cost, how much it costs us to acquire the film. So in this case, we can see the distribution of that, right. So most rentals are here in the beginning. And then we have more profitable rentals, were making up to 60% above the rental. And we can quickly analyze the mean and the median fit right to have a quick idea of all that. Finally, selection and indexing, if you want to start focusing, if you want to go into data, right, you want to zoom in, you want to have a better understanding. So you start filtering, in this case, we can filter by customer, but if you want to do it per city, if you want to do it per state, if you want to do it per film, per price category, etc. It's very simple to filter to filter and zooming, which is one particular characteristic of your data. So you can perform a more detailed analysis. So in this case, we have all the the films are rented by the customer last name, Hanson, which doesn't mean it's the same person. But again, it's very simple to filter dot. And here, we can do we can very quickly see which ones are the price, the film's sorry, that have the highest replacement cost, right. So basically, what we're doing is we're going to isolate those films that have the highest replacement cost. And also we can see right here just for you to have an idea, all the films that are in the category PG or pG 13. It's very simple to to filter that data. So this is the process we usually follow. we imported the data, we reshape it somehow create columns, there is an important process of cleaning up or not highlighting this part of the tutorial, we're going to talk about it in the tutorial itself. There's the process of cleaning, then reshaping creating new columns, combining data and creating visualizations. This is the process, right? We're following here with our Python skills, but it's a tone more to odd as you might imagine, from creating reports to running machine learning processes, creating linear regressions, etc. For now, this is just a quick understanding of the process. We follow. Now starting now we're gonna move forward with more details of each one of the individual tools we're going to talk about. We're going to talk about Jupyter notebooks. We're going to talk about NumPy. We're going to talk about pandas, we're going to talk about mapa, lib, seaborne, etc. Starting now, right? The first thing we're going to see is, what is this whole thing that I've been using this Jupyter Notebook, I want you to now too, if you want, if you if you don't have experience with it, I want you to have an idea of how it works. And then we're going to move forward the individual tools, NumPy, pandas, etc. Remember, there are exercises also associated with this particular lecture. So you can always go back again, and work with them. Once you get more a better understanding of the tools we are using. Before we jump into the actual data analysis course, and we start talking about Python, pandas, all the tools, we're going to use import files, read data from databases, etc, I want to show you the environment that we work with. It's our primary environment, it's the tool that we use 99% of the time on its Jupyter Notebook, there are going to be different terms here, I'm going to be referring to it as Jupyter Notebook. But as you are going to see in this, in this part of the of our tutorial, you can see that Jupiter is actually a whole ecosystem of tools. And it's a very interesting project. Jupiter is a free and open source, again, ecosystem of multiple tools. And primarily, we're gonna talk about first, what is a Jupyter Notebook. What you're seeing right here, and you're gonna see live in a second, I can actually show it to you is this thing we're going to use. And we are also going to talk about Jupiter lab. Okay, which is the evolution of the regular Jupyter Notebook. So, I think this could be familiar to you already. Usually the questions in the question is, what's the difference between Jupyter Notebook and Jupiter lab? Well, the difference is that Jupiter lab is just a nicer interface on top of Jupyter notebooks. It's not just the plain notebook. This is a notebook, but I'm scrolling right now. It's also the addition of tree view, it's an addition of get tools, as an addition of command to lead and multiple other things. You can open some files with a nice preview in it, etc. So, Jupiter lab Jupyter Notebook, they are similar Jupiter lab easy, again, the evolution of a Jupyter Notebook. And that's what we're using. Again, Jupiter is a free and open source project. So anybody can install it, anybody can download it, it's very simple to get it set up in your local computer. In this case, we're using something we call notebooks AI, it's a project that provides Jupiter environment for free in the cloud. So you don't need to install things locally, you don't need to put things in sync in your own hard drive, right you That means you don't need to buck it up, for example, because it's just a service, it's all worked in the cloud. So said that, I want to tell you that we have compiled a very quick list of everything, we're going to talk in this part of the tutorial, in this list of two, it's just a thread of with multiple, multiple hints of how to use Jupyter notebooks. So after the video after the course, if you forget some of these concepts, you can always go back to this to it, it's a quick reference for you to have. So let's get started. Why do we use a Jupyter Notebook? Because it's an interactive real time environment to produce our or to to explore our data and to do our data analysis. It's a tool you're gonna fire commands, and it will immediately respond with something back. It's a very interactive tool, when we're working with data analysis, and this is mainly main difference with some other tools like for example, Excel, tableau, etc, is that we are not constantly looking at the data, there is no visual reference, like for example you have in Excel, right? So in Excel, you're constantly looking at the data, you have it in front of you, there are 100,000 cells and you can stroll and see them. The problem is that that's not scalable, right? It's like nobody can work with 100,000 rows in their, in their, in their mind, we will always forget something. So the way we work with Python indeed, analysis is by always having a reference of how our data looks like but always at the back of our head and we're not constantly looking at it. We're like this person from the matrix, you know, the, the the commander of the matrix that commands people to get get in and out. We're basically telling people telling people that basically asking data, right asking questions to the data, and having a picture in our mind of how that's going to work, we're not constantly looking at it, we're just having a reference, or in our in the back of our heads of what our data looks like. So that's why this tool is very useful. This tool is useful Also, if you're just training your Python skills, and or their permanent language skills, because what you're gonna see is it's just a regular Python interpreter. In this case, I can execute some code, that's two one times, actually one plus three, there we go. And the result is four. Right. So this is a Python is a fully featured Python interpreter. The good thing is that again, it's going to respond to us pretty much immediately I create a command and I immediately get a response, I can do something a print here, hello world. And I immediately get a response, I can do Hello, world, times, times three. Again, it's a again, a Python interpreter, a fully feature Python interpreter, but it's not being accessed from a terminal, which you can write this is the good thing about Jupiter lab to have a terminal, I can do Python, right. And I can do two, time three, and I get an answer back. But this is not convenient to work with our data, we need something a little bit more interactive, we can also mix with documents, that's going to be the advantage of a Jupyter. notebook. So what what's the way we work with Jupyter notebooks, there are a few concepts, very important concepts that we are going to follow a Jupyter Notebook is just a sequence of multiple cells, okay, everything is a cell. And as you can see, when I click on these cells, even if even if it doesn't look like being a cell, it is, you will see that these blue thing right here, right is pretty much following me because I'm clicking on the cell, and I'm selecting that particular cell. Everything happens within a cell, if I want to execute some code I can do, again, one plus five, and to get a result or a result back, right, that's, that's how it works. So I'm creating a cell, I'm deleting a cell, I create another cell again. So it's everything happens with a cell, and I'm going to tell you how to add the cells, how to remove them how to execute code, etc. The interesting thing about a cell is that it can either be Python code, or any other programming language you're using in this case is a Python data analysis course. It can be Python code, as we're we were doing before one plus three, this is Python code, or it can be what we call markdown, okay, which is a formatting format, right? To create text, that will be a render with sort of HTML ID at the output. So in this case, this is what the source code of the markdown looks like in markdown, any line that starts with this part, it's going to be a title, in this case, it's going to be the largest, the biggest title you can have is just one pod, and then you keep adding to reviews the size in this case, level three title. And then you can have for example, this is a quote this is bold, this is it Alex, this is a link, right? So let me actually, I could copy the cell and open the source code. There we go. So this is a link right issue, issue is created or it's rendered as a link. So markdown, what is is that is a text formatting tool, right or protocol, we could say that in this case, we just specify us we have some some rules to use in our in our text, and markdown knows how to interpret them and format right or return a formatted document after them. So for example, here, we have green divider, which is a picture and we know it's a picture because it starts with an exclamation marks. And that's that what you're saying right here. So again, a cell can be either Python code, or it can be markdown. markdown is an entire thing on its own. You can get any tutorial online free, it's it's fairly simple to get started with. And it's also very important because when you're formatting your reports, right, when you're creating your reports, you want them to look pretty, you can use markdown for not and what we're going to see later So you can export these notebooks and they will generate PDFs, right. So this whole thing can be a PDF or an or an HTML page. So after you're done with your data analysis, you can hand over to whoever asked for the analysis, a PDF report, which is pretty neat. So moving forward, again, any cell is going to be either markdown, or it's going to be code right here. So these ones code, and you can switch the modes, you can say, this LS code, or actually, let's make it markdown. So right now, if it's a code, it doesn't doesn't matter, or just, it's not executing anything, because the cell is interpret as markdown. So now, I'm switched back to code. And now it works. Again, I said, Sure. It can also be raw, but to be honest, we don't use raw very often. So again, you have this this general cell type, this cell we're using, what type is it? Is it code is it markdown, you can switch it with these with the selector right here. So a few more things that I have to tell you right away, so you can start internalizing them, and it's gonna take some time to get used to it. But once you get used to it, you're gonna move very fast in your data analysis with Python Jupyter notebooks. The first thing is, as you're seeing right here, every cell has been given an execution number. So any, the cells will be moved, right, they will be moving around, you will be moving them around. But you will always know which one executed before another one. And that's because every execution, you run will be assigned an execution number. In this case, this is the seventh time I have executed code. If I execute code again, for example, I don't know, two times two, this is the eighth time that I've executed code. And if I move this thing, right here, if you're reading this thing, top down, you will not be full, right? You will understand this thing. It was moved, the cell was moved, the structure of the notebook changed. But these thing was executed after this other cell, right? xact. And this is seven. So the execution order is always preserved. So that's an important thing. Something else that you're seeing me change the structure, and do things with the notebook without using any menu. And that's because I know how to use keyboard commands keyword shortcuts to run most of these commands. So for example, how can I add a new cell I have these is a markdown cell. This is a code cell, if I need a cell before these one, what's what's that command that I'm going to issue in order to create the cell, in this case, the command is going to be the letter A, I just type A, and there is a new cell creative. How can I delete the cell, it can be two times that the key two times the D key. And again, this is all these reference with built. So for example, right here, whereas hit at some point, you can. Here, you can type, you can press A to create a new cell, you can press B to create a new cell, what we call below. So let me put something here, this is a reference. And I'm going to put here the letter B and it's going to create a cell B below the currently selected one. So the selection here is here in the blue, I hit let me delete this one, I hit B. And again, it's going to create a cell below the previously selected one, if I hit a, it's going to create a cell above that previously created one. So these, these are the mnemonics of the creation. Something else and it's very important why when I'm in this cell, and I hit the letter, a leader, literally it just hits the letter A in my keyword, no control, no command, just a, it creates a new cell, and it doesn't type A inside the document, right? So right here, if I type A, it's adding an actual a character in the cell. Why didn't that happen before. And you're going to notice that when I change, when I'm going to call a mode in a second, you're going to see that the content of the cell is grayed out, show what now when I when I press on the letter A it actually creates to sell and it's not adding content to the sell itself. If I go back again to the other mode, and I'm going to give you a better explanation in a second. If I type anything, in this case, a it's actually appended to the text within it. So this is my interaction to sell modes and this is very important. The Jupyter Notebook is a mode base editor, right? So there are multiple editors are, for example, vim or VI, vi, those are mode based editors, which basically, the behavior of your work will change depending on the mode that it's currently activated. So for example, in this case, I am in addition mode, because any character that I type will be appended to the cell, A, B, C, D, etc. If I switch out of editing mode to what we're gonna call command mode, I switch out of that mode. Now the cell is grayed out, and any key that I hit, it's gonna do something different associated with that key. So A is going to create a new cell above, B is going to create a new cell below, Double D is going to delete this cell, right. So that's, that's the important part of Mo. That's one of the most important parts in order to understand how to work with Jupyter notebooks, the mode that you're currently working with, and there are only two modes, so it's fairly simple. This is command mode. And we recognize command mode, because this cell is grayed out. When we get into edit mode, there is a regular prompt, as you're saying before, the number one on the cell is actually subjects of addition. So that's the way we can realize that, how are you going to switch from modes, in this case, I'm in editing mode, if I'm using my mouse just pointing, I can click outside, I'm gonna get out of the edit mode into command mode. If I point inside and going back again, to the Edit Mode, but let me tell you something right away and then say, we don't like to use our mouse, we don't like to point and click, because that's very slow. We like to use our keyboard, we move very fast with our keyboard. So how are you going to switch from, from editing mode back to command mode, that's going to be with the Escape key to go from editing to command, edit as Escape key, it's going to switch out of editing, but when mode. And if you actually want to make modifications to the cell, basically, you want to get into edit mode, you're going to hit the return key, that's going to get you into edit mode, again. So we have tackle multiple things are writing, again, we said in Jupyter notebooks, we're going to use Python code very quickly to interact with our data, we need a real time, you know, I'm asking a you're answering type of editor. That's what the Jupyter Notebook is. The Jupyter Notebook has these two modes, edit and, and command mode. And then the cells which is pretty much everything is the most important, it's a fundamental part of the notebook, the cell is going to have two types can be either code, or it's going to be markdown, right. And now I'm going to start showing you more features. And I'm going to show you, I'm going to show you the most important commands. And of course, how the what the keyboard shortcuts for those commands are, so you can move freely. And and and work with Jupyter Notebooks in the most efficient way. So let's get started. First of all, for for from the most important commands is moving right. So navigating, it's very simple to navigate, just use your arrow keys, up and down, up and down. And you're going to move around in your notebook. If you wanted to switch the type, right going from markdown to code, etc, you can switch use these drop down or you can press the specific key is to switch to either markdown or Python. So for my markdown, you're gonna switch sorry, hit the M key, that's going to make it markdown. For Python, you're going to hit the Y key, that's going to make it Python code. So M and y are going to switch you back and forth. Keep an eye on the selector you're going to hit y m y m is going to switch it from code to markdown. What else how can you execute code once you are within your typing code and you want to execute it, there are two types of executions you can run. The first one is going to keep the selection the currently selected an active cell is going to stay the same place you are and that's going to be my by keeping press the Ctrl key and hitting return that's going to run decode on the cell there the prompt or the current selected cell will remain being the same. So I'm running this thing a couple of times already on this selection or the currently highlighted cell stays the same, I can change that by using shift return. So I keep the shift key pressed. And I hit return and is going to execute the code. But it will immediately switch the prompt or the currently selected cell to the following one. And that's useful when you have multiple cells, you want to execute one after the other. So you can keep hitting shift, return, return, return return, and it keeps you moving right from top to bottom. Alright, so Ctrl return or shift return to change the execution is the same is just what's going to happen with the currently selected cell. We already saw how to create cells with the A key, we create a cell above with B key we create a cell below. To delete a cell, you're going to hit the D key, the D key two times one after the other very quickly, dd is going to delete these the cell. What happens if you made a mistake, and you want to undo the previously issued commands? Well, the mnemonic here is going to be Ctrl Z, you know the mnemonic, it's not the command, it's going to be Ctrl Z, you only need to press the Z key, you know, you don't need Ctrl Z, and it's gonna undo whatever you did in your previous command. Alright, so a B, D deletion, and then Z to undo the all the commands were saying they all have a correspondence in this toolbar or in this command palette. So for example, right here, I could run this code by pressing these play button right here you see it, the execution is changing. There are multiple ones and you can search them if you don't remember right here. And the neat thing about it is that you actually have the shortcuts to issue the same command. So let's say you don't remember how execute and stay stay in the same cell, or move whatever you can search for run. And you can see what's the name, and what's the actual command that you have, right there, right. So you can, at least for your first ad or a month working with Jupyter notebooks, you will usually need to go back to these commands, and try to remember the the quick shortcuts. And with time and practice, those will just come naturally. So moving forward, what else, we have a few other commands, in this case, we have something to cut and paste the cell somewhere else, just cut and paste, that's going to be x to cut it, or you can also use the scissors here, x to cut it. And to paste it, you can use this button or actually these buttons sorry, or you can just press the V key V is going to paste it wherever you're currently standing it. So I'm going to cut it, I'm going to remove it from here, and I'm going to paste it below there. Or you can also copy it. So instead of cutting it, you can press the C key just going to cut, sorry, copy. And then you can actually say where you want to paste it. In this case, we have duplicated the same cell. And it looks something interesting here, the execution count remains the same. So again, there is like this unique identifier for your executions, which means that you know, when and where something was executed. Moving forward, we're going to use some code here, we're going to import some tools, you can see some characteristics or advantages of Jupyter notebooks and why we use it so often compared to, for example, the regular Python terminal. One very important thing is visualizations, we as data analyst, we're constantly getting data on expressing it through images, or animated animations, right. But most commonly, images. The main library we use in Python is model live. And model lib is a first class citizen in Jupyter notebooks, which means that you can just run the figures from matplotlib. And they will just show up directly in your notebook without the need of doing anything. Crazy. So can you imagine showing these these beautiful picture in this terminal? That's that's very hard, of course. So again, that's one of the main advantages of a Jupyter Notebook. Moving forward, what we're going to do is we're going to first we're going to get some data from a public API. So there is these crypto watch service, which basically has crypto information, Bitcoin, ether, etc. And you can check the docs, we can actually open them. It's gonna give you market data Tesla. You can check the docs and How you can get in this case it's BTC Bitcoin to euro, sexual see if we can change it to USD USD price. There we go. So this is the current price of bitcoin results, surprise, etc. And we're actually going to do markets do we have crack and BTC USD, let's do, let's actually issue the same query we're going to use which is open high, low, close Oh h LC. And don't worry, this looks ugly. But this is actually what we're using. There's a list of results write for all different candles, we call them, we get the idea of the open price, close price, high price and low price. So we're going to issue those, we're going to issue these requests to the internet to these API, the crypto the crypto watch API, so you can get information about bacon to do some analysis, I say they can, you can actually get it from ether for for ether for author different types of crypto or currencies. So the function we're defining is get history, get historic price, it's a very simple function that uses pandas is one of the most important tools, we're going to be using this course. And the requests library, which is also very famous library for Python. And what we're going to do here is we're going to get Bitcoin on ether prize for an entire week. Right. So from ferreted that the second February sorry, February 25, up to today, right? So depending on when I'm shooting this video, and we're gonna get a quick reference of the prices open, high, low, close. So in this case, we have four information per hour. Okay, so this is something you can actually change in the in the, in the request you're making to the API, you can reuse the candles eyes. In this case, we're keeping it per hour. So we have by the hour information about Bitcoin, in this particular market, which is bitstamp. Here, we have these day these day, and these are right, when I'm in the morning, open, close, highest price and lowest price, and also the volume that was operated within this time period. And we're gonna immediately plot the price. So we see that in these time, which I think is an entire day, we the price dropped, it's actually a few days, like an entire week, the price dropped from $9,600 below, right 9000. So it was a pretty significant drop. Let's see ether highperformance. We have here all the records, and how it moved. So this is what I tell you that when you're doing data analysis with programming tool like Python rar, you're not constantly looking at the data. So what I'm showing you right here are the first five records, we actually have. Let's do that. We actually have 169. Records, okay, 169 Records. And this is per hour. So if we do 169 hours divided by 24 hours, we have seven days, right? So we have seven days of data 169 Records, and then we have a little bit more information keeps this to go. I'm gonna get to that in a second. But basically, this is one I tell you 169 Records, to be honest, something you could be saying in a spreadsheet. But I want you to get the concept here. We're not just looking at our data, we have it in our brain, we know what did it we know what shape it has. We know how many records it had, we know information standard deviation, what's the mean of that? Right? So close price was the standard deviation, right? What's the the average, the mean, the median, right? So we have information about our data. It's sitting behind, you know, in our brain, but we're not looking at it. And that's because with a very simple example, with only 169 Records, but in real life, we're dealing with millions of records, so it's impossible to see it. Have you ever tried is crawling in an Excel spreadsheet through millions of records. It's crazy. It's not possible. It's just unusable. So that's again, the way we work with data analysis in Python and R and other tools. We don't constantly keep an eye on the data. We know the shape of it. And we just have these quick references like show me the first five records. I mean, the last five records, show me this chunk here down there, but that's it. So again, these are the visualizations we're creating on Jupyter notebooks. Again, it's just very simple to get the plot done right there. We're going to also see in Jupyter notebooks, a few other pretty neat things. The first one is that we can use another library, which is called bokeem. And the difference is that boakye will have charts that are interactive. So I'm moving it right here, it has JavaScript. And it's interactive, you look back again, to what we had here. This is a static chart, it's just a PNG, you can actually export it as a PNG, there is nothing you can do with it. With bokeem, it's actually a dynamic, dynamically generated interactive charts. So I can, I can zoom in piece of data, right, I can move it around, I can just do whatever I want with it. I can refresh and reset it to whatever it was. And it's a dynamically generated chart. The difference is, if you're working with data, dynamically in your analysis, sort of in your exploration, then boek is a planning tool because you can zoom in, right, so what's going on here, let's, let's look at these things. If we're working on a mean, reverting strategy, for example, we see a high volume, we see a low volume, the mean is going to be here. So we see some mean reversion in there. It's very interesting. If you need to, for example, export a PDF, export a huge HTML file, then static images are going to be probably better. So that's the difference between them. To be honest, model lib is a lot more popular than bogey, we use model live a lot more because it's we actually have a few other tools like seaborne that make it very easy to access and use it. What else Jupyter Notebooks work very well with some Excel, Excel files with all the file formats csvs, XML, Excel files, etc. And that's also the the availability of Jupiter lab. So Jupiter lab can immediately interpret and opens his v files can open with some extensions, XLS files, XML files, JSON files has a very nice editor and tree view for Jason. So the Jupiter lab environment combined with Python Jupyter Notebooks will give you a good idea of Jupiter in general. So in this case, we have just saved I'm not going to execute these you can try it out. But you can execute and run what we have just done and export this crypto file us an Excel spreadsheet. So you can just click on here and you can basically download it, you're going to open it and see what has There we go. So let me reduce the size of this thing. There we go. So you can see that we have just exported to spread two sheets, in this case, Bitcoin on ether, right? With the data that we had in our previous notebook, right. So that's all again, the combination of Jupiter, the combination of Python and the combination of Jupiter lab, which are tools just work very well together. So we're gonna keep moving forward, in this video, this tutorial, I'm talking about more data analysis, in general, we're going to talk about Python, we're going to do a quick review of Python. Maybe when we when I was running these commands, you felt you felt a little bit lost what I was doing with it. So we're gonna do a quick review of Python and all that. And of course, we're gonna get directly deep into data analysis with pandas with some other tools, I want to tell you something before we finish this chapter. And it's not, it's very important for you to get familiar with data analysis, with sorry, with Jupyter notebooks, because you're going to spend a ton of time with it. And it's a very, very valuable skill that you can get if you get proficient, comfortable with Jupyter notebooks, you know, like creating cells, deleting cells, cutting, pasting, moving things around, etc. For you to generate reports Jupyter notebooks are going to be excellent. So keep an eye on it. Keep practicing, it's the only way to learn it to the to the analysis. Keep practicing it, keep open the command Polat. So you can always want if you forgot, how can it caught a cell? Well, there is here it is command x, right? It's gonna just tell you upfront, keep an eye on it, keep working with it and practicing it. And once you get familiar with Jupyter notebooks, you're going to move very, very fast. Remember, they have these nice list of compiled commands and reference you can always access if you need extra help. And we're going to keep moving forward now with more data analysis. Now it's time to talk about NumPy, one of the most important libraries in the Python ecosystem for data processing. In general, it's the one that got pretty much everything started. And if you trace back NumPy, it, it's a very old developed library. 20 years, maybe it's it's an extremely popular library and important library, I'm not gonna say popular. And I'm going to explain why in just a second. But it's a very, very important library in the Python ecosystem for data processing. NumPy is a library that will lead you it's a numeric competing library, it's just to process numbers to calculate things with numbers. And that's it. So NumPy has a very limited scope, we could say, and this is an on purpose, a very simple library, when you look at it, and when you look at the API, which is very consistent, by the way, why is NumPy so important? Well, in Python, numeric processing, and just pure Python processing numbers is very slow. Okay, Python is not slow as itself compared to other programming languages. But when you go down, right to very deep levels of performance, when you are processing large amounts of data, right, and you need to squeeze, even, you know, that tiny bite at the end of your pipeline, you need to squeeze every flow up from your CPU, then Python is not the right tool for non Python as as a pure python programming language. NumPy is actually solving that NumPy is a very efficient numeric processing library that sits on top of Python, and gives you the same API as you're going to work with with just writing Python code, as you're saying here. But low level, it's going to be using high performance, numeric computations and, and arrays of numbers and representations, etc. That's it. That's it for pi NumPy. It's extremely simple from from an API perspective, but it's extremely powerful. Why did I say that? It's not so popular. But yes, it's so important. Well, because in reality, we don't usually employ NumPy directly, you will not see yourself using NumPy. So often, but you will be using other tools in Python, like for example, pandas, and matplotlib. And they are all working on top of NumPy. They're all relying on relying on NumPy for their numeric processing. So that's why NumPy is so important. So the for, at least for this part of the tutorial NumPy. I'm going to divide it into pieces. The first one is going to be a very detail, low level explanation of how NumPy works, why we need to use NumPy. And what are the differences between different bite sizes for numbers, we're going to talk about integers. But this is going to apply for decimals and data types also. And why you need a very low level, optimize to us number. Now you can, you can skip this part, you're going to find in the description of this tutorial, the precise moment in time. So you can just skip and go directly to the second part, which is when we actually start using NumPy. And I show you how to create arrays, how to make computations, etc. So for now, we're going to divide it in two parts, we're going to start first with the low level explanation which you can escape if you want, because it's not going to be crucial, you can easily use NumPy. Without it. We have found that for some of our students, it's it's important to understand the low level basics of it, especially if you didn't have a computer science background, it can help you get you know, raise your right your level of understanding of computers, and how to make your computations more efficient. But don't worry if you if you don't want to go through that now it's fine. You can skip this part and come back later or any other at any other moment. You don't need the ease to use NumPy seriously, you don't need it. It's going to be beneficial, but you don't absolutely lead so you can just skip and come later. So with that said, let's actually go into into a deep understanding and explanation of how computers store integers, numbers in memory and what are bytes bits etc. In order to understand why NumPy is so important. We have to go back again to the basics. What are numbers, how they are represented in computers, etc. As you might know already a computer can only process ones and zeros bits, it can't process numbers or just decimal numbers to be more correct, sorry, it only can process ones and zeros. A computer is just always storing and processing ones and zeros. It's a binary machine. Your memory is the central place around the random access memory in your computer is the the central place where your computer is storing the data that it's actively processing, right. So you have, for example, a hard drive, which stores long term data. But the computer can process data directly from your hard drive. Before doing that, it has to load it into your ram into your random access memory again, usually, right a computer is going to have what eight gigabytes 1632 doesn't matter. Let's say you have eight gigabytes of memory, that at some point is going to translate to number of bits that your computer can store. So if you follow, if you follow these we have right here, you can see the total number of bits available in a regular computer with eight gigabytes of memory. Why is this important? Because again, the objective of these of these tutorial is the objective of this bar, at least is to explain how you can squeeze out of every single bit you can in your computer, right? How can you make it more efficient? For your numeric processing, both in storage? use less memory for the same data? And also how to make it faster, right for your calculations. So in terms of physical storage, or actually memory storage, right? How can we make it? How can we optimize to use the least amount of memory for this given problem? That's the objective of optimizing it, we need to understand how numbers decimals or sorry, integers into decimal numeric system are represented in binary, right. So these table right here shows you the first nine numbers, 01234, etc. and their binary representation. In your computer. Let's say you want to store the age of user age of a user, which is 32. You can't store 32 in here, because your computer again doesn't know about decimals, it only knows about binary. To do that, you will need to find the correct representation in ones and zeros of 3030. All right, sorry, two, which is not this one, to be honest, I'm just making it up as we go. But again, you need to know the correct binary representation of these number in norther. To store that data, how can you know that? Well, there is this whole binary arithmetic, right? There's a whole part of math dedicated to binary doesn't matter for now. But I'm going to just drive the intuition of it so you can have a better understanding. And if you're interested, you can dig deeper later. So basically, any decimal number needs to be stored in a binary format, which of course only steaks ones and zeros. And what we usually do is just we keep increasing zeros and ones in positions, right. So in this case, we have the number zero, the number one, that's fine. Once we need to store the number two, winning now to increase the number, the position right here we need to increase, right, so we need to go from two to one zero, we'd go to the number three, it's one one, and then we need to go to number four, we need to increase positions again, because we only have two symbols, zero and one. So as you're seeing right here, up to these level, we need only one position. Up to this level, we need two positions. This level, we need three positions. And these levels going to need four positions. And you'll see how the size of each of these is increasing. And it has a an explanation behind it that we're going to see in a second. So the question is how many decimal numbers you can store with n bytes and bits, sorry, bits. So let's say we have n bits. And let's say n is equals to three. That means that you only have three positions, right three bits, how many total decimal numbers, you can store with it? Well we can store 000 we can store zero, we can store 100 we can start stores are you one zero, right? So in this size, we can store up to here, we can store up to seven numbers 111 is equals to seven was, once we've filled all the positions, right, we've reached the limit, right? The largest number, the largest binary for this amount of symbols or positions. That's the number seven. So these means that with three numbers, you can start from zero from zero, here, zero up to one, one. In total, you can store eight decimal numbers, here you have eight decimal numbers 012345678, total decimal numbers from zero to seven. The equation if you want behind this is as follows. If you have n equals three, and it's, in order to know how many decimal numbers you can store with those bits, it's two to the power of n, in this case, is total a bit. So if we go back into our drawings, we said that with three bits, we can store up to eight decimal numbers. And again, the equation is two to the power of n is going to give you how many decimal numbers you need. You can always do the opposite process using logarithm and get how many bits you're going to need to create to store a given decimal number. I'm, I'm not going to get into that. So we don't complicate it. But again, the math behind it is extremely simple. So now, moving forward, we're going to delete this whole thing. Moving forward. Why is this important? When you're working with your data, when you're doing your data analysis, you know what, what data you're what type of data, you're working with their own numbers, but numbers only usually have a connotation behind, right? So let's say that you have here it's a table of people, and you have the total net worth of the person. And also you have the age of the person. The age is a value that will range between what zero, right? Just born to, I don't know, 120, we can say I don't know, what's the maximum age registered right now, the oldest human being but zero to 120, it seems, seems reasonable. In your other column net worth for this person, the range is it's completely difference. We can go from something like $0 up to, I don't know $60 billion, I think Mark Zuckerberg or Jeff Bezos or one of those. So we go from zero to 62 billions in this case, if there are dollars, what happened if this is a highly devaluated currency? Would we have to go to trillions, right? So these two even though they're just plain numbers, and we can say they are integers, even though these are pulling numbers, they have an integers, they have a different connotation, and they will need different requirements in terms of storage size, right? So if we say that nh goes from zero to 120, we don't need so many. So many bits to store it in memory, right? We can do the math, actually, how many bits Do we need in order to store 120 100? And what do we say 120. Right? Well, if you do the math, you will see that two to the power two to the power of seven is 128. So if you have if you have seven bits here, seven bits, you're going to store from zero, up to 1111111, which is actually 127. Okay, these number, all ones, seven ones in binary is equals to 127. in decimal, in total, we can store 128 numbers 00 matters, up to 127. So that means that for our column right to column, age, here age, we only we can use the size of the men We need to use is going to be seven bits per user, or costumer or person, whatever. What about these number right here, if we have to go up to a couple billions? Well, in that case, the numbers a little bit more complicated, we're going to need, for example, we can say 64, or 3232. It's actually 64, probably, but with 32 bits, right, you can store up to from zero up to these volume. So again, I don't know about the currency we're using or anything, so we can assume. But here, we need 32 bits in order to store that. And now you can do the math, how many how much memory space you need, in order to process this data? How many records Do you have, if you have only 1000 Records, that's not significant. You can use whatever, you can use 64 bits here to store the age, and you're not going to have a problem. But what happens if you have more what happens? What happens if you have the entire population of the earth, you have 7 billion records here 7 billion records, then every bit that you're saving in these columns is going to be important? Because he's going to take a ton of data. And of course, you have a ton more columns, right? What happens if you are processing trillions of records from financial transactions, right, you want to be very, you want to be very efficient and optimize every single bit, you can't. And that means again, selecting the correct number of have a bit per the columns you're currently processing. So so far, so good, again, when there's 10, that the the number in decimal that we need to store has a correspondence with emits, right? eight bits is one byte. And the more we can optimize that, the less memory we're going to use for our obligations. Where does NumPy come in place? Why are we talking about data in these NumPy lessons? Well, they're right. The idea is that NumPy is a library that will lead you has a very advanced numeric processing, in order to let you select the number of bits you want to take for an integer. Even more, let's say you for forget about NumPy, you want to process this thing with pure Python. So you x equals five, for example, working with Python, you create a number, we're storing age as a five, how many bytes? How many bits? Do you think the simple variable takes in memory? How many? Well, in reality, even though we think it should be around, what, three, three bits, eight, let's say to be simple, too simplistic. In reality, for Python, this is going to take around 20 bytes. Okay, so we are wasting a ton of memory in order to store this number. And why is that? Well, because Python is a high level, object oriented programming language. The reasoning behind it is that Python is simple to write, write simple to also read and, and, and code on top of it. But again, in order to create that simplicity, in its syrup, all the numbers in objects, which have all these attributes, that if you know, advanced Python, you're going to recognize that are not necessary. So these is taking a ton of memory. And a regular, very simple number in Python ends up consuming 100 times more memory than what it should be consumed. And this one NumPy comes in place in NumPy, you can create numbers that are for example, you can control the size, in terms of bits, you can say I want to create a number that has only eight bits. And that's it, that you're going to create a one byte integer, and you're very precise and how much memory it takes, you can create a number that it's actually need a little bit more space, we're going to do NP int, and we can hear us a talkie, you're going to get auto completion 6016 bit or eight or 32 or 64, right. So we can actually be a lot more precise in the number of bits that we need. And this is extremely important for again, our high level processing. On top of that, NumPy is our array processing library at NumPy is 99%, about processing a race constantly processing erase the data structures we have in Python, the built in data structures we have in Python, for example, the list dictionary, they are not optimized for high level computing. So if you have a list of numbers in Python, let's say you have, I don't know, l equals 3224, right, you have three numbers in your list. In Python there, it's not guaranteed that the least they'll the list is gonna contain all the numbers, three to four in contiguous positions is gonna, it might put them in separate positions in memory. On top of that, you can't rely on advanced CPU directives and instructions for processing matrix matrices, sorry, because Python, again, is wrapping these things in objects. So there is no access to these high performance, low level instructions with NumPy, that changes because when you create an array NumPy, you say, I want to create an array of three numbers, and they are all into eight, then imposition forget about this is not these are not bytes I am, I'm using these drawing as a general representation of memory. So in that case, in NumPy, when you create these three element, int, eight array, it's going to create those three elements in contiguous positions in memory, three to four, and they are only going to take that amount of memory the police said they were going to take and on top of that, we can rely on a bunch of very efficient low level instructions from your CPU for matrix matrix calculation, this is something that it's a little bit more advanced. And it's something that has exploded in the past 10 years CPUs with more with richer instruction sets, and the same thing for GPUs, you might have heard, especially with machine learning and all that we need, we need fast array processing, when we are storing features and weights and all that's a topic for for different story. But again, the idea is we need right a ton of week, sorry, we can use all these important and very efficient, low level directives from our CPU, which makes our computations a lot faster. So again, as a recap, you don't need to know all these to work with NumPy. That's the first thing. Second, you don't need to get extremely, extremely conscious about all the numbers you use. At the beginning, you're just going to use NumPy as it is, and you're going to use just the default types that it picks in 38 cents or in 32. In 64, that's fine. But then, with when you get into bottlenecks, when you're working with with larger amount of with more amount of data, then you might need to get into the details of that size of the integers that you're using. And this all applies to float. So I'm just using integers because it's simpler. But it's all applies to floats. So again, NumPy, the main advantage is that it's it has built in very fast and I raised kit, take advantage of CPU instructions for matrices and arrays and all that. And it also has a very efficient representations of numbers, right are not the regular objects of Python. Again, recap, you don't need a list. If you want to get into more details, I recommend you to get a little bit more understanding about binary arithmetic, and how numbers are uncomputable architecture, how numbers are stored in memory, etc, especially for floats and all that's a completely different representations. So with that said, we're going to see now how we actually use NumPy without worrying so much about the low level details. And that's the beauty of NumPy. So we have already done our low level explanation of binary arithmetic, why unknown vice important and all that if you skipped it, that's perfectly fine, you will not need it. The reasoning was to include was that if you're in this tutorial, you're probably looking for fast and efficient options to process large volumes of data. And that's when all those things come into play. So let's without further ado, let's just get started and start using NumPy as a library. So again, as I told you, a NumPy is a very simple library for array, processing and numeric powers. To sing, it has a few objects, numbers, floats, integer floats, arrays, and that's it. And it's very simple, but it's extremely powerful. So, in NumPy, we're going to create these arrays, which look a lot like Python lists, but there are going to be significant differences. The first one is, of course, performance. If you go to the previous part, when we were discussing the binary representation of an array of numbers, in Python and NumPy, you're going to see the difference between them. So in this case, we're creating two arrays. And you will see right that the creation is extremely simple. The only thing that changes we need to add this NP dot array, and then we're passing in this case, a list of numbers. This is something we will usually be reading from external sources. Now, how can you access individual elements of a NumPy array is this works in the same way as with a Python list, you can say give me the first element, give me the second element. And it's zero index, like, again, in a Python list. Slicing works the same way. So in this case, up a zero to something, a one up to three rights, just getting low level, right, on high level of the index, negative indexing, and steps, they all work in the same way as with a Python list. So if you know how to use a Python list, you will know how to use a NumPy array. There is one new thing right here so differently from a Python list. And it's what it's called multi indexing. Let's say you have a, an array this case B, and you need to extract three elements out of it, you need the element of the first position, third position and last position, you can just type B of zero, B, A to B at minus one, or, and this works, this also works for a list. Or you can use again, mod the indexing, which is from B, I want to select the elements in zero to n minus one first element, third element on last element, right, so you pass an int, another list containing the indices of the elements that you want to select. And in this case, the important part is the result. It's another NumPy array, it's not just individual elements, you're creating another NumPy array, which again, if you're processing, it's gonna be a lot faster. So arrays have types associated. And this is related to what we were speaking before. As a NumPy array is a continuous is continuously assigning memory, the NumPy library needs to know what's the type of the object you're storing, you can't just or you know, anything, a string a number within it, because it will not be able to provide performance and optimizations for arrays or non consistence insights. So for example, when we create these arrays only had injures by default NumPy selected in 64, is because of the platform, it's a 64 bit platform, you can tune this, and you can select us, we're going to see other sizes in a second, when we created the array B that contain decimals or floats, it assign a different type, which is float 64. Again, the default type is always six, at least in this platform that is 64 bits, it's going to be float 64 and integer 64. You can always change that you can say Actually, I want these, even though these are all integers, I want you to create them using a float type, or as we saw in our previous video, we can say it should be actually type integer x. So smaller integers, for performance, for performance for better performance. Alright. So moving forward, we were also going to see a few other types like for example, strings on the regular objects. But as you're going to see this, there is no point of storing these things in NumPy NumPy, stores numbers date Booleans, but not a regular individual objects as we're seeing right here. There is a way to store strings, it's perfectly valid and it has its own time. Its own type sorry, and it's related to the Unicode representation memory etc. But again, NumPy is usually used for numeric processing. So the idea of NumPy arrays is we can create multi dimensional arrays we can create the what we had created before. This is a one dimensional array right? Just one dimension, you can create matrices, which in this case are two dimensional, we have two rows and three columns. And NumPy has a ton of attributes and functions to work with multi dimensional arrays. So the first thing we're going to see is the shape of an array, which is two rows by three columns, how many dimensions it has, it has one vertical and one horizontal, we have two dimensions. And what's the total size of the array in this case, the total size is six, the total number of elements we have, let's go one dimension. Further, let's create a three dimensional object, a three dimensional array, which is basically a cube. In this case, for B, we have that the shape is two by two by three, the number of dimensions is three, and the size is a total count of elements. 12, you always have to be careful when you're creating these multi multi dimensional arrays. If the dimension dimensions don't much, like in this case, right here, where we have this second list that only has one less than bits in it, then the dimensions will not match. And it will just tape you they'll use sorry, that the array is of type objects. And the shape is only two only has two elements, these one element, and there's another element. So in this case, we've done we've done it wrong, basically. And you have to be careful when creating these these objects by hand. So how can you index and slice matrices? We've done it for a one dimensional array. So we were selecting elements, individual elements, give me the first element, give me the second element cetera? How can we do it with a matrix with a matrix, what we're going to do is going to be very similar to what we did before. The difference is that now we have to account for multiple dimensions when I do give me a at one, is it the column add one, or is it the row at one? Well, as you can see, it's the row. So this is going to be right here. 012. Right. And there is also another dimension, right? So this is 012. In terms of index, index positions for our slicing. So here, how can you get the first element, the first element of this second? rope. In that case, you're going to first select the first row, the sorry, the second row, and then select the first element. And that's what you get number four. But there is a better way, which is by using the multi dimensional selection of NumPy. In this case, you're going to say from this matrix, I want to select and here you're going to pass a in this case, you're going to pass dimension one dimension to dimension three, dimension four, etc, right. And these are selectors for each one of those dimensions that you're passing. In this case, we say, for a row level one, the element, the position one second element, and for a column level, we want the first element in it. And it's the same thing as we did before. The advantage of this index and keeping in mind and remaining it is that it will also let you add slicing, right, so you'd say I want to select every thing from dimension one, which is rows. So in this case, you say from zero up to two is these two ones, the two is not included upper limit the same as as Python. And then you can also pass other other dimensions, you say, I want to select every row, that's fine. But then I want to select from column level, I only want to select the elements up to two. So these two and these two, and the two, right, so 124578. These all works as intuitive as it gets. Remember this syntax is the important that you need to keep in mind. Moving forward for modification, you can say I want to assign these new array to this entire row, right? So if the dimensions match, that is going to work now 10 is equals it's added to the second row, or you can just use what we call usually an expand operation. We're just going to say for row number two, I want to assign the number 99 and NumPy is going to take care of expanding it into this corresponding array, given the number of dimensions that you have So so far that selection, it's simple, we're going to see also is that NumPy has a huge advantage of containing a ton of operations you can perform on top of your arrays and matrices, your multi dimensional arrays in general. So the first one is the all the summers basic methods we have. So given an array, all these methods are already built in the sum, the mean average, right, standard deviation, variance, etc. And that also works for matrices. So in this case, we can get the sum the mean standard deviation, or we can do it per axis. So this is very useful, we can get the, the here, let's compare these two, there we go, we can get the some of these, what is this first column, the second column or the third column, we can get it the first row, second row and the third row. So it's either this dimension, this dimension one, or it's a vertical dimension, which is x equals one, right? So per row per column. Or, if you have more dimensions, you can just keep increasing the number of this answers. And that's just going to work as expected. Broadcasting vectorized operations, this is a fundamental topic that we're going to talk about. And it's going to be extremely related to Boolean arrays. And these are a few new things that you have to keep in mind with working with NumPy. And now we're going to talk about vectorized operations and broadcasting, which can be a counterintuitive topic at the beginning, but then you're going to understand how much sense it makes. It's one of the fundamental pieces of NumPy. We've seen how NumPy works in a very general way we saw the multi dimensional arrays and all those advantages. But you might be thinking, I mean, I don't need another library just to complete the summer domain. When I show you the vectorized operations and broadcasting part, this is going to make a little bit more sense of why NumPy is so important. So to get started, we're going to have these array, which is a right, that's just very simple array vectorize vectorized operations are operations performed between both arrays and arrays and arrays and scalars, like in this case right here, which are extremely fast, they're optimized to be extremely fast. In this case, what we're going to do is we're going to sum the entire array plus 10. And what it means we're going to see an example of what happens without with Python. But what it means is that let me show you the results, that each one of the elements within the array will be applied the same operation. So usually, that's the concept of vectorizing an operation you have the number and then this operation is applied to each one of the elements in here are actually in these other one, right, so here and here and here. And here to result in these new array, the operation is expressed at an array level, right, we say a plus 10. That's it. But then again, internally, this is broadcast said to each one of the individual elements within the array. And this gives me how a plus 10? Well, a times 10, for example, which also in this case is we're playing the times 10 operations to each one of the elements in the array, resulting in a new array with the result of that operation. And these resulting in a new array is very important, because as we're going to see, NumPy is an immutable first library, it will not any operation, you performing an array will not modify it, but it will return a new array, if we check the status of a, you're going to see that the elements are the same, it has never changed, we are creating a new array and returning it. There are ways to override these behavior if you want. And this they all these operations were performing these way always have the interface of plus equals minus equals times equals etc, which will indeed modify their rights. In this case, we're making a broadcasting operation, adding 100 to each one of the elements in this array. And now this operation was immutable. A was modified and did it hasn't returned a new operation. If you remember from your pure Python skills write the correspondence of vectorized operations are list comprehensions, in which you're expressing an operation for each one of the elements in your collection. Right. So that's a list comprehension. It's a it's pretty similar to what we're doing with NumPy. The main difference is that this is all optimized and extreme. It's extremely fast. So, the operations are these vectorized operations are reduced broadcasting doesn't need to be only between arrays and scalars can only be between arrays and arrays. So in this case, we have a and we have B and showing you right here. And we can do something like a plus b. And what you're saying is that if there is a correspondence, right, so zero plus 10, one plus 10, two plus 10, right? Let me, let me do it in this way. 110 210 and 310. There we go. And that's the result that we get right here. So these for these to work, you of course, need the arrays to be online and to have the same shape. But when that does work, then the operation is extremely fast in memory. And it's aligned, it's a vectorized operations with seen so far. Why is this topic of vectorize operations so important? Well, because of the following, which is bull in a race. And this is a very, very, very important thing. If you don't completely get it now, I asked you please, to go and check the exercises we have for this lesson, because we're gonna use it a ton. And we're gonna, we're gonna see that in pan, this, the same syntax, the same primitives of Boolean arrays, a play apply, and we're going to use the same things. So why are Boolean arrays similar to vectorize? operations? Well, all these operations we've had performed here are just arithmetic operations, mathematical operations, plus something times something, etc. If you look at the operators that you have in your programming language, it's it's not only mathematical operators, like plus or minus, or times, you also have Boolean operators. And the question now is going to be what happens when you apply Boolean operations, when you apply Boolean operators to it. So given our right, we had, what ways we had to select different numbers. For example, in this case, we need the first and last element, we do zero and minus one. That's, that's the way we saw with NumPy. We also saw the traditional Python one, right, so we can say a zero, and also want to get a minus one. So this is the first, the first way of selecting these elements, we know there's a second way with multi index selection. And there is a third way and this is new, which is with Boolean arrays right here. So in this case, we're gonna say I want to select the elements in this order, right? And you're gonna pass either true or false if you want to actually select the element or not, right, so if you have four elements, you have to pass four Boolean values, saying, I want to select this element, I don't want to select these ones. I mean, I don't want to like this element. And I do want to select this element right here. So I want the first one, and the last one, and the result will be the same 030303. So so far, it's it's nothing terribly new, right? So this is new, but it's not extremely complicated. We are showing you a brand new way of selecting data, you can select regular Python multi index, or a Boolean array. Now, you might be thinking, well, I manually write true false false, true, true false, for I don't know how many records you have a million records, this is not scalable, right, you will not say to write all the strong forces. But this is actually very important, because these arrays are the ones that are the result of broadcasting Boolean operations. So we saw again, regular arithmetic operation like this, but we also have it for Boolean operations. So we what happens if we ask a greater than or equals to the number two, right, and array A is this right here is 0123, then the result is false for zero, false for one, because they are not greater or equal to do true for number two, of course, and two untrue for number three. So all the individual elements that match this condition will have true and false. In other cases, this is the power of Boolean arrays, we will be able now to combine these operations. So now we can do a greater than or equals to two, right that a equals A being greater than or equals to the number two. The advantage of this is just filtering, we're filtering No, no numeric arrays very quickly with a very familiar syntax a greater than equals to and we just provide that as the index of the operation. It's pretty much What is happening right here? We're saying use these Boolean array. It's a willing list, right? is a Python list with Boolean, to filter or sorry to select elements based on that. But the question is, how do we construct that list of Boolean? Well, in this case, we have constructed it by including a predicate by including a condition that needs to be matched. The result, again, is filtering. It's a query method, you're looking, looking up some data, you're saying, Give me all the elements that match this condition. So you can say, for example, these values can be of course calculated, you can say, give me all the elements that are greater than the mean. Or you can actually provide other Boolean appraiser operators like for example, all the elements that are not greater than the mean. So that means they're less or equals and the mean, or you can also include all their Boolean operators like or, or, and so or n and in NumPy, are expressed with a pipe or an ampersand ampersand. Because we can't use just the regular or and then in Python, we can, but it's a good choice, they've selected this. So again, this is the concept of Boolean arrays, we are going to construct these arrays that artist Boolean representations or Booleans, based on conditions, right, so we have this matrix, and we're gonna say I want to select these one, and these one end is one, etc. So in that case, this is the result right here. This is the result of that. And we can generate a dynamic Boolean array, we never manually type all these right, we don't sit and say true, false false through etc. We just Run Query filtering option, a Boolean operation, which results in a Boolean array. And now we can use it as filtering. So again, the idea here is that the operations we saw in broadcasting before, a timestamp are also defined for Boolean operators. Boolean operators return Boolean, a race, which can be used in filtering, that's the idea of all of it. And you can even combine these operations, you can say, A equals zero, or a equals one, a less or equal to two. And it's also divisible by zero, you can combine all these queries. So now it looks a lot more powerful than when we were doing before. So moving forward, talking about linear algebra very quickly. And this is we're approaching the end of the NumPy lesson. The part the important part of of linear algebra is that NumPy already contains all the most important operations for it already optimized with low level semantics, it's going to be extremely fast, adult product cross products, and all that transposing majors is all that works as expected. And again, these might be very important, specially, for example, machine learning, etc. It's it's extremely important. And finally, to wrap up what we saw in our, in our binary explanation at the beginning, what you might have escaped is the difference in sizes between NumPy and Python, the differences in terms of performance between them. So in Python, a regular number, this is just a regular engine in Python, that total size is 28 bytes in order and just let this thing for a second. The total number of bytes, not bits bytes that you need in Python to store a simple number, as the number one is 28 unit 28 bytes to store just the number one is extremely, super space consuming, right? It's not very efficient, larger numbers will even take more bytes to store them. What's the size of the integers? Well, we've seen it we have, for example, we can create integers with eight bytes. We can create integers with one byte right which were something like here we have np.int eight will already know how many bytes has only one byte, right, but you can have control of how many bytes or bits write your numbers will take. And you can see here the difference between the size of an integer in Python which is extremely large 28 byte on NumPy and also the difference in performance. Let's say for example, we want here you have the ultimate difference in size of lists, which is also significant. But I want to focus on performance, we have two elements two, we have one list that has the first 1000 numbers, I will have a NumPy array that has the first 1000 numbers, we're going to perform the same operation in both of them. Let's use the Python one. First, we're going to do the Python one first. In this case, we're, we're squaring all the elements in the list, okay, the elements A squared, and then we're summing all the operations might so we express it at saying, create a new list, x times x, sorry, squared, 4x, nl, and then some everything, how much time it takes 321 microseconds, we're gonna do the same thing with NumPy, we're gonna say NP dot sum, a square. And you're gonna see that it's a lot faster in the NumPy perspective, then the Python perspective. And these are all very, very tiny, tiny operations with small numbers. What happens if we add more numbers, let's add two more numbers here. That's odd. Two more numbers here. And we're going to do the same two operations. So as you see here, that that the units have even changed, we're still in the microsecond layer here with NumPy, we've gone to the millisecond layer in Python. So as the size of your objects increase, NumPy will prove to be extremely fast compared to Python. So there are a few other functions you can see here, for example, extracting normal, random numbers, etc. I'm going to live let these for you to look, if you're interested in them, I remember you have the exercises, which can help you solidify all the concepts we discussed. And we're going to move forward now to work with pandas, we're going to see also visualizations are gonna keep moving forward this data analysis with Python tutorial. Now, it's finally time to talk about pandas is the most important library that we use for data analysis in our day to day basis with Python. It's a library that will aid in the entire process of your data analysis project, you're going to start getting the data, step one, getting the data from multiple sources, like databases, Excel files, CSV, files, etc. That's all gonna get into pandas, you're going to be processing the data, right? So you're going to be combining merging, doing different types of analysis, you're going to be visualizing the data, right, so a bar chart, you're going to be visualizing the data with pandas, and you're going to be creating reports, you're going to be also doing simple statistical analysis, you're going to be doing machine learning close to it, with the help of other libraries, but everything from the platform that provides the pandas library, it's, again, one of the most important libraries in in in the data analysis data science ecosystem with Python. pandas has recently released the version 1.0. So we are talking about a very mature library. It's been around for a long time now. And again, it's the primary library that we use in Python for data analysis and data science. So I'm going to do a quick introduction to the data structures of pandas house, and we're gonna understand how they work. So you can start building right the phone, we're gonna start building the foundations, I need you to be very familiar with the way the data structures from pandas are processed. And then we're going to move into other things like reading files, grouping data, etc. So to get things started, we're going to talk about the first data structure to pandas house, which is this series. In reality, pandas has two main data structures that it uses all the time, and it's the series under the data frame. The data frame is the one you will probably be more familiar with. It looks just like an Excel table. But we're gonna start first with a series. Okay, so just stay with me here. We're going to talk about a series for a second. In this case, we have important pandas, and we have also imported NumPy. As, as you might imagine, as I told you before, in the NumPy part of this tutorial, we're saying NumPy is fundamental for data analysis because every other library pandas, matplotlib, they all sit on top of NumPy and you can see it right here. We're gonna be using some features from NumPy within this lesson, too. So these is a series in pandas, what you see right here, it's The concept of a series is this ordered sequence of elements right? Or indexed right with they are all indexed by a given index, of course. And you might think that this looks a lot like a Python list, right? So in this case, we're storing the population of countries, right in millions of inhabitants. In this case, it's jevelin. g7. pub is because we're getting the population of the Group of Seven, you can console the Wikipedia page. But basically, we are storing population in here in this series. And again, it looks a lot like a list, but we're gonna find a ton of differences in here. So the first one is that the the series has an associated data type. And this is something we saw in NumPy, when a NumPy array couldn't hold different types of objects, we were all we were only having one type of object. In this case, it's float 64. So all the numbers of the series will be of type float 64, the underlying data structure, the 10, this is using to store these objects is a NumPy array. So a second difference we see very quickly is that zeros can have a name, right. So now when we display the series, we see that it has a name. Now it might not make a ton of sense. But once this series is part of a data frame in the form of a column, then the name is going to make a lot more sense. So moving forward, again, we saw that A has a type. And again, this is because the backed the data is backed by a NumPy array that you can always consult, you can check values of a series. And you're going to get the array that it's backing up that pandas series, right, so you can see that it's a NumPy array. Once you have these series, we were just consulting here, design pop, you can in you can select elements as you good in a regular list, right? So for example, give me the first element, give me the second element, the last element, etc. And that's because a series inherently has an index, similar to list a list when you create a list in Python, right? So if I create L equals a, b, and see, but there is something wrong here missing, quote, this list, we don't say it right. But the idea is that there is an index here, zero, this is one, and this is two, right? In the pendous series, this is a lot more explicit, each element has an associated value within it. And you might think that is pretty much the same thing. They're all they're both the list on the series, there are both sequences, they're ordered sequences of elements. But we're going to see that there is a fundamental difference, and is that we can arbitrarily change the index of a series. So by default, when we created it, we didn't assign any indices. So by default, it was a range index from zero up to n minus one elements. But you can actually arbitrarily again, say, what is the index of your series. And in this case, these data structure these series has now these indices that we're seeing right here. Why is this important? Because now we're going to be referring to these values, not by a sequential position, but by a name, but by a label by the index, which has a meaningful name for us humans. Okay. So now, these thing looks a little bit more like a dictionary we could say, than a list, we started thinking that a series was similar to list but now, we can think that a series is limit similar to a dictionary. But wait, don't get me wrong here. The series has a fundamental trait, and it's that it's still ordered something that didn't happen with. With dictionaries, dictionaries in Python, are not ordered, actually, in python 3.7. They're ordered, but we shouldn't be thinking that they are ordered their unordered data structures. In this case, a series is in the order. So it has both those advantages. It's ordered candidates always before friends, that's as we decided to create it, but also it has names or labels or keys associated with the values as a dictionary. So this is creating the series from scratch. Right? All these methods, you can see you can create a series bypassing the index it doesn't have To be a two step process where you first created the series, and then add the index, in this case, you can do everything at once. And the indexing is now going to be done by those indices, right. So those labels that make up the index will be used to index specific data. So g7 pop, we see has these countries with these population. And now, before the index, we were saying, I want to get what's the population of Canada, and then we had to remember, what was the position of Canada, oh, it's the first help countries, we have to do g7, pop zero. With the index, now we can just consult what's the population of Canada, what's the population of Japan. And as you can see, the syntax is the same as with a Python dictionary, it's just pretty much same, you pass the key and is going to get the value. So again, summary, the advantage of a series is it's it's a ordered sequence of elements, backed by a NumPy array, very efficient very fast. But it also has an index that can take any labels we pass, so it's going to make it a lot better for indexing, you can steal when you have a series, you can still get the elements by the sequential ordering. After all, it's a sequential data structure, and doesn't matter if you have in an index, you can still say, Hey, I know we have on the index. But if you want to get the last element, or the first element or the second element, you're going to do that by using the attributes, I look at it and say to this series from this series, I'm going to ilok locate by sequential position, these element the element in position zero or the last element. And that still works as expected series also support multiple indices as we saw with NumPy. So in this case, we can get two elements out of two, three n elements, you can pass multiple indices. And the same thing happens with more with sequential multi index series also support range or selection or slices. But there is a fundamental difference here, this is very important here attention, there's a fundamental difference with Python, and it's not in Python, the upper limit of a slice is not returned. So from our list that we created before, if I do l, up to number two, I don't get the index See, right, so this is zero. This is one, this is two, two is not included in our pandas series, the upper limit is indeed included. So if when you asked from Canada up to Italy, Italy is in the result. Okay, so this is something to consider when using index selection in pandas, I think this is still valid, it's very, I understand the reasoning behind it's just different from Python. So, you should remember, Boolean arrays, which was a topic we discussed in our previous lesson of NumPy. Boolean arrays is still a thing in pandas, the difference is we instead of saying Boolean arrays, we should say Boolean series right, the idea is that we will be able to perform operations on top of series. So for example, right here we have mathematical operations on top of series in this case, we have the zero D seven pop, which as I told you the beginning is in millions of inhabitants. If we want to get the series of interest units, we will need to do Jessamine pop times 1 million and there we go now is in terms of units these operations right these vectorized operations the bras these broadcasting operations can also be performed with Boolean operands. So instead of a multiplication, a summation and subtraction, etc. We can add we can use a Boolean operators. So in this case, we get asked what are the countries that have more than 770 million inhabitants we will receive receive their assault is a bull in aerates, Nebraska, right? Well, let's hear it you know, but basically, it's the same concept of with us with a NumPy Boolean array. Canada, friends, they do not have more than 70 million inhabitants in Germany does have seven more than 70 million inhabitants here. 80 on the same for Japan, so Japan Here is the same on the same for the US, the US also have past more than 70 million inhabitants. So again, the Boolean array or Boolean series in this case, works in the same way, as with NumPy. And selection also applies. So I can now select, I can say, give me from these series g7 pop, all the countries that have more than 70 million inhabitants, the value is more than 70. So now, again, we are building filtering, we're building a query language if you want on top of pandas, we're selecting data based on this condition. Remember, when if you ever have trouble remember all these, the idea is that you can always track down the way this index is being built. In this case, we are it's not that the selection knows anything, these first election knows anything about how to select countries with more than 70 these operation was performed first, which resulted in these series. And now this series will be indexed by these array, this Boolean array. And the result is as you can see it, and again, these operations can be run with calculator methods, and all the operators we saw in our previous lesson, which was not, which was or this irregular pipe, or, and amberson, which is the and all these can be applied in any order you want. So if we read this thing, which is complicated in purpose, it's worth saying give me all the elements that are above the mean, minus two standard deviations or below the mean, actually, above the mean, and here was below the mean, or if this isn't correct, but it doesn't matter. It's just an OR operation between two ends of the it's actually, it's above the mean, minus the standard deviation. So we are applying this operation or right, that operation we have before so they're not the or, and the and they all work with Boolean selection as well. The operations we saw from a mathematical perspective mean in in statistical operations, we saw a NumPy. Some mean, average standard deviation, we're actually using standard deviation before, they're all still relevant in this case, but also you can use traditional NumPy functions with our pandas series, because again, a panda's series is internally backed by a NumPy array. So this is all the same, as you can see, here is an example that it's a little bit more clear, we're getting all the countries that have more than 80 million inhabitants, and all the countries have less than 200 million inhabitants. So it has to be above 80. But it also has to be below 200. Okay, or in this case, we say either above 80, or below 40, or below 40. Right. So that's with the OR operator or the NOT operator. Modifying series is relatively simple. Whenever you have a value, you can just assign it all together. In this case, we're saying Canada is now 40.5. I don't know why we just wanted to do it. This is by index, you can also do it by sequential positions. So in this case, we're going to say the last country should have 500 now. So we're going to see a right here, the last country has 500 now, or you can also modify elements based now bool and selection. So you can say all the countries that have less than 70 million inhabitants, all these from our previous query, all these will now be 99.9. So as you can see, it has changed all these countries. So this the assignment works by direct indexing, or also works by Boolean indexing. And this is going to be extremely important when we are cleaning data. So let's move forward and start talking about data frames now before you have exercises in for series, and also for data frames, so I recommend you to check them out. So talking about data frames, this is what a data frame is going to look like. It's pretty much the same thing. us an Excel table. So this was our series and this is going to be our data frame. It's a table. So it looks a lot like an Excel spreadsheet. And actually, it's very common to create pandas data frames out of CSV files, which are tables basically. And I'm going to create it we created with these data frame object I created. There you go, these are data frame. And as you can see, right, it has columns that we have assigned. In this case, we were designing the columns, and we have rows of values right below each one of these columns. Why? What's the similarity with with series, and it's not a data frame column will be basically a series. So we can think a data frame is a combination of multiple series one per column, we're going to assign an index to the data frame the same way that we did with our series. So in this case, this is our data frame. Sorry, right here. This is our data frame that has the index, right? And it has the columns as we had before, what columns Do we have, what's the index of the data frame, these are all attributes that you can consult, there are a couple of very interesting methods from data frames that we use all the time. The first one is the info method. That's going to give you quick quick information about the structure of your data frame. Right. So it's going to tell you what columns you have population GDP surface area, HDI continent. And it's also going to tell you the types and how many no values you have, it's actually telling you how many non null values you have. But we use these when we're cleaning data to quickly then define those columns and have missing values, we can check for the size of the data frame, we can check for the shape. And this is similar to a matrix right, a two dimensional array in NumPy is pretty much a data frame. And also similar to info the voice again, to check a summary of the structure of the data frame, we can also use this cribe, which is going to give you a summary of the statistics of the data frame. And in this case, what we see is that for each numeric column, only those columns are numeric continent is not here, for example, this is continent so you can see the type is object is a string, basically, all the numeric columns, we're going to have summary statistics for them. So for example, for population, how many elements we have, what's the mean, right? What's the average Romney, what's the standard deviation, the minimum, the maximum, and in between a couple of percentiles 25th 50th and 75th percentiles. So this is quick summary statistics. And we do this a lot. So keep in mind, his crime method is very popular. As you could see, in the in the info method, the columns have associated types, okay, so this is very important. They continent is an object that means that it's basically a string HDI is a float and surface area is an integer. And that's because NumPy, pandas is automatically with through NumPy, is automatically recognizing the correct type to assign to each one of the columns. This is similar to what we saw with a series in which the series contain natural datatype, a series was part of a given data type. So that's something you cannot change. And in this case, checking value counts, you can have a quick reference of the types of your series. So moving forward, how will we we will be selecting data from series Well, there are a couple of methods. And this might be a little bit confusing. So what I'm going to do is I'm going to skip and just going to give you a quick reference first, and then you can read if you want through the process we follow here, given a data frame, and this is just two quick rules, given a data frame, you're going to select by index using the lock attributes. So the lock attribute is will let you select individual rows. So for example, when I get Canada and that's the value of Canada, when I lock attribute will let you select similar to the series, the row by sequential position. So let's say We want to select the last row. In this case, it's the United States of America. So again, look lets you select a select rows by route by index, give me the row under this index, I log will let you select rows by sequential position, give me the last row, the first row, the second row, etc. And finally, without using lock without using a lock, just by saying the f up something, you are selecting that column, give me the entire give me a V and tire column population right here, the entire column population. So what you're seeing here, first, first of all, this is a quick reference dot dot Lok will give you an element by index, I look we'll give you an element by position, I wrote by position and just doing df, on some things gonna give you the element, the column sorry that you are passing. So it's like, both look on I look, look and I look work in a horizontal ladder, give me this, while bf art, whatever works in in a vertical montanus, which is getting you a given row. But something more interesting here is that all the results, these one and these one and these one, they're all series, what are being returned our series. So that's what we saw before. And the way it works is first, if we focus in this last example, we're going to see that it's pretty standard, just these series right here was is a one return I remember it has a type and everything. So that's, that's fine. If If we ask for a row, like in this case, we can get for example, here easily. There you go. The result is also series. But what you can see here is that this thing is kind of transposed in a way dot here was the volume of this year is population is here, and GDP is here and surface area is here HDI on continent on here you have volleys. So it's it's again, it's it's being transposed, right from vertical to horizontal, in our regular series manner on the index of this series is extracted as the name that the column hot. So in this case, the name right there is the value of the index that it had. So you can read more about it right here. But I just want you to remember these rules don't lock you select by index dot I lock you select by sequential position, the F at something you go by column, there are times when these might not apply. So or not want to apply, there will be some issues. So for example, if your rows if your index is numeric, you might have issues with these form or dot form, just respecting these three. For now, it's gonna get you any element you want to get either by row or column. So from what we've seen, the oldest slicing also works as expected. So we can get, for example easily, or we can get friends up too easily. So the upper limit is included. But again, it's built look and we select by indices from France to Italy, we can also do the second dimension similar to the way we worked with NumPy, we can do second dimension here. And we can get all the countries that are from France, or to Italy, including Italy, but only the population column or population and GDP. So here you can see the second dimension being applied at the concept of of multiple dimensions in selection being applied also to famous for ilok. It works in the same way that in that then multi index and the slicing. So we get for example, from one to three right in sequential positions. In this case, the upper limit is not included. So that's another difference from what we have. And we can also do multi dimensions we can say give me the countries from one to three and the column should be 0123 should be the third column, the fourth column, the column under index three which is HDI, so that also works as expected. And again, recommended, always use Look, I like to select rows and just use the naked data frame to select columns as we saw before. Now moving forward, conditional selection Boolean arrays will series, whatever you want to call it. This also works for data frames. And it's very important, it's a way to filter data, it's a way for us to consult the data when the when, when it so in this case, what we have is, we want to select all those countries, which the population is greater than 70. Okay, so all the countries that have more than 70 million habitants, similar to what we were we did with a series, but in this case, we want to do it with a data frame. So what you're going to see here is that we're going to construct a Boolean series as we did in our previous video, right? So every country with more than 70, false false, true false. And we're going to inject that result, that Boolean series in a dot lock selection, give me all the countries which match here than that the true value in it. And remember, just this is kind of mnemonics are a way to remember, the way pandas knows how to filter things is by matching this index, right from the resulting series. With these index of the resulting data frame. These are two different objects, they are completely different objects, but their index much. So here, Japan, March, Germany March, so here, Germany, on Japan, they are the same, and that's why that thing is working us expect that they This is just the first dimension, which is give me these rows, you can also on the second dimension, saying give me these column, or these columns, right. So that's steel, that's the awards us desire. So what about dropping stuff, you can say, whenever you have data from you can say give me just these pieces, or you can say drop the others, right, it's just pretty much the same. Dropping is very simple, you can drop by index, drop this value drop Canada altogether, period, or drop these indices can in Japan, or you can also drop columns, drop population, and HDI as columns. These ways also have a more advanced usage, which is with access similar to NumPy. I don't recommend them so much, but you can still use them and see them here. So all the operations we've seen, so far, they're all working. The most important part here is the broadcasting operation that we're going to do between series. So we're going to create a new series crisis. And I'm gonna show you what it looks like. So we have here crisis. And we're going to perform a broadcasting operation between between these, I'm going to show you what this thing looks like first, between that two, these data frame on the crisis. And the result will be that we will subtract, I don't know what's this number 1 million, subtract 1 million for each volume in here. And we're gonna subtract 0.3 HDI for each one of those. So what you can see here is again, this alignment between columns and indices, the GDP here is matched with these GDP and the HDI is much with these HDI. So there are two different objects, two independent objects, these series and these data frame here. But when we combine them with an operation like this, the the columns in this case are aligned GDP, and HDI and they work together. So you're gonna subtract these value in all these column, let me remove this, you can subtract these values in all this column for all these values, I'm going to subtract these value here in these column for all these values. That's the way it's going to work. So moving forward, what about modifying data frames? Now I wanna I want to show you something. And that's when we were dropping stuff before. We were not actually modifying the data frame. So here we did df dot drop Canada, but df still has Canada in it. And that's because similar to what happened with NumPy these operations are all immutable. They are not changing the underlying data frame. We are storing. We are storing we're creating new data frames that store the result of the given operation. So in this case, you have to drop Canada. The result is that the these new data frame but the underlying That iframe is not changed. That's because again, they are immutable operations. 99.9 operations in pandas, it are immutable, there are ways to change it, there are ways to make the changes permanent. But for now, I want you just to think that everything is immutable. Whenever you want to perform an operation, it's going to create a new series. If you want to keep track of this, you will just need to do something like df two equals that, or even df equals, you know, just to modify the current data frame. Again, there will be a way to not do that. But we're going to save in a sec. So modifying series more explicitly, that affrontare modifying data frame more explicitly, how can you create a new column? Well, very simple. Assign a column, I said, let's say in this, this column right here, it says similar to say, here, language. Oh, it's just read only. But if I say language equals, and I can just write whatever I want. In this case, what we've done is that the language, let me show you what Lynx had, in this case, was a tiny series, it didn't have elements for all the indices in the data frames, but that doesn't matter. pandas will match all the indices of our chill exist. And it will live like the rest. This na n is what we use for a blank. It's another number from NumPy. We're going to talk more about it when we start doing cleaning data. Data cleaning, sorry. So again, links France, Germany, Italy, you can see the volleys are all up there. What happens if you want to change a value the language series already exist, you want to change it or column or read exist, you want to change it. So in this case, we're going to say df language equals English. So we're going to change it all together, df now will be affected, and all the values of language will be English. How can you relate How can you realize when there is an operation that is changing the underlying data from the underlying series or than the line NumPy array, it's usually when you have an equal symbol, remember, NumPy, we saw something plus equals, in this case, whenever you have a plus and equals symbol is you're modifying the underlying data frame. So for example, check this out, the Rename function or method of a data frame will let you pass columns and indices to rename. So in this case, we want to change the United States to USA, the EU, United Kingdom to UK and Argentina to AR, Argentina doesn't exist in this data frame. But that doesn't cause a problem. And that's why we want to show you, the US, UK were modified correctly, and HDI was modified correctly. And a PC which doesn't exist, didn't cause any problems. Now, why am I showing you this because remember, these operations are immutable. If I check what's the state of the data frame, we see that the original data frame has not been changed HDI a steel HDI, it doesn't matter if we renamed it before, it's still the same data from the same thing for days, indices, all these operations are immutable. A few more examples of modifying data just for you to look at. And something that is very common for us is creating columns that are combinations of other columns. So again, this is read only, but you can you can imagine, that I could do is hear something like for example, GDP per capita, right? If I go here, and I do GDP per capita, GDP, p per capita, per capita, and here I say is equals to the GDP, this column divided by this column, right? So I do something like B, B three, actually, C three, C three, divided by b three, right. And then we would extend the values all the way along here. In pen this, we could do something very similar. We can do just any column, we can just perform operations, broadcasting operations between them, in this case, GDP by population. And we can assign that series which is a result right there. So it's a series we are going to assign that series to a new column. So GDP per capita Now, there you go is now a column of our data for. Again, all these broadcasting operations are extremely fast, they are backed by their NumPy array, and they result in a series. So very quick statistical information, a few methods, right to do summary statistics. We saw them with this crime method. But minimum maximums mean, median, all that works as expected. Something that I want you to note here, if possible, is that with pandas, we have, I'm going to change colors here, we're going to use red. With pandas, you have this concept of a data frame, right data frame that has multiple columns, multiple rows. And these operations are resulting operations are resulting in just one series. So in pandas, you have your data frame, and you have your series. And we could say we have individual numbers. And it's like always, the data frame is always resorting back to this, it's like some operations will just return a series. And the series can be used in a data frame, right. So in this case, these resulted in a series, but then we merely use the series to set the value of a column. Right. So that's why understanding series is so important. So there are a few more assignment exercises for you here. So you can check them out and complete them if it's going to make a little bit more sense once you're working with it. Finally, I want to give you a very quick introduction to reading the external data on plotting. And to do that, we're going to use a few methods that are very popular in there, maybe we can look them up very quickly here, we can say read CSV, use the read CSV function from pandas. So these function, read CSV. And as we have read CSV, we actually have a few others read sequel, read Excel, read XML, there are multiple adjacent or multiple ones, read HTML will be able to automatically parse an HTML page and read it. So a few functions like these like, what we're going to do with these read CSV, right here is the structure of it. A few of these functions will let us import data from an external source into our pain this workflow. So in this case, what we're going to read is these BTC market prize volumes, so it's right here, if I open the CSV, this is what it looks like. It's the date of the price taken a read and devalue the bread, the timestamp, and the value the timestamp of the value no decide the price of bitcoin 2017. Now it's close to $9,000, I think. But just note inside, but again, this is a CSV, and this is a CSV that we're going to be writing. To do that, again, we're going to use this method read CSV, the method will automatically parse the CSV, as expected. And there you go. And the process now will be for us to start tuning it to get to the right point. So I'm going to show you a few customization SP customizations, we can do with the receipt, read CSV function. So the first one, and sorry, let me tell you first, we have a ton of attributes here. So we have a ton of customization to do with read CSV, you will not remember all this, you will not remember everything out of the top of your head. So don't worry, you can always go back again to the documentation and just practice, it's going to come naturally. So the first thing, the first row of the CSV was considered to be the column names. So in this case, this fine lesson have a column name, let's say I add it, I'm going to do timestamp, timestamp price, you're going to save it, I'm going to rearrange the file and re re read it. There you go. So by default, pandas is assuming that the first line of the CSV is the rd columns. I'm going to go back into what it was. Right, and I'm gonna show you again, that's the assumption that pandas is doing. We're gonna Of course, of course, change that assumption, because in this case, our CSV file does not have column names. So we're going to just say Heather equals none. And this is when we start seeing the attributes that we're going to use from the read CSV function, read CSV. When I do hether equals none for us going to be known. That means don't infer don't read a header. Don't try to infer a header, a header from the CSV file. And the columns are zero and one. So now I'm going to change the columns. And I say, actually to be time something prize. And now what I'm going to do is show you the first rows. So you're saying here that I have these df dot head method that I'm doing. That's because this is a significantly large file. So we're going to say not not that long, but at least it doesn't fit in my screen. What's the shape of the day CSV or the data frame? It has 365 rows, and we have two columns. So we can do df the info, for example, to have a little bit more reference about we have 365 values, there are no no values, and price is actually float, that Tamsin is an object and we're going to fix that in a second. I'm sorry, that the F that head on the F dot tail, are the methods we used to get either the first and files or the end row sorry, or the last n rows, which are five rows, by default, you can change that and say, Show me the last three rows, for example, that's something you can do. And again, the types so the types is the timestamp in this case, the timestamp column was not properly parsed as a date, he was parsed as an object as a string, which we don't want. So we're going to use the function PD dot today time, something we're gonna explore in more detail in the reading in the cleaning data cleaning course. Part sorry, if it weren't tutorial, we're gonna use it today time function to turn these column D f, the timestamp into an actual date. And now we're going to say, the F that timestamp equals to this function resulting, and now everything looks as expected, there is one more change that we want to do, we want to set the index of the data frame to be the timestamp, because by doing so, we can quickly access price information led me see what was the price of bitcoin in 2000 1709 29. And I make a mistake here, I forgot to do the LLC. There you go. So we have the value of Bitcoin. On these particular date, forgot, look, remember that to get value from a particular row, you have to do dot lock. There we go. So we are getting Dodd's particular value. Because we've made a timestamp the index, we get artists value directly from the index. So what happens if you want to turn this thing into an automated script, for example, when I run this process, every day at 5am, whatever we can, we want to read the CSV, strip the columns, rename them turn into timestamps, etc. This is what we've done so far. Read the CSV without a header, create the columns, turn it into a daytime timestamp into a daytime and assign it to the index. And that's the result again, well, actually, the read CSV, oh, sorry, the read CSV method is so powerful that it will let us do all these actions in just one call of the read CSV method, we there are parameters that will let you customize the behavior to achieve the same results that we did with four lines of code right here. So in this case, we're gonna say, read this CSV, don't assign a header, that's something we do already or don't don't infer our header from the first line. These are the column names. So we don't need an extra line, we can just say these are the columns names. Oh, and by the way, the first column is going to be the index of the data frame. Oh, and also part of the date. They've the index, it's a date, so part of the date, and we have the same result as before. So now I'm going to pro try and same thing. There we go. So you can see it's work. So very quickly pan this plotting. Alright, so we're going to be doing here is I want to show you very quickly, I don't know what's this thing is as a vertical scrolling. I want to show you very quickly that you can create plots with Hannah's interest a breeze. It's so simple to create a block. So in this case, what we're going to be doing is, given a data frame, you can always invoke the plot method. And the plot method, what it's doing, it's using the map plot live library, something that you can check if you want in the docs. But for now, it's not necessary with these, we're going to be more than enough. What it's doing is just using, again, the regular plug library, as you can see dimopoulos Library, which is part of the standard PI Data stack. And again, for us to access using pandas is extremely simple, just df dot plot, you're done, you can set the plot as you want, we're gonna see more details of matplotlib. So don't worry too much about that later. So there is a more challenging example here that I can just run very quickly, you can inspect the process we follow to fix the data. But this is what we have, there we go. And what you can see right here is the difference between the Bitcoin and ether in this period of time right here, and they are both loaded in the same chart. And that's because this is the resulting data frame, we have Bitcoin on one side, and we have ether on the other side on we are plotting it right here, we're creating one plot with all of it. And we are noticing these empty value right here. So what we can do is we can go from December 1 up to January the first these period, so we can select that period, is in that lock. And we can just go ahead and plot it again. And this is what you see right here, the gap that we're seeing. So again, this was the introduction to pindus. We have a real life example of pandas following up. Also we have a little bit of data, more data cleaning on reading all the interesting files and sources of data for in getting more data into the pipeline, right. So the idea is going to be showing you how you can import data from Excel from SQL and then do the actual processing and analysis. Now it's time to talk about data cleaning, we have arrived to that point in our tutorial, in which we have pulled the data, I've shown you how to manipulate it with pandas, the beginning at least the introduction to data manipulation with pandas, and now it's time to properly fix it. For the sake of brevity, we are skipping a few parts of the process of data cleaning, especially you're going to find it in this first notebook that we talked about basics, conceptual, missing data with Python with NumPy. And we're going to miss a few other things. But I'm just going to mention them. pretty generic, pretty general form. And then you can of course dig deeper, you can check our courses if you want to know more about it. Usually when we talk about data cleaning, where it's in from a more conceptual level, we're going to talk about a four step process. The first step is usually finding missing data, which is the simplest problem to identify from a data set when something is missing. So you have car sales data. And there is a car that has no name right? Or there is a card has no price, right? So there is an number missing or there is a category missing and there's a string missing. And of course, each one of those is going to have a different meaning how to solve how to fix data set that is missing data, it can be very simple. If you can just for example, drop the record, if you can fill the value, right. So for example, the prices fill in these missing, you can fill it with the average value of the sales data or something like that. Or it can be very complicated if the volume is important if you can't move forward until you actually find that missing volume. And it can involve something like picking up the phone calling your ETL team asking what's going on that the data is missing. Or even if you're buying the data, you have to call the vendor, ask them why their ID if you've you're paying for that and there is data mentioning etc. So it can be a very political process. It depends what's your use case. But again, from a technical perspective, identifying missing data and fixing it is going to be extremely simple. Once you have fixed the missing values, then you start looking for the data is assuming the data is not clean yet in this process of data cleaning. The second step is when there are invalid values. So you have for example, column that is price and there is a string within it right here. We're expecting only numbers and there are strings in it. So then It's not going to be complicated to identify, it's not going to be too complicated to fix it. But again, we're increasing the complexity until a deeann of these data cleaning process, we're gonna reach problems that have to do with the domain of the day you're looking right. So for example, you have a column that is customer age, and there is a value that is 170. Right? So that is not an invalid value, it's a perfectly valid integer. The problem is that given the domain, right, but speaking about customer age, is highly unlikely that a customer is 170 years old, right? So in that case, the vowel is completely valid, there is no missing data, there is no invalid values, etc, is just about the domain. And this is when things get very complicated, because in this case, that example of age is something that resonates with all of us, we know about age of humans. But if you're working in a domain, if you're working as a data analyst, in a domain that you don't know much about, right, then you might not be able to judge if a value is invalid or not. If I am working in a biology lab, and I have something like white cells count per milliliter of blood, I don't know what's what it's a good value, or what's an invalid value, right. So it's, it's something you need to know the domain. So that's usually the the most complicated part of data cleaning, when you reach the limit of everything is valid, everything checks out. And now I need to make sure that these value is valid for these domain that we're working. So again, this is the spectrum that we're going to be revisiting today. So to get things started, the way pin this works with no values is is it has four functions, which actually there are synonyms, it's going to be it's going to be relatively simple, just trust me on that. There are a few things first, everything that pandas does in the process of missing values, is related to the way NumPy works. So again, we're skipping it, you can go to that notebook, check it out by yourself. But it's extremely simple. NumPy has these objects and n not a number to identify a missing value or no value in Python world to have the non value. But again, in pandas and NumPy, we're going to use na n none on there, or in this case, at the beginning, we have these two functions is no n is na, which are complete synonyms, we're going to find also is no and no we have it isn't a and they're also complete synonyms. So no n na for pan, this is the same. You can use the one you prefer. Sadly, I like is na because it's the way I learned it. I think for my students I usually recommend is no, because it feels more correct. And it feels more self explanatory. So you can use the one you prefer, if you can use is no, I think that's going to be better. If you get used to ease in a then you're going to be on my side, just do whatever you prefer. So again, it's no one's gonna say true or false, depending if the value is no or none, right? And of course not No, it's going to be or not na is going to be the opposite. So not na have not a number is false, and not an A of three is true. If you get to this first notebook, you're going to set all the false e values on the true fi values in detail in terms of Python, anything that is not empty or non etc is going to be considered to be truthy. So anything you pass here that again, is not an empty string or a no is going to be considered a true fi value. So it's no not no or is it a and none an A, they both work also with entire series or entire data frames, right? So it's not just for one of Valley you can pass an entire series. And the result back is going to be if the series is if the series what values in this series are either no or not no, depending what's the question you're asking either is null or not null. So in this case, we say which one is of the series are no, this is not, no this is not No, this is no so this is only true. And the opposite for the following method we are applying are actually function. And again the same thing works with not entire entire data frame. So something we do usually is if you look in to not know unknown, a few hacks that we usually apply are the count on actually this be the sum of all the no values or not no value. So we have this entire series, we can say how many not null values we have. And if we sum those, not no values. In this case, we're going to get a result out which is the entire the entire summary Have the nod no bounds we have asked, and the same thing is gonna happen if we say is no. So if I do here is no, some, we're gonna get how many novels we have? And it's pretty much the opposite of this question is no. And the way it works is in Python bullions are pretty much integers, they're ones and zeros. So every true Val is going to count as one and every four is going to count as zero. So if you ask for the sum of a Boolean series, you're going to get out the result of the number of truths that are available in that series right. So, in this case, we have to know values we ask how many knows value we have is know that some we get two out, you can use these tricks to filter the data with a series. So in this case, we can say give me all the values that are not known. Right? Just not know. Also, something interesting is that both for data frames are for series. The not not no is no isn't a not an A methods also, sorry, functions also work as methods. So in this case, we can say instead of PV dot know, we can say s.is, no load s, that is no. So now, it gets a little bit more, a little bit simpler. But if the final objective of these core as equals alzarri, s selecting only the boundaries are no no, was to drop the null values, then there is a simpler form, which is dropping, okay, so in this case, we can say s dot drop in a, and we're basically invoking the same thing that is happening here, we're missing we're just excluding sorry, all the missing values in the series or the data frame, because this also works for data frames. So what's the one, one important thing to remember here is that all these methods are immutable, we are not actually changing or modifying the original series, the underlying series is not being modified, there is a new series that is returned. So if I invoke s, again, this thing has is not modifying their series, you're creating a new series, and that's the one that hasn't, that doesn't have the missing values. Everything we've said also works for data frames. So right here, with these on a frame, we can say how many, right? The first thing usually is to start with an info method, right? So we have info, and we see that there are in total, four entries, four rows, we can also do a shape, if we need more information about the structure of our data frame. So there are four rows, four entries in our index, column A has only two no no values. So that means there are two values that are actually no no, sorry, no, there is column B that has three nought non null values. So that means that one value must be known, and that's for column B, again, so usually info gets you very close to understand the structure of your data frame and how many values there are missing. The same thing happens with some, we can just do df.is, null isn't a and then some, we're gonna get a quick reference of how many null values we have in that given data frame. Drop in a works in the same way, but there is a significant difference. The way drop in a works in a data frame by default is by dropping any row that has at least one, no value. So these row has no value dropped, these row has no value dropped, these row has two new values dropped, this is the only one that it's not being dropped, right. So it's very harsh in that respect, you can change that to make it to the column only, only keep the column that has no no values, and that's by switching the axis equals to one. And there is also a way to select a subset or thresholds. So only delete rows that have less than three valid values. For example, in that case, you're going to use something like the strategy of the drop in a you're gonna say, drop the columns, the rows, sorry, are the columns because it is also works for columns that have all the values and no, or drop. The This is the default behavior, drop all the rows that have any value in an NA or specify a threshold, which you mean by basically saying, I need this amount of valid values in order to keep the rope it's the way it works. Now, which ones to drop is which wants to keep based on the fresco. So once you have identified it No values, it's extremely simple to clean them to sorry, fix them. So the first method we're going to see is fill in a within a particular value, we're going to say from this series, I want you to fill the blanks or fill the missing values with or fill the anaise. fill them with numbers zero in this case. So these two are numbers zero, or, of course, you can use any statistical method you want. In this case, we can use the main. Remember, this is not altering the series, the original series is still the same, we're not changing it, it's creating new series because all these methods are immutable. The following method is or this the following way This method works is by passing a method which is for field or backward fields, these are the possibilities. And basically the way it works is it's overflowing, all the values top down, at least in Fairfield, right starting here, it's dropping this value here, dropping this volley here. And dropping now three here, as this thing is a nun, it gets replaced. So this thing is three now, which gets throw up here. And now this thing is three again. So that's what we have right there. And of course, backward fields works in the other way, starts with four and moves, it moves it here and then moves here, etc. You have to be careful when using these. Because if you have no no values at the beginning or the end, then you're gonna end up again, with no values because there is nothing to fifth forward, right, this is the first volley you have India. And all we've seen also works for Donna friend. So both boggler fail for field or both in terms of rows for feeling, right, so we have, we have these, these data sets. So we do for field row base is going to be one to here too. And then five. So that's going to be for field x is one, if you use for field x zero, then it's a vertical filling, right? It's going to go here, one 130 30. So that's for the column, that is y here, one 130 30. So it's either for filling in, in, sorry, this direction for failing, or it's going to be in this direction, depending on the axes that you are passing. And actually, let me we're going to put the correct forms with axes equals zero, it's going to be columns, it's going to be visit direction with axes equals one, it's going to be row based. So it's this direction, right? So we had a no volley here, that got fail in this way. Okay, moving forward, we what else we have, we have here, checking for values. And we've pretty much seen this already, you can use the is know, the sum method to get how many values you have. And there is also any an old, which will give you very quick. These are usually called Boolean tests, you can say ask if there are any values are valid, or all the values are valid is just to build more complicated queries. So so far, so good. So the process we said was at the beginning, we were fixing missing data, missing values, there is nothing in there. We have read a data frame, where's our data frame right here? We have read our data frame from CSV from a database, and the value is missing. No, there is a hole in it. So we have quickly identified it with isn't a or is no, we were able to drop the ones we didn't want to keep dropping a or we were able to fill the volume we wanted to fill fill a name that was simple isn't a drop in a fill in a what happens when you're cleaning data that actually has a value, so there is no nothing missing. But those warnings are invalid. So for example, here, the sex column is a categorical column that only accepts an on f. d on question mark, those are invalid they are, it's very simple to see an invalid value here because it's completely out of the scope. The same thing as we have, for example, question mark in the age column where we have we have a string in the age column, it's very simple to identify that, how we're going to clean those. Let's start with sex first, because it's simpler in this case, the first check we can do is with either unique or with volley counts, I'm going to use value counts. We've seen this method before. It's a quick summary of all the unique values you have. And in this case, volley counts also gives you a total count for those values. How can you fix them? Well, there is a replace method which is extremely intuitive. You can just replace in this case, we're changing all of these two F's and The End two M's, and it can work in multiple columns. For those volleys, that again, we said were more complicated to fix, like, in this case, we know age, in this case, is 290. And we know because we know the domain, that 290 as an invalid age for a human. So we will need usually in those cases, we're going to need more complicated fixing, and it will involve more programming, that's the reality, you have to be better coding. In this case, we know that these volley is invalid, because it's probably an extra zero. So all these values, you're pulling a CSV with ages, and there are a total of 180 290 32 320, for example, invalid values out of 100, right in the 100 places. And that's because there were typos when they were creating the ages. So how are you going to fix that? Well, in this case, it involves a little bit more programming, we're dividing everything by 10. So also, something that may be useful is dealing with duplicates. And we need to first define what's going to be a duplicate value. So this is, this is usually a little bit more political, if you want, you have to define what's going to be a duplicate. In this case, we have a series that contains ambassadors, and each, their master is the index, the country of the ambassador is going to be the value, right? This is usually the important part. The rating here says the word conducting a party, and we want to invite one Ambassador per country, we don't want to repeat ambassadors, ambassadors. So in this case, what's going to happen is that these two in our humanize at least, we can click clearly and quickly see that these two belong to the same country. And these three belong to the same country. But here again, we have to define which ones are the duplicate, if you want, and which ones are not duplicate. So for example, maybe we can say the first one is duplicate, or we can say the last one is duplicate. So this is the first one not duplicate, or actually can say this, the last one is one, and when I bite, it's not to duplicate. So we're going to have political rules if you want for each one of those. So let's see the duplicated method and the way it works by default. By default, duplicated method is going to return true for duplicate for all the it's it, I'm going to invert it, it's going to not treat it as a duplicate as the first instance that it says. So the method is actually walking top down right now saying, Do I have friends? No, I don't have friends. I'm going to keep it here. Because it's the first time I see friends. Do I have the UK? No, I don't have the UK, it's just gonna keep it here. Then it sees the UK again realizes the UK is already there, too. It's already present. So this one is going to be considered a duplicate. Italy is here, it's fine. The first occurrence of Germany, it's fine wrightstown, Germany, but then it says Germany two more times. And it realizes that Germany was there. So those are now duplicates, right. So the way it works by default, we can change that and change it to last to the last element is not considered to be duplicate, and the other two are considered to be duplicate. And the same thing here. Kim, here is the one consider duplicate. So it's either top down or bottom up depending the way the parameter you're passing, it's either keep default or keep last, or you can be a little bit more harsh on say everything duplicate it is actually to be needs to be considered duplicate. So these two are duplicates, and these three are all duplicates, as you can see, right there. Similar to the duplicated method, which pretty much tells you which values are duplicated, it's it helps you identify them, you also have the drop duplicates, and in this case, what this method is going to do is basically the same thing as before, but dropping all the values are checked for true, right if the method is if the value is missing, it's gonna just drop it. And the same rules apply default, last and false. For subsets in this case, we have Ace, we have multiple, we have multiple players in the data frame. But what happens is that these player Colby is present three times for humanize we see Kobe three times. What is going to happen here is that the The way we're going to think about duplicates is by understanding the correct subset that we should check. In this case, Coby plain as sn SG is duplicated two times but COBie, playing us in SF could be considered a different player if you want, because maybe it's a different season, or it's a different, a different position they played. So in that case, we need to pass What's this subset that we are going to consider duplicate, only check for the column name, or check for the column name on or not check for the column name, which is the default is going to check the entire data frame. And when that happens, then these two are considered to be duplicate. So these one is a duplicate with this rule, if we put keep last, sorry, keep false, both are going to be considered duplicate. So this second occurrence is the duplicate one. And the last one is a completely different row, because the the value in position is different. That's the way it works here. Moving forward with more cleaning of values, we're going to talk about string handling. And this is a very neat feature of panelists, that special types of columns will have special attributes. So given the column type, so df info, which is an object, which is a string, right, in pandas, that all the strings columns are going to have these special attribute which is str, all the daytime columns, something we're not going to cover, but you need to know, all the daytime columns have a.dt, Math attribute, all the categorical columns don't have a.ca t cat attributes. And those attributes, str DT cart, they have a special methods associated the domain of that column. All the methods associated with string are of course, we're string handling, or the methods associated with DT r for data handling. So in this case, we're going to review all not all very good subset of the string methods we can apply. And something interesting is that all these methods have a very good have a lot of relevance. And they're related to the ones in pure Python. So if you have a pure Python string, there's a split method. There is a contains method or I don't know if there was a contain actual, it's actually, I think it's the in operator, but there is a strip, and there is a replace, right, so most of the methods under the str attribute in pandas have, have an analogy in the standard library of string handling with Python. So starting at the beginning, this data we have, I'm going to delete this this data we have, what we are going to do is split the values right by an underscore. So in this case, that's what we have, we have split all the volleys with that underscore, and we're going to use the special attribute is expand, expand sorry, equals true. And what it's going to do, it's going to create a data frame out of that. So we create a data frame with 70 columns. And this is what we have now. So we can keep applying methods. So for example, contains or content contains, regular or contains with regular expressions rights for you to see the power of it, we can just strip replace, and we can do even regular expressions with replacing so we could fix something like this question mark in a string, we could fix it with regular expressions if you know how to handle them. And finally, something that is going to be very helpful when you're doing data cleaning, is looking at the data from a visualization perspective. data cleaning has a ton to do with statistical understanding of your data to when a volume is considered an outlier. For example, it might be invalid, and you want to claim it. So but that's a lot more about statistics. And this case, I want to show you very quickly, the mottled leave library, I've been promising for some some time now, the mapa lib library. So far, we've accessed it directly from pandas, from pandas, or we're doing a data frame dot plot. It's these library mapper lib is the one backing all those methods and we're going to see how to use it directly. Now. The model live library has two important API's we're gonna call him one is the one that I don't prefer, which is the global API, but it's the most common one. It's the one you're gonna find around the global API. And the second one is the object oriented API. So it's around here. And usually there are there are ways it's just two different ways of doing the same thing. Okay. The global API is an API that it's in part inspired in MATLAB. It's been around for a long time on sadly Most of the answers you find in Stack Overflow tutorials and books will be using these global API. The way the word the one I prefer the most. And I'm gonna explain you why in a second. It's going to be the object oriented API. But I want to show you both. So you have a reference. If you follow me in this feeling of preferring the object oriented API, you will always have to translate global to Opie. Why is it considered a global API? Well, we have imported matplotlib.pi plot as PLT. So we haven't imported the whole module, the whole Python module, depending how much you know about Python programming is going to make sense or not. We have important the whole module. And now what we're doing is we're invoking PLT dot figure. And finally, and then we're going to do a title. And then finally, we're planning two things. We're plotting x, our plotting x squared and minus x squared. And why is this global because we're invoking functions that are at module level. And there is an object, the final plot, that it's being modified by these very generalistic and global courts, right. So by by doing these call right here, I'm modifying the final result of the plot. Let me show you a more complicated example. So you see the problems with the global API. If you look at these line, if you could delete everything, let's actually delete everything. What is this line doing which plot is affecting, you do not know, there is no object oriented way of saying in this second plot the plot on the right or the figure on the right, or actually the sub plot on the right, I want you to plot this thing, you're just saying it to the entire module. And depending the order that you set it, is where it's going to land, that particular figure where it's going to land in which plot, it's going to lend. Again, it's a global API. So we start saying, I'm going to create a figure, trust me from So from now on, I'm going to start drawing on it, there's going to be the title. And hey, by the way, it's going to have one row, it's going to have two columns. And I'm gonna start drawing in the first plot these one right here, these one right here on the left, okay. So now I have kind of activated if you want that plot, it's active. So now I'm going to start drawing on it. So every action that happens after this line is going to be affecting these blocks, these blocks, right. So then I plot x and x square, I plot this vertical line, I put a legend, I set labels, etc. And at some point, I just stop and say, Hey, now I want to switch the plot, I want to now start plotting. Sorry, I want to start plotting here in this second one, because I have just changed that the first line these one. Oh, sorry, the way it works is by saying the first row, second column, but second plot. So now I want to start plotting in here, every successive line will affect that line. And again, you can see that understanding a code, given the order that the order in the sequence of lines is very hard. If you have to debug a report that has a plot that takes 100 lines, then you have to keep in your brain, what's happening top down, a different approach is going to be the object oriented approach, in which we're creating a figure. And we're creating axes. So in this case, we have in this case, we have right here, one entire figure in red. And we have in here, purple, we have two axes. So these axes one, and this is access to so we have two axes. We're going to create those using an object oriented approach. And we're going to keep references to them. So we're going to say later, to these blocks to these artists, sorry, I want to plot something. And that will be very explicit, it's going to be an object oriented way. So the first thing is creating the figure on DCE. The axis in this case, we have just one axis, that's it, but you can have more and then you say in this axis, I want to plug this thing in this axis, I want to pull up that thing, etc. When you have multiple axes, so I could show you. I'm going to go back again to that in a second. But In this case in which we have four axes, right, so we create one figure. And it has four axes, we do it with this subplots, method saying and rows and columns. Now we say to the axes number one, I want to put this thing to axis number two, I don't want to put that thing, right. So it's 1234. And now it's a lot more explicit, it's not depending on the order, I could change this order, that doesn't matter. They're that the results are gonna be the same oxes number four has yellow, regardless of the position that we're following. So the map will live. And now that we have clear out the differences in both API's, maple leaf has this very simple plot function, or method, depending on sugar enter global, that we'll plot something you specify. In this case, we're passing all the values in x and all the values in y. And in this case, we're passing a given line style, this can change with these type of syntax, you're saying, I'm plotting this thing in X, I'm blowing this thing in y second parameter and why. And I want you to use a straight line, it's a straight line, yes, with this marker, the dot and in green. So this is if you are very familiar with it. If you're very familiar with my bullet you can use to send links in other games, you can just say line style market marker, sorry, color specific keyword arguments for each one of those. So do we only have line plots in APA live? No, of course not. We have a huge variety of plots. And by the way, there is another one here, if you want to see more events are grids, you can create these grids and put different things in it. And again, not only land plots, one good example is a nice scatterplot. So basically, we're plotting X and Y correlation. And there is also our value, our color map, right. So given the volume, there is going to be a change in color. So these kind of lets you plot three to four dimensions of your data, the volume x, the volume, y, the size of the bubble, and the color of the bubble. So where you're pretty much encoding four dimensions in just one figure, right. So in this case, we're just using two different scatter plots, there's more information here, we can also block histograms, that we've very quickly seen that with pandas with pandas is, is very simple with just plot type histogram, current histogram hist, actually, you can look it up in our previous lessons. So just go back into the index in the video. And the histogram is extremely simple just takes the valleys you're plotting and how many bends you want, or some more advanced arguments here, like the alpha level, etc. But it's simple. And similar to the histogram, you can also create kernel density estimator diagrams, which is very similar to distance to simulate if you want a continuous distribution. You can combine these plots if you want, in this case, we are creating the plots were plotting a histogram. And they were plotting the lines and they were plotting our changing limits. But that's pretty much it. And you can also create bar plots, right? So in this case, we have PLT dot bar, or here we have two bars are stacked, right? That's the different way to look at it. And finally, check in outliers. You can always plot histograms or box plots, right? So box plots are also a nice feature to have in here. So this was all with data cleaning, we're gonna keep moving forward this tutorial, I want to mention one more thing here. And it's there are notes here for kind of a task that you can follow with data cleaning, which where we are identifying where indentifying missing values in given positions with is known as an A. And right here, we're looking into more detail about some statistical properties of the data, in case we need to clean it. Okay, so this is little bit more events. And it's it's related to the concept of cleaning data given the domain. So the statistical analysis can tell you that this value is an outlier. For this distribution, the value might be valid. So for example, a human being is 90 years old. That's, that's valid, that's a valid age. But if you're analyzing data about high school students, and a human that it's not a year soul, it's going to be completely invalid or it's going to be an outlier in that distribution. And you can treat it as such You valid valid and clean it out, remove it, for example. So that's, that's deal a little bit more with the whole statistical analysis you can follow here, it's a little bit more advanced for the scenario. So let's move forward with the rest of the videos. Now it's time to get into more advanced features of pandas to import external data. So we've seen already in our real life example, the way we can import data from CSV files, and from SQL databases, right, we had actually those two lessons, the objective of these part of the tutorial is to show you how you can improve or get into more advanced use cases of importing data. So we're going to start for example, with csvs, and text files. And again, you've seen it already. But here, we're gonna give it an extra twist. So I'm going to show you more advanced features. And for special use cases, txt files, CSV files, is, conceptually speaking, a CSV file is a text file, it's just human readable text, right? That it's encoding information. The idea for CSV file is that it's tabular. Right? So it's a plain text file that contains tabular data in it, and it's separated. csv stands for comma separated, but it can be separated can be anything, we can see more examples later. But basically, the idea is that it's a text file that it's tabular into in a tabular format. So though, both CSV files and text files will be read with the same method. So to get things started, I want to show you the basic way we import will read data from, from from external sources using Python without even starting yet, with pandas. So you don't need to know this, it's usually it's usually productive if you want for data scientists or data analysts to understand a little bit more how fire reading and writing works in computers, because there are multiple, multiple concepts align, here, they evolved, operating systems processes your language, right, it's not same thing to read a file with our or with Python or with another language. So there are multiple concepts here. And even though pandas in this case can make it simple, very simple to read and write data, you can get a little bit of a more advanced use case, if you know the internals of again, both the operating system processes on your language. So this the way we read data with a reader file, sorry, using pure Python, we use a function open. And in this case, we're using a context manager, just a security feature, again, related to to the advanced usage of reading and writing files. But it creates a file pointer, right. And with a file pointer, you can then use the very simple API x point post. But they but that pointer, which is something like red line, red lines, read a number of bytes or characters, or you can just even trade FP as an iterator, just do a four line in FP. But basically, we're going to do something like this, we'll start reading data from top to bottom, just a month to, I don't know, we hit I've given in this case, we're doing it just for a couple of lines. What else we can, it gets very difficult when you're reading text files to process them, because it's usually hard to parse the structure of the file. So it's not the same thing to have a funnel that is separated by comma separated by colons separated by pipes, spaces, etc. So you're gonna see that once you want to get a little bit more, I don't know a little bit more with an advanced usage, right, or a little bit more fancy your calculations and and the way you parse the data, it's gonna, it's gonna get harder. So that's why we're going to use pandas, or I'm going to show you in a second, this is the module that is part of, of Python. So this is the file that we're going to be reading. It's the XM review file, and I'm going to open it. And even though it doesn't look like a CSV, it isn't either CSV. The difference is that here the separator is the greater sign, it's not the comma, it's a greater sign. That's going to be what marks the elimination between different fields in our CSV file. So we're gonna use the CSV module. And the way right here to parse the data using that module is by passing a special delegator, right? So that's gonna be the type of work you might need to do when you're parsing the data. It's not the same thing to have that limiter dates a greater sign. It's not the same thing to have numbers for example, that are enclosed in quotes. All those things right will change the way you work on all days is going to be abstracted away by the pandas module. So to get things started, again, with pandas, at least, pandas has multiple read underscore something methods that will work for different sources, right. So we saw already have read sequel we've seen read CSV, there's also a read HTML to directly parse information from a table, it's literally you can just you pass a website's going to read information from a table, or read Jason read more advanced formats like pocket, or Stata, etc. And, again, each file format will usually have a correspondence in pandas, it's, I've never had the chance to rewrite my own stuff. To be honest, the same thing is going to happen for something like Excel, which might need external modules, it's not directly provided by pandas, but by installing those modules, you can easily incorporate Excel files in your day to day work. So the read CSV file methods already has a ton of parameters. So this day, the main characteristic of all these rate something methods, given the amount of possibilities you're going to have with these files, there exist a ton of different ways to customize the method invocation. Alright, so again, CSV files, we saw, there are multiple things happen. csv is a passage that have a header don't have a header, different delimiters different and closing of strings or numbers, multiple things, blank lines, etc, multiple things are going to happen. And that's all you're able to customize all that with the read CSV method. So this is the reference of all the attributes you can pass to it, usually something that I do, and I do this very often, and I use pandas a lot, and I still do something like read CSV, and I get the documentation right here, to look into the, the parameters that I think I need to pass to my particular use case. So keep an eye always in the docs, because it's impossible to remember all the parameters in the CSV. So in this case, what we're gonna do is something very interesting is we're gonna parse a CSV file, but it's not located in this computer, it's not locally available in the computer. The CSV file is these one right here, which actually is the source, if I get the raw version is this thing. So this is CSV file, what I could do here is download the file, right, so just do File, Save, get the CSV file on my computer uploaded here, right, so just copy and paste here, drag and drop it here. But actually pain this has this nice characteristic that it will read a CSV that it's either locally as we did with BTC market price, or you can also do it remotely, it's automatically going to download the content of those files. And it's going to provide, it's going to save it in memory for further usage. So there's a very neat feature. And again, this is the the CSV file that we are using. And again, the same thing, if it's a local file, it works in the same way. So a few features you've seen already, in this case, we can do Heather known, if you don't want to treat the first row as a header. Or what about missing values, we can treat some of these values like a question mark, or like an exclamation mark, or dash etc. us not a number, not a value, right, so it's a missing value. And now any of these values we have passed, will be transformed into another number for easier and easier process cleaning, we can pass names, which is going to be basically the column names for each one. And we can also specify column types, as you can see, right there. So now the types are going to be float. And object. We've done this already in one of our lessons, we are parsing the time and there you go. So putting all together, we get to these advanced forms of reading csvs where we're passing column names were passing types, were asking to read dates, were passing no values, Heather's etc. So this is a pretty common thing we are doing. So what about XM review, if we try parsing this thing, we get this very ugly format. In this case, they put the parameter to specify the what we used to call delimiter in CSV is now set from separator so the separator, it's going to be the greatest sign and that just works as it needs. So, a few more examples you can check on here the most important part is following right, the documentation to find those particular use cases that you are having so for example, some Like skip blank lines, or whenever there are like empty rows at the beginning, right. So if you have empty rows at the beginning is something you can also say skip rows. So you don't need to parse those out, it's not going to break, etc. So that is all part of the read CSV file. And to finalize these part, at least csvs, I'm going to tell you something that applies to pretty much every other data format. As you have a read something method, there's going to be a to something method, it's basically the process of writing. So you can do read CSV, or you can do to CSV. So these CSV that we imported from the external source and the remote source, we can just do to CSV and it's going to store it locally. Alright, and there are multiple options also to pass the CSV delimiter, or actually the separator, if you want to include a header if you want include an index, etc. They're pretty much the same as the other one. But the idea is that for every read something method, there's gonna exist a to something method that it's basically the process of writing. So let's move forward with a few more data formats. And interesting, we're gonna get to read directly HTML pages in just a couple of minutes. And now it's time to read data from databases. We have already done that in our real example with Panis part of the tutorial. But I want to show you a little bit more details details for you understand how data is being processed in case, this is a common scenario for me importing data from databases. So the libraries you will need first thing, depending on what database engine, you're using Postgres, MySQL, Oracle, etc, you will need to install different libraries. But the API's, once you have installed, those libraries are going to be the same. There's actually p Ep from Python that actually defines the interface for databases, libraries, unpin, this can work with pretty much any any database of these SQL common database that comply with that interface. In this example, we're going to use SQL lite because the database right here, there's nothing, no server to connect, etc, is extremely simple to get started. And the example we're going to use, or the danavas example we're going to use is actually different one from our previous video is reading in the previous one, we were using circular, in this case, we're going to be using chinuch, which is smaller both in structure and in size. So it's going to be a little bit simpler. So to get things going here, the same thing that we did with our previous part, that was how to read data from files, I show you how to actually read data using Python. So forget about pandas for a second, I told you, if we go back again, to the beginning of time, there was no pain this, this was the way we were writing, finance, open FP, FP, the red lines, etc. So I now want to show you what predates to pin this, what was the default way to read data before paying this, which is with the regular again, interface from Python. So the way it works is we're gonna import SQL lite three, we're gonna create a connection. And now with this connection, we have these common interface that again, it's common for pretty much any other database that you're used to. And the default behavior is we're going to create a cursor. And we're going to execute queries using that cursor. In this case, we're going to execute a regular Select star from employees limit, Fox will want to have five, five records out of the table employees, once you have executed a query, it's like they're waiting, you can do a fetch all to get all the results of that query. And here are all these results. As you are noticing this is the result is a list of tables. So it's not extremely useful. Now, if you combine it with pain, this you can just create a data frame out of that info. And we're close. It's not perfect, but we're close. So let me show you now before we were gonna close it Kurt Dickerson on the connection. Let me show you now how we work with pandas. With pandas we have as we have a read CSV method, we also have a read, see as read SQL method, and in this case, what this method is going to receive is the first parameter is going to be the query that we're passing and the second parameter is going to be the connection. That's the object the connection object to actually issue the connection by panelists. So it gets a simple as writing the query. And now everything has been imported into a data frame, including column names and all that if you want to get a little bit fancier, you can either specify the index column, there's going to be use, of course as a index, and also what types to parse for a specific column. So now we have pretty much all the work down. So we're going from something very manual as processing things with a coarser etc, which might also be as low to using pain this to do Actually imported data from the database. There is actually a caveat here that I'm going to tell you is kind of a very deep detail of the way pandas works, and is that the read SQL method is actually a shell for two other methods, read SQL query and read SQL table. Alright, so right SQL table on read SQL query, when you're using read SQL, it's actually kind of forward in the work to either query or table, or an SQL query is the default behavior, what we've done so far, so in this case, it's just going to issue a query and the connection is going to read it for you. In contrast rate SQL table is can I read an entire table, you just pass a name, and it's going to automatically give you all the information for it. So in this case, all the column names, etc. So it's a lot simpler to read an entire table, the only thing to keep in mind is that to use this method, you need to install these libraries, SQL alchemy, and the connections generated from it. So in this case, we create an engine on we create a connection objects, and now we can pass an actual auction object sorry for pandas to do it. So again, it's pretty much the same, if you find yourself doing Red Star from this table, Red Star from that table, it's a lot easier just to write SQL table, and that's going to do it just advance. As we saw that read CSV files hard to CSV, sorry, read CSV method had a to CSV method, the same thing happens with read SQL, there is a read SQL and the results are to SQL, what's what it's going to let you do is get the from the database and write it down into a database table directly. So it's going to also receive the connection, right? So to SQL, it's gonna receive what he will name of these data frame, what table name is going to be, and a connection object. Now something to keep in mind is that to SQL has an important parameter, which is what happens if the table already exists, that in the default way, it's going to fail, just going to throw an error when you are trying to save data to a table. And this makes sense, because as data analysts were usually reading data and processing it, we're not so much writing it. So we want to meet make sure that it's not by mistake. But if you do actually want to write data, you can just change this parameter if exists something like replace or append. Usually, we're writing to intermediate intermediary table tables, again, you can choose either to replace the whole concept of the table, be careful here, or to append, write, just write it a dn of the current table. So that's just for to see. So this was the way to read data from databases, of course, we're not touching on anything like SQL and all that, that it's a lot more advanced, it's just for you, if you already know SQL, if you're already working with databases, you can pretty much copy and paste what we're doing here. And you're gonna, you're gonna get your data import into Python. So let's move forward to read some HTML files. And now very quickly, I'm going to show you how to read tables or data frames directly from HTML web pages. To be honest, this is a simple method is going to be just read HTML, but it depends a lot on the structure of the web page. So if it's not well structured, or the tables are not correctly created, you're going to have issues and you will have to do a ton of data cleaning. In my experience, whenever I try to parse a table from a well structure site like Wikipedia, or some stats site, it usually works very well. And it's a very quick way of hacking. You know, whenever you have questions, you know, like, I don't know, I need to know the GDP of countries. Instead of looking for a GDP data set, you can just go to Wikipedia page, there is usually a table there, you can directly parse it and you are done. So again, it's it's a relatively simple way to get some data for quick hacking and exploration. The way it's going to work is we have these HTML creative. It's just for testing purposes. To get started, usually, of course, you will try to read something from a live website. So you're going to pass the URL to the read HTML method. And the read HTML method will download the content of the page and parse it. Let's suppose we have the the content already the HTML, and this is what it looks like. This is a exactly the same HTML we have on top, I'm just displaying it here in a book. And what we're gonna do is we're gonna invoke the method, read the HTML. And the read HTML method is going to parse the entire HTML and look for multiple tables, not just one site will potentially have multiple tables, even if you don't see them. The is a common way to structure things in HTML to use tables. That's why it's going to pause multiple tables. In this case, we stored them all in a DFS, multiple player like multiple data frames. And we see that there is only one. So in that case, we're just going to get the first data frame. And it has correctly parsed what we had before just working in the same way. The same is going to happen with for example, things for headings and all that if the table doesn't have a header, it's gonna automatically right understand.in that case. So that's pretty much as we know it already. In this case, what you're going to see is what I told you before about data cleaning process that these table does not have a header like the previous one that has a T head to head attribute, in this case, a header is just another row. So that's why read HTML is going to have issues and you have to provide a little bit extra information. So let's see another more realistic example. And we're going to parse data directly from a website, let me tell you here, just just for educational purposes, you always need to understand if you have if the data is public, so you can actually parse it. Again, for Wikipedia, at least what I do, the content is created comments, so you can get a hand on it. There. What we want to show you here is that a very complicated table that has multiple headers, etc. So that's why we're using this example. So we're gonna get the URL, and we're gonna directly do NBA tables. Equals read HTML, the only table in this page is this one, the large one. So that works. And now we're gonna get NBA is going to be that and we see that the all the players in this case have been parsed. What about something else, let's actually open this page right here to Wikipedia, for the Simpsons. And here, we will probably find several tables. See, we have one right here, this one. So I'm going to import it. We have 27 tables, again, you don't see it. You don't see them, sorry, but they are there. And the most important one is the one we care is these one right here. So the problem you're gonna have with this table is that each using both columns, pans and rows pans. So in this case, this column here is pans for one to three columns. And these row here stands for 123, at least three rows. So that column spans results in these very ugly data frame, and you will need a little bit of extra cleaning. So that probably you're going to find with HTML tables that usually there are things that are not well formatted for machines that are formatted for humans. So for example, in this case, we have this header repeated, when you parse this data, you're going to find that every 20 rows, there is going to be header row, and you will have to clinic every for in this case, to enter rows, you will need to drop it you will do something like df the drop, let's see, actually, if we can see it haven't tried this, but let's just do it like that head, and you're going to find 25 Records now. So here, record 22, we find that, Heather, so what we're going to do is you will need to do something like df the drop df dot drop, range 22 starting in 22, up to the F the shape, zero, right, these many rows plus one plus one and every 20 rows, I don't know this is going to work, just run it. Hope didn't it didn't even work. It didn't compile. Oh, this is NBA actually. There you go. So maybe it works, you can check it. But what I'm going to say is, again, there is some cleaning to do because HTML pages are optimized for humans, not for machines. So usually, it's going to take a little bit more time. The good news is that there is usually a service associated that you can consult. So for example, there is a Wikipedia API that you can use instead of a page. But again, sometimes just easier to pull the data directly from Wikipedia. So that's it. You can also write data to CSV or of course or HTML. That's pretty much the standard. As we've said, this is up all we had for the read data portion. And we're gonna move forward now with a few other methods, especially what we call data wrangling. We're going to do a little bit of grouping and keep moving forward with our tutorial. We have decided kind of last minute to our final source of external data that it's going to be an Excel file. It's just a common Excel files, you know it, because we imagine that you might come from an Excel backgrounds, you can just export the data you have in your Excel files, Excel spreadsheets, and load them into Jupyter Notebook and start working with them with him this so you can try things out and kind of draw the pearls in between Excel and what you do with pandas and Python. So the first thing is, an Excel file is not a text file. So if you try getting the content of it, it's not a text file, it's not so simple to parse it. So that's why it's gonna require external tools that they already installed in notebooks AI, there might be a student's holding goal up, but it depends on your computer, how you're going to install it. So just keep in mind that there might be issues when importing data from Excel, if they if there is low compatibility between the library you're using another spreadsheet version you're using. But without those without getting into those details, there is read Excel method, which pretty much takes care of everything for you has different parameters, like the finding the the sheet that you're reading from, of course, the path, etc. So we're going to start reading these file, which is products file that has three sheets, products, descriptions, and merchants. And it's actually something we use in an Excel file to sorry, in our data analysis, from Excel to pandas course, to show how to merge data and all that. And from this file, what we're gonna do is just read Excel. And what you're gonna see is that it reads the first sheet of the Excel file, I mean, a data frame is just corresponds to one sheet only, right? And the first one is product. So that's what we are writing. There are different behaviors for it, you can change the way you parse, Heather's etc, you notoriety defining and specific index, that's pretty much everything we have seen. So far, it's selecting specific shifts is simple, just pause the sheet name, and you can share the rate story either products, merchants, whatever is available in the current Excel file. There is another format or a new specific class that it's a little bit more advanced. But it's the Excel file class. So it's not, as we were doing here, right, Excel directly is going to read thought Excel file into a data frame, but you're going to instantiate this Excel file class, with the parameter being the file name. And now these files gonna have just a reference of everything you have. In this case, we can do for example, sheet names, it's going to tell you how product descriptions merchants, there's a little bit more explanatory data analysis. So let's say you can't use Excel to actually see the contents of the Excel file, this is going to be helpful, you're going to first parse the Excel file, get the sheet names, and a little bit more of an understanding of it. And now we can say from these files we have previously parsed right here or instantiated, we can parse the product, the product sheet, and that's going to get you that that frame. And the same thing is going to happen with all the parameters weekend pass, they are the same as read Excel. Finally, you can see that the results are to excel file. And it works pretty much the same way as to CSV, and decide if you pass an index or not. And also you can define if you're going to pass a sheet name or not, are just going to be the default one. So as you can see, getting your data into a from an Excel file into a CSV, data frames array is extremely simple. There are more customizations to do, let's say all your file is shifted array, either rows or columns, you can change that with Star row or column that's going to work, too. So that's pretty much the only thing we need. If your writing process is a little bit more complicated. Like for example, you want to write specific sheets in our multi sheets. Excel file, you can use what we call an Excel right and it's also part of fantasy, you instantiate the rider, and then you can start the ride process saying which shades you want to ride with each one of those, that friend. So again, reading and writing data from on to Excel files is relatively simple. It all depends on the libraries are installed. It depends on on what libraries you have in your current environment, if it's windows or if it's a Linux slash slash mark, the documentation of PD dot read Excel might have more details for the given platform that you have. So let's see if it names per document, if it's not here, it's gonna be in the pandas documentation, but there might be a requirement For each one of the platforms, that pan This is supported. So just check it out, check for your own for your own platform if you're in Windows, Mac Linux, how to get those libraries installed. So in case you're just getting started with Python, and you might come from another language, the objective of this quick section is to show you Python. Ideally, in under 10 minutes, I think it's going to take a little bit more. But there's a very, very, very quick reference of Python, again, just the high level features of the language, how to use it, how to code functions, how to import modules, variables, data types, collections, etc. You can just scroll through this notebook, if you want to take less time, I will be providing an explanation on top of all the topics, but there's a very good reference of the entire language. So to get things started, Python is an old language period. It has card, it has caught more attention in the past five to 10 years. But it's a very old language. It's even older than Java. It's up here in 1990s. And it was created by this person good by Guido van Rossum. And it's an important actor in our ecosystem he is used to be I think he still the one deciding discussions etc, when it comes to defining features of the language, etc. Python is a high level interpreted dynamic language. And this means a tone actually, if we read these entire sentence, interpreted high level, general purpose, this is basically high level programming language, it's object oriented. And it also includes functional attributes or functional features like functions as first class objects, etc. And it also, of course, it supports imperative programming. And it has a wide variety of applications, you can do web development with Python, you can do scripting, it's a lot use for system development for configuring machines in general. And of course, you can also do data science, it has multiple applications has a couple of interesting features like indentation, for defining blocks, etc, that make it and very good language to get started with programming. So if Python is your first language, you should be comfortable with it. It's a very good idea for me, it wasn't my first language. And I hope it was, it wasn't. But I, I have taught people programming with Python as their first language. Seriously, it's always been very good for them, because Python doesn't have weird things like my have in JavaScript or Java. So it's a very concise language and consistent language to be honest. So let's get started very quickly. First of all, when you're going to install Python, your own computer or you can use notebooks AI or Google call up. But if you're installing in your own computer, you might see that you can install either Python two, or Python three, or actually, if you're reading tutorials online, etc, you might see Python two and Python three, the reality is that Python two was deprecated in 2020, so the you cannot you should not use it anymore. There are still ways to install Python two, but it was deprecated. So you shouldn't use Python two, you should stick with Python three, which is the evolution of the language. So ton of fixes from Python to the bay where, where things happen in the language and used to confuse beginners. So that's no longer a problem. Python three, again, is what you should use, you will read in multiple tutorials, etc. What they are using Python two, you should try using Python three, and sometimes the code will break, but the changes to fix it are not very hard. So to get things started here, I will be drawing the problem of this and with regular syntaxes. For example, this is the way you will define a function in for example, JavaScript. And it's also very similar to something like C or Java based languages, the function keyword, curly braces, etc. So I will be drawing a parlors and with these sort of languages. So to get things out of the way to defined function in Python is in this way. And the main characteristic of this language is that the way we're going to define blocks is by Using different indentation levels. So this is a valid function in Python def is the key where we use the name of the function the parameters it receives. And the way to define that the body of the function is by just indenting. Everything one level to the right. Usually, this is just for spaces. Another example is an if else statement. So if this thing happens, do that if else do something else, right? This is JavaScript. In Python, again, it's defined by indentation. If this thing happens, we indent one level to the right, do this else do something else, if there was another if statement here, if I don't know, language, ends with something like I don't know, three, then do something else. Print pi, three, for example. So we're indenting everything to the right, every time we start a new block, whenever the block finishes is just when you go back again, print this as first block, right, that's the way it's going to work it by indenting. Our blocks, this is very good, because first, we don't have debates of where we should place the curly braces. And also, because it makes it a lot more readable, it's a lot easier to read these code because there is obligated obligatory indentation to even make the code work to. So you can see that's that's just how it works. How we're going to make comments in Python, just by using the number pad symbol, there we go. And the way to define variables is just by specifying the name. So it Python is a language that you don't need to declare variables, you just declare and define everything and just one pass, you know, you find a variable, as it goes. Python is dynamically dynamically typed. But it's also strongly typed. And these might kind of cause confusions. But basically, you can assign variables to any value you want. And you will see that collections etc, are heterogeneous in terms of types, etc. It is a very dynamic language. Talking about types, I'm going to show you the most important types that we have in Python, especially we have numbers, of course, integers, we don't have so many like, like, you might find that other languages, like different precision cetera, we have integers, there is also the concept of Long's that has changed with Python two. To be honest, on Python three, to be honest, we use just integers, that's the way we work. It's a, it's a smart enough type to save storage when needed. So that's, that's good. And it will also have floats, right, which is the regular float type for floating point arithmetic in other languages. And of course, it suffers if you want from strange behavior from float floating point arithmetic, like in this case, you can prevent that by using the decimal module, which, as you can see, doesn't suffer from from this issue. So numbers, we have integers floats, and we also have decimals, strings are just a type str, and they are defined literal, right, as in this in the st, you can see right here, you can just type the string as it goes. There is a difference between there was a difference already in Python two, between Unicode and strings, etc. In Python three, that has all been fixed. So we Python three, this is all Unicode. And there is the concept or the difference in terms of the concept of something being the type. The Unicode code points as it's this string, and the underlying encoding will turn it into binary. So in Python three still have we have a few ways to differentiate between whether it's a binary string or whether it's a text based string. For you shouldn't worry about it, I just want you to know, if you're writing a Python tutorial, for example, you might find a difference between Unicode strings and regular strings, which is, is no longer something that we should be worrying about. If you have a string that it's too long and it expands multiple lines, you can always write it using three quotes can be double quotes or single or single quotes. So just to create multi line strings is extremely simple. Boolean there are two Boolean type do Boolean objects are unique, right? It's kind of a single tone which is the true or false objects. For example, They are of type Bo. There is also the concept of No, in Python, which is none, we don't have no, we have none, but it serves pretty much the same purpose. In Python, everything is an object. So even this strange, strange objects, like none will have an associated class, if you want, everything in Python is an object. So all these types of you have seen. So for example, we have this string, which is H of a string. The type is str, you can use the int, str float bool types, right, but it's the result of the type also as function. So in order to cast in this case, a string into in order to cast a string into an integer, you will use it you will do it using the end function, which is the same thing that you get with these, for example, so this is the same as this, as you can see, what we have to show. So functions again, death is the key word we use, we don't use function, we use death I, you can use define, as a mnemonic, the name of the function parameters are optional, and finally have the return keyword, you should always include a return you usually 99% of the time, the function should return something. Because that's going to be the result assigned once we invoke the function just this is pretty regular. If your function doesn't return anything explicitly, if that means if you haven't written down a return statement anywhere in your function, the function will still return something so that the fact that you haven't included a return statement explicitly doesn't mean that the function is not returning anything implicitly, actually, it is returning something, it's returning none, right. So by default, if you don't include a return, Python will do this. Just for you to know a function always returns something as specified parameters and passing parameters is pretty standard. Python has some advanced features with parameters like for example, variable length arguments, we can pass as many arguments, we want to make it very dynamic keyword arguments, named arguments, etc. So all their ethic operators, you know, already, the shin modulus, in this case, were doing a power its operation, all this is pretty standard. And the same thing happens with all our Boolean operators greater than greater or equals then etc, there are type checking. So this is when we have the strongly typed feature, even though Python is dynamically typed. It is the types are enforced. In this case, you cannot compare a two with this doesn't make any sense. And Python is going to complain about that. So this is an example of an error in Python. The exception type error was raised on the same thing with bolens and not on or operators. As we saw before control flow is defined by the indentation so every new block is defined with an indentation level. Python includes if else and also l F, which is very convenient. And this is an example If this happens, Elif, Elif, etc. Python does not have a switch statement. For example, loops, how are you going to loop through something in Python loops on lists, or collections in general, are very interconnected. Because in reality, when you're looping the Python, you're not doing a regular in Python, we don't have something like in, in Java, you're gonna have something like int i equals zero. What else I it's been decades. And I this is I haven't coding in Java. So I, I don't know, minus 10, less than 10 less than 10. And here we do I put last There you go. So we don't, we don't have these in Python. We have a way to mimic it. But we in Python, we always eat iterate over a collection. So what we're going to do is we're going to create a range elements, and we're going to iterate over it. So the way it works is very close to one other language is going to be a for each. Alright, so in this case, we have all these elements and we're going to do for name in names, that's it. And at any moment, the name is going to be associated with an element in the list. while loops are part of the language, they are usually discouraged in favor of for loops. If something can be coded with a for loop, it should be coded with a for loop and not a while loop. Because as you might know, already, these my trigger or these might result in an infinite loop if you're not checking the conditions correctly. So the collections we have in Python, are the fundamental ones, the primitive ones, the most important ones are first the list Python is we do a heavy usage of lists. And it's just a heterogeneous data structure. So you can put anything in it. And actually, all these collections are heterogeneous, you can mix volumes as you want. And in this case, we have three elements that we have added one string, one integer, one string, and one Boolean. And let me say something here. Even though pythons, Python supports mixed types in the collections, it doesn't mean that you should do it. To be honest, we should, you should usually avoid mixing types in collections, because that means we don't, we don't know what we're putting in it, right. So it's, we should be consistent. So it's possible, revisit your code, if you have too many different types in it. I'm checking the length length function accessing elements is by by zero indexed, and we use square brackets. So in this case, give me the first element given the second element. And also we can index starting from the from behind from the end. So in this case, minus one, minus two, minus three. So in this case, minus one minus two, again, give you different elements, you can check the operations associated with all these elements. Very quickly, a list is L dot append, we're going to append the new element. So the list now has that element at the end. And we can check if that element is part of the list in this case is true in this case is false. topples are similar to lists, they are also sequences, but the main difference is that they are immutable, there is no way to add new elements to a tupple, or remove elements from a tupple once it has been created. So in this case, we have created a list with three elements. Now tupple, sorry, with three elements, we can access it, we can check if something is in it in the same way that we did with a list. But in this case, with a tupple. Again, you cannot modify it tupple never changes, you can't add elements to it. Another important data structure is a dictionary. In Python, a dictionary is a key value, right and mapping, it's similar to an object in JavaScript or hash table in in, in Java, it's a key value mapping type. And in this case, we are going to associate values to names. So you can see this, the way I like to explain it is if you create a topo list, right? So let's say we're going to create a list, out of all these elements, give me one second, we're going to create a list. There we go, we're gonna copy these elements. And we're gonna associate that to our list. There you go. So these are a list, we could very well store the information about our customers in a list, right? That works. I mean, I can get it done. The problem is that whenever I need to access information about this list, we're going to say, for example, I don't know I want to give me the email for this customer, I have to remember the position that the email is located so in this case is going to be position number one, if these information grows, instead of having 1234 values or four pieces of information for our user, we have 100. Right, then it's gonna be very hard to access those individual volleys. So that's why we create dictionaries, dictionaries are collections of values. The important part is on the right, the important part is the value. But they are instead of just indexed by the precision, we give them arbitrary names, we tell them very explicit names. This is the name, this is the email. This is the age. And this is if they are subscribed or not. So once we create these dictionary, we can access those values by the name, give me the email of these user or is the age present of the user is the last name present of the user in the user in the user dictionary. So again, it's a way to store information associating later In order to make it simpler for us later, let me delete this. And I move four sets sets are very common data structure, he is when you're learning about a collections and, and and yeah, the instructions in general, it's not so common in too many languages. I mean, it's not very popular in Python, we use it often because it has a very interesting feature, first of all, and it's something that I forgot to tell you about dictionaries, both sets and dictionaries, are what we call unordered. data structures, you never know, the order of the elements. In Python, with recent versions, there have been changes, which make Python dictionaries ordered. But for now, I'm going to say you shouldn't rely on it, you should think your dictionaries as they are completely unordered data structures, and the same thing for sets, sets are, it's a bag that contains elements, you know, it's a big bag, you keep throwing elements inside of the set, there is no orphan in it. And what's gonna happen with it, you're gonna odd elements, for example, to the set, or you're going to remove elements to the set. And there is one important thing that makes this set so useful, and it's the membership operation, I'm gonna write it down here, membership, ship operation, there you go. So you can access these notebooks later. So in the membership operation, the the, the process of checking if something now, nine in s, the process s of checking this is extremely fast, it will be called oh one. And this is because as you might have seen here, when I created this set, I included a couple of repeated elements, 333, write 11179, the resulting set doesn't have those repeated elements, these are two features of the set, the set will only contain unique values. And by the way, it's implemented behind the scenes will make dot these unique values are extremely simple to check whenever you pass these membership operation is extremely simple, or sorry, is extremely performant. It's very fast, different from for example, a list. So keep it in mind sets are very, very useful when you're checking for members. So again, as I told you before, we're going to iterate over collections with the for loop. So in this case is if we have a list, it's going to be for element in list. There you go. If you have a user dictionary, use a dictionary, sorry, in this case user, we're going to the default iteration is by key, we're going to get for name email age subscribed, and we have to extract the value out of the of the dictionary, we could also do for value in user dot values. Oh, there you go. Or you can iterate over both key and value with items. Key. And value. There you go. So each iteration in in in Python is very readable to put it in a way. And again, remember, we're always using the for loop that assumes that you're iterating over a collection, we don't have the for Ei equals zero equals zero, I equals zero, i less than 10. i plus plus we don't have that right in Python, we can simulate it with for i in range. Five, for example. Print. We've got simulated with the range function, which generates pretty much those elements. Something that you might have heard about Python is that it has a huge library of built in modules, right that you can just import and just gonna work. There are so many things already coded in Python, that it makes it very simple for you to create something on top. Do you want an a library for I don't know security cryptography Math, numeric processing NumPy, right? machine learning web development, creating games through is pi game, do you want to create a graphical user interface, whatever you want to do, there is usually a library that has already been coded and will make your job easier. On top of that, the bill team is down there library, right, which is already included with Python, it's not third party. In this case, it's already created by the Python core team. It's a huge library, so many modules. And the way it works is by importing this module, so this is the way we work with packages and modules, there are differences between modules and packages, third party ability, and this is a little bit more advanced. But again, this gives that random number generator, it's already built in. And you can check the docs right here. exceptions, whenever you do something that doesn't work. So in this case, we say, if the age is greater than 21, but age is a string, it's an it's not an integer, this is going to fail. We can catch exceptions before they happen, that's going to be with a try and accept lock. Right. In that case, if this fails, if anything here fails, these blocks going to be kicked in. And you can catch the exception without the program fail failing. And you can be more explicit about the error aspect. So again, this is just an introduction. It might be useful if you're coming from another language, especially to keep this notebook as a reference. We're going to be using Python a lot, of course, and it's a great language if you want to do scripting, work development, of course processing with data, data analysis, etc, visualizations, machine learning, Python is just great. So I hope this tiny tiny reuse lesson helps you port your knowledge from other languages into Python. And that's it.