Hello friends welcome back so with this video of introductory video of data brick certified associate developer and predominantly for the spark development developer certificate uh so this course whatever is we are bringing up you can see or you can follow this playlist so uh and this is intended for the data brick certified associate developer if you are if you want to get certified in data bricks for this certification that is uh spark developer associate certificate certification and so once you uh once you follow this course in a sequence and without any break and you will be able to understand the spark Concept in in detail and you'll be able to take that certification course anytime after the completion and uh so I would I would recommend you to please watch till the end of the end of the course so that you will get a maximum out of it and this is as it is in the YouTube platform so we are presenting from this channel so which is completely absolutely free and I hope this will be useful for many of the people and I would recommend you to please share subscribe with your share and subscriber and also uh share with your friends so who will be uh who who would like or who would be interested to learn regarding databricks so with that uh like why why I'm bringing this course is a spark definitely in a big data space or the data processing or data engineering space or data science or machine learning space as spark is uh predominantly used right and uh so whatever you see the Hadoop cluster is mostly built on the concept of mapreduce and Spark so it is very important to understand spark as a data engineer and if you are in a data space that is data science and machine learning space and also like uh in the today's words what we are dealing with is completed data driven approach and Spark has spark is the standard Big Data cluster processing framework so there is no competitor I would say in this space for Big Data clustering for especially for the processing with respect to spark because spark was spark came up as an open source right and so it has been utilized by data breaks for the for their processing engine and in this course we will learn in detail about the spark architecture where core apis and uh so and and the different topics of the purchase Park which we will see in a in a moment so uh before starting if you are new to this channel we would recommend you to please subscribe and also press Bell button for instant notifications so with that uh let's get started [Music] so before we start up the video uh the most important points to note is uh make sure you are you use a headphone for the best experience and adjust volume accordingly because each devices might be different you are using mobile or tablet or desktop please choose the volume accordingly so that you will not since the video will be long so you will not get irritated or will not get exhausted and also please make sure you pause and take a notes wherever it is needed so because so that is how you learn uh like whenever you you take a notes you actually kind of uh make sure you know things or acknowledge things so we would recommend you to take a notes as and when need data during this course and also we recommend you to full watch the full video right and don't skip any part of the video because you might skip the important piece and follow the sequence of the course as designed in the in this playlist and also practice as you go through the course so it is not just the we're seeing the video or following the course I would recommend you to please go through the data breaks and uh in the data breaks you we would recommend you to run the course run the core commands or the notebooks that we'll be providing as part of this course and any suggestions uh questions please comment them in the comment section if you need a practice note notebooks so then please comment your name in the comment section and also send send a request mail to uh YT dot the data Channel at the gmail.com as mentioned here so with this uh let's get started so whom the this course is designed for the audience for this course right so if you are a data engineer or if you're a developer maybe you are from amateur from to the professional so this course is for you because any data engineer or would definitely uh helpful and the person who is actually who needs to understand the Apache spark and its capabilities so definitely that person also uh for that those people also this course would be helpful uh and uh who is and this is important who is the people who is looking for data breaks certification that is a data break certified associate developer for Apache spark so if you follow this course we would uh uh we would guarantee or we would uh climb that uh basically if you follow this course in a sequence and you will practice along with us so we would uh it would definitely help for your databricks associate developer certification and any data Enthusiast who is in the data domain and want to explore more about the data and its related technology so this course definitely will bring more perspectives for them as well okay and what are the different prerequisites so when we say prerequisite what you what is expected from you before you start the course so we are not expecting much things so few of the understanding of the basic understanding of SQL and uh so that will be helpful and basic knowledge of Scala or any other programming language would be helpful because if you learn any programming if you have this uh some programming background so it will help because the course is mostly designed using pi spark and the Scala programming language so it would help but it is not mandatory and also the basic uh knowledge about the data and so what is data and what is the some database Concepts so that would also uh kind of a help and definitely no clear spark knowledge is required here the reason is we are explaining both theoretical and practical aspects so we are not expecting any uh programming or any kind of a spark background so we go in a step by step from the theoretical to uh practical and with from Basics to advanced level in a sequel in in a sequence so if you just follow the course so that is that should be enough for that Source content as you can see at a high level so the course is designed for these topics or these sections but as you see in the playlist in this playlist in YouTube channel so there might be multiple videos but uh so all those videos will fall under these any of this category but at a high level so it's very important to understand the Apache spark architecture how it is designed for distributed processing distributed execution and distributed uh data storage that means uh it is it is always important to know how you store the data in a distributed manner so that will depend that will definitely impact so how you uh process that in a distributed manner so both are inter interconnected how do we store the data so that will definitely impact how do you process the data right and coming to the data Transformations as a different data Transformations so as part of the Apaches worker so as a developer it is important to understand these data transformation techniques or data Transformations so we will in detail we will explain different kinds of data transformation with all the Practical knowledges and uh with all the Practical uh knowledge and uh executions of each and every data frame kinds of data frames right uh Transformations and we also kind of uh uh like this this will uh we will also cover the certification uh exam certification details and tips so how what are the details uh exam certification details how do you kind of attempt the certifications what are the different uh like questions you can expect in the exams what is the duration and all of the details and we will have those uh we will have a section for that so we would recommend you to please watch the video till the end so to get the maximum out of this course so you have seen at a high level what are the different uh things or whatever whatever the different topics at a high level but in detail if you want to see what this course is about so these are the topics that you can expect as part of this course so definitely the architecture part of it which is a combination of more of a theory and some practical and you see the spark execution and cluster notes and execution of hierarchy of spark and there are different data frame operations as you can see from here so data data frame schema and data type data data frame API and SQL function data frame rules to filter data data Frame data sorting and how do you handle nulls in data frame and also uh data frame creation from files and also Scala selecting columns from different uh different different data from a data frame and manipulating the column of a data frame and saving the result to external Source like Amazon S3 or Azure data Lake storage Etc and how do you store how do you create a user defined functions and what is the use of it and what is the spark SQL functions and data frames as a groups and uh like how you also understand how to use a databricks Community Edition so this is important for you to uh there's no installation for data breaks so definitely it will be uh since it is it is a cloud hosted you will understand how to use the community version in your web browser so if you can just create a community version you can use the browser and execute your commands as and when it is needed right so this is a this is what at a high level these are the topics you will be uh you can expect to learn from this course in detail and so so thanks for watching definitely and so as we mentioned so we would recommend you to please follow the entire playlist so this is just an introductory video but we are coming up with a full set of detailed videos and uh so if you have any questions or if you need any notes so definitely please reach out to us in this uh in the email ID mentioned here and in the description you can find uh different social media uh connection or links to connect with us so thanks for watching