Transcript for:
Drone Programming with Python

hey everyone welcome to my channel this is the drone programming with the python course here we are going to learn the basics of a drone including the components and how the drone flies then we will look at the installations and the basic movements along with image capture from there we will create exciting projects including surveillance drone face tracking and line following drone if you are a beginner don't worry this course is for you and we will focus on the implementation part so that you can get the most practical experience out of this and even though we will be using a specific drone for this course eighty percent of the material covered in this course can be used with any drone if you would like to view more of such content then don't forget to like and subscribe so let's get started the term drone means a continuous low humming sound now this might not be the best way to represent these unmanned aerial vehicles but still it's the most commonly used term drones have gained a lot of popularity over the past few years as the technology has advanced and the motors have become cheaper this has allowed the drones to be used in much more commercial sectors such as photography filmmaking surveillance and even delivery systems in spite this drones have a long way ahead so learning how to program a drone to perform different tasks is becoming an increasingly popular skill drones come in different shapes and sizes a popular term you might have heard for a drone is quadcopter this refers to a drone with four propellers or motors similarly a hexacopter has six propellers and octocopter has eight propellers so let's have a look at the main components of a drone a drone will consist of frame motors propellers electric motor controller power distribution board a flight controller battery receiver or bluetooth a camera video transmitter and some sensors the frame or the body is what holds everything together most good drones use carbon fiber as the frame material since it is quite lightweight and strong nowadays a lot of diy drones use wood and 3d printed frames as well motors can be considered as the engine of the system that creates the lift and allows the drone to fly there are two main types of motors used in a drone brushed and brushless brushed motors are simple and cheap whereas brushless motors are expensive but have a better weight to power ratio propellers are what generate the lift force when the motors start rotating there will be two types of propellers for any given drone one that rotates clockwise and the other that rotates counterclockwise this is because if all the propellers are rotating in the same direction the drone will start rotating as well but since we have half of them rotating in one direction and the other half rotating in the other direction they can cancel each other out and the drone does not rotate propellers can have multiple pairs of plates the more blades you have the more thrust it will generate but the more inefficient they get the electric motor controller or the esc is what controls the speed of the motors they convert the dc signal from the battery to ac signal for the motor you can have separate esc for each motor or a single board with multiple esc's on it power distribution boards allow battery to power all the components including the motors and the flight controller the flight controller is the brain of the system and it decides what should be the speed of the motors based on the input received from the sensors and the receiver the battery will power all the components of the drone this has to be light and efficient at the same time to allow maximum flight time the receiver or the bluetooth allows transmission of the signals between the remote and the drone most of the times we will also have a camera attached to the drone along with either an sd card storage or antennas for wireless video transmission now to transmit video we need video transmitter this device is responsible for the transmission of video through the antenna to your cell phone or the fpv remote there are multiple sensors attached to a drone for example the pressure sensor measures the altitude or the distance between the ground and the drone the gps locates the position of the drone whereas an imu measures the acceleration and the angles at which the drone is the movement of a drone seems quite complicated as it has four or more motors to enable all movements but in reality it's quite simple and intuitive the drone can move with four degrees of freedom which means it can translate in three directions and rotate in one as mentioned earlier two propellers of a quadcopter rotate in clockwise direction and the other two rotate in the counterclockwise direction this generates a zero angular momentum that allows the drone to stay stationary rather than rotating in one direction let's have a look at the translation movements first first we will run all the motors at the same speed that will generate the lift greater than the weight of the drone this will cause the drone to move up this is the phase translational movement if we want to go down we will decrease the speed of all motors so that the lift is less than the weight of the drone if we want to hover in the air we will change the speed of the motors so that the lift is equal to the weight of the drone this will allow the drone to hover above the ground now coming to the second translation if we want to move left we will reduce the speed of the left motors and increase the speed of the right motors to move towards the right we will decrease the speed of the right motors and increase the speed of the left motors similarly we can translate in the third direction which is moving forwards and backwards by decreasing the speed of the front motors and increasing the speed of the back motors the drone will move forward for moving backwards we can decrease the speed of the back motors and increase the speed of the front motors since two motors are rotating in clockwise and the other two are rotating in counter-clockwise direction we can use this knowledge to rotate our drone to rotate the drone clockwise we will decrease the speed of the motors moving clockwise and increase the speed of the motors moving counterclockwise similarly if we want to rotate counterclockwise we will decrease the speed of the counterclockwise motors and increase the speed of the clockwise motors now that we understand how the drone flies let's have a look at the drone we will be using for this course the drone that we will be using for this course is the teledrone now this drone is recommended to be used with this course but that does not mean that you cannot apply the same techniques to other drones in fact 80 of this course is applicable to any drone and only 20 is specific to the teletron the teledrone is produced by the company ryze and it contains dji and intel technologies for flight control it is a compact drone with a camera attached to it that can shoot up to 720p video at 30 frames per second it is quite easy and safe to use indoors because it has multiple safety features now this video is not sponsored by any of the companies including rise dji or intel and this is my honest opinion that telo is one of the best drones to get started with if you want to learn programming of drones the best part about this drone is that it is programmable which means using an sdk we can control the movement of the drone and apply computer vision techniques without any external wiring that's right everything will be done over the wi-fi and you do not need even a router for it in fact they have added two antennas which will allow stable video transmission the total flight time for this drone is about 13 minutes and the flight distance is about 100 meters with this top speed at 8 meters per second when getting the drone you have multiple options the first one is regular pack and the second one is jumbo pack there are also two types of tello drones one is a regular and the other one is edu which is the educational this course is compatible with both types of drones if you get the regular pack it will come with a single battery extra propellers and the propeller guards the charging would be done connecting a usb cable directly to the drone on the other hand if you get the telo jumbo pack then you will get everything as before plus two extra batteries and a three slot battery charging hub the jumbo pack is much better for programming because with three batteries you can do a lot more testing and while you are using one battery you can charge the other one so that you do not have any delays so first we are going to go to app store or play store and we are going to search for telu now once we find it we are going to install and open it up now the first thing we will do here is to connect the telo device so we need to turn it on you will see the lights and once they stop blinking with all these different colors it means it's ready and there you go so now it's blinking and it is only one color so it means it's ready so we can go to our app and we can click on connect hello and then we are going to go and connect to our wi-fi so here we will connect to the telo once that is done we will go back now we can see the live stream from our telo device so here you can see the live stream and now it means that it is running so once you have the tele working you will go to settings and you will go to more and here you have to make sure that your firmware is up to date so here you will see firmware version and you have to click on update so it needs to download first so you will have to connect your wi-fi again you will have to remove the wi-fi of the telo and you will connect to your home wi-fi and you will download the firmware and then once it's downloaded you will go back and upload it to the telo device by connecting it again so here you can see it's already the latest firmware if it's not the latest it will allow you to download and once it's downloaded you can upload it to the teledrone once everything is set we are going to try to run it to see everything works fine so we will click on the takeoff button and we will slide to take off and now you can see the drone is flying so we can go forward we can go backwards we can rotate we can rotate back and then we can go up and then we can go down [Music] so i think i'm about to hit my sofa so i better get back now one thing you have to note here is that even though this is okay let me uh okay we have two options here tap to land and then hand landing so most of the times when you're programming you will be working with tap to land okay so i want to mention here one thing which is very important even though this is an indoor drone and it is very suitable for indoor flying you have to be precocious when you are running this drone so it has a lot of safety features but still you have to take care especially of your surrounding you have to be mindful of where others are and you have to be mindful of where you are so that nobody gets hurt so once this is done we can go to the installations on our computer and we can start programming this row to program this drone we are going to use the python programming language now to do this we are going to first install python and then we are going to use an ide which is an integrated development environment to actually write our code so we will first go to python.org and over here we will go to the download section and you can download the latest version or my recommendation is to download the 3.7.6 so that is the most stable version that i have found to work well so you can go to all releases and over here you can find 3.7.6 so you can download this and then you can go to the pycharm download section here you can see you have for windows you have for mac you have for linux and here you can download the community version which is free and open source now if you want to understand the ide a little bit more you can think of it like a notepad but with more functionality so whenever you are writing the code it will tell you where you are making mistakes it will help you debug the code and overall it has much more functionality than a regular notepad so this is why we need the ide so one of the most famous ides for python is pycharm so that's why we will be using this now the thing is that i have already installed both of these things on my computer so i'm not going to do that again but i have the footage from one of my previous courses which was about opencv python course so it it follows exactly the same procedure so i will show the footage of that right now and you can follow the same steps the only difference will be that when you are naming your project in pycharm name it something like tello course or telo programming course or something like that so let's start with the installations first we will go to python.org and we will go to the downloads section here we are not going to download the latest version instead we are going to download 3.7.6 as this works well with opencv so we will click on download and then we will head on to the windows version now if you have mac you can download the mac version as well once the download is complete we will install make sure to add python 3.7 to the path so once the installation is done we will close the dialog and head on to pycharm so we will head on to its download section and we will download the community version as it is free once the download is complete we will run the installation file here we will associate the dot pi files to pycharm and we will also add launchers directory to the path next we will restart the computer and run pycharm so now we are going to create our new project here before we start the project we will look into the project interpreter now this has to be 3.7 so if it's automatically detected then we are good to go so let's write down opencv python and hit create so now this is the pycharm environment over here we have the area where we will write the code and on the left hand side we have the project files and the project folders so once the installations are complete we are going to create our project now i have named it stello course so it will come with this main dot pi file we can write in this file or we are going to create a new one in which we will store all our code so we can delete this one we can right click and we can go to new and then python file and here we can say basic basic movements now to actually program the telo we need its library so to install that we will go to file we will go to settings and then we will go to the project then the project interpreter and then we will add here we are going to write dgi tello and here you will see that we have the dgi tello pi library so we are going to click on that and we will hit install now while this is installing let's have a look at who actually created this library so let's go to the link and this is the guy his name is damia funitis he's the one who created this dji tello wrapper which is the wrapper for the original dji tello sdk so what this does is it makes it a little more easier to communicate with the telo and to command all of these different things and to perform different tasks so there is a documentation for it as well this is an example so you have to import the library and then you have to say connect you have you can say move left move right move forward and you can give it some values so a big thanks to the guy who actually built it and all the contributors who have contributed to this gift hub repo and have made it easy for us to use the telo library and the license for this is mit so this basically allows you to use it commercially you can modify you can distribute but you have to make sure that you include this in the distribution so this is the license if you are distributing you have to add this to the license as well okay so that being said let's go back and now you can see it's already installed now another library we will be needing later on is the opencv library so we can install it now so it's called opencv python we can click on that and we can hit install and this will install both of these and now you can see we have all these new libraries that we did not have before so we can hit ok and now we can go on to our coding and first of all we are going to import our library so here we will write from dgi from the gi telepi import tello and then we will be needing uh the time as well so we will write from time import sleep so we want to add some delays between each command so that is why we need this so to connect to our telo is very simple all we have to do is we have to create a telo object so you can call it anything i like to call it me so me is equals to tello dot tello t-e-l-l-o and this will create our telo object and now we can simply write me dot connect and this is how simple it is you have to just write me dot connect and it will take care of all the ip address and all the communication part for you all you have to do is you have to run this and to make sure that it is running properly we will print out the battery so how much battery do we have on our drone so we can write here me dot gets battery underscore battery so there you go and if you are wondering where am i getting all these uh functions from so you can click on control and then you can left click on your mouse and this will take you to all the functions that you have available so you have get height get temperature get altitude you have all these different things that you can use okay so that being said actually let me show you more uh you have flip left you have move back you have to give the x value the x value is basically in centimeters so this is 20 centimeters to 500 centimeters you have more forward move right but we are not going to use these because this is very simple and you can use this in one of the third-party apps as well that move move forward for 10 centimeters not 10 actually let's say half a meter then move half a meter to the left then rotate 90 degrees all this stuff you can do with third-party libraries what we will be doing is we will be using the rc command and i will show you later on what that does but before we go there let's run this and see what happens so we will right click this and we will hit run but before we do that we have to make sure that the telo is connected to your wi-fi so we will turn on the telo drone and then we will go to wi-fi and connect so i have turned on the drone right now and then we will go to the wi-fi settings and here we should have the telodrone pop-up there you go so now it's connected i have pressed automatically connect that's why it connected automatically so once that is done we can go back and we can right click and run basic movements so if we run that we get the response first of all we connected so we got the response okay and then we asked for the battery so we got the response as 61. so right now the battery of my drone is 61 okay that being said now we need to see how we can control our drone so as i mentioned before you can simply write for example go forward so if you wanted to go forward you can write me dot go is it go forward or is it just forward move forward so it is me dot move forward so again all these functions you can read about them so it tells you telo fly forward with distance x centimeters so you have to give a distance here so you can right here go forward for example 30 centimeters and then you can write 50 centimeters and so on but we don't want to do that what we want to do is we want to control the speed so as we discussed in the introduction the drone can translate in three directions and it can rotate in one direction so we want to control that so we will write here me dot send underscore rc control and inside that we are going to give in the four figures that we were discussing so first of all as you can see here it says left right velocity so let's say this is zero and then it will say forwards and backwards let's say that is zero as well then we have up and down let's put it as zero and then we have the yaw velocity which means how much it wants to rotate so all these speeds we have and if we click on it and if we go there you can see here this is the left right velocity we have forward backwards up and down and your velocity so all of these we can use from minus 100 to 100 so if we want to go forward we will write for example 50 it will go forward we will write minus 50 it will go backwards with half the speed so we can go back here and let's let's write here let's go forward 50 then we will give it a delay so we will write here sleep for two seconds and then we are going to tell it to land so we will say me dot land now one thing that i did not write earlier was the takeoff so before we actually send the command we have to make sure we are asking our drone to take off so we will write here me dot take off and then we can send our command then we can sleep and then we can land now if you want to land properly and if you don't want the drone to be going forward while it's landing then you have to repeat this and you have to write 0 over here as well otherwise the drone when it's landing it will be going forward as well so first we will stop it completely and then we will ask it to land so it will land like a helicopter rather than an airplane so let's try this out and see what happens [Music] so as you saw it took off it went forward for two seconds at the speed of 50 and then it landed so now we can move the drone towards the right as well so we can copy this and as we know the first one is basically left and right so we can change this to let's say 30 and let's see what happens [Music] so as you saw the drone first took off it went forward for two seconds then it went towards the right side for two seconds and then it landed so if we write here minus 30 it will go towards the left if we write here minus 50 it will go backwards so this is the basic idea and if we want to rotate we can rotate it for example we can say over here we can say 30 and let's try this out what happens as you saw the drone first went forward and then it took a right turn so this time it is in uh the yaw not actually the right side and then it landed so these are the basic movements that we can control we can control up and down left and right forwards and backwards and we can control the yaw if you like the video so far give it a thumbs up and don't forget to subscribe so now we are going to learn how to capture image from our drone so that we can process it and do lots of different experiments on it so we will go to our tello course and we will create a new file and this file we will call its image capture and the easiest thing to do is just to copy the standard you can call this the boilerplate for the telodrome you can copy that and now we can continue with our code so the image acquisition part is not that hard what we have to do is we have to write me dot stream and we will turn this stream on so this stream will give us all the frames one by one and we can process them now because it is a continuous number of frames we have to use a while loop so we will write while true and here we are going to write that our image is equals to me dot get frame gets underscore frame underscore read and then we are going to write dot frame so this will give us the individual image coming from the tele drone and now what we can do is we can resize this image if we want to process it faster so to do that we need to import our opencv library so we can remove this this time we are not going to use it so we can write here import cv2 import cv2 and at the bottom here we are going to write image is equals to cv2 dot resize and then we are going to give in our source image which is img and then we have to give the size we want to resize it to for example we can say 360 by 240 so we are keeping it small the size of the frame so that it is faster so we can format this and then we can write here cv2 dot i am show so this is so that we create a window to display this result so we will write here i am show and then we will write here image and we will say this is the name of the window and then what do we want to display we want to display img so this is the idea and then we are going to write cv2 dot weight key now we are writing a weight key because if we don't write this the frame will shut down before we can actually see it so we have to give it a delay so we are giving it a delay of one millisecond so this should work well and actually let's do one thing let's remove this for now and let's try it without the resize and then we will try it with the resize so let's right click on image capture now again we have to make sure that the teledrone is connected now it has already turned off for me because i didn't turn it on for a while or i didn't use it for a while so let's wait for it to connect okay so now that it is connected let's go to image capture right click it and we will run the image capture so we have the 40 battery left for image capture it should not be a problem and there you go so this is the stream coming from my uh drone at this point you can see there you go so right now it's one two eight zero by seven 720 and what we can do is if we resize this it will become quite smaller so there you go so now it's much smaller and you will get some warnings here don't worry about these as long as you're getting the frames it's fine and you will see a delay in the frames now this is because of uh the transfer and this is because of the viewing in reality this delay is not that much but because we are viewing it we are continuously trying to uh perform all the graphical computations that's why it's giving us this delay so normally we will keep it at this size so that the processing is a little bit easier and it will be a little bit faster with all the different functions that we use so now we are going to use our keyboard to actually fly the drone to do this we need to get the commands of the keyboard first and then we can implement it with our basic movements and we can run them together so to do this we are going to create a new module called key press module so a module is basically a piece of code that can run individually and it can run from another script as well so if another script wants to use its functions it can do that as well so we will create our module we will write it as key press module and we are going to use pygame to actually get the commands from our keyboard so we are going to import pygame now the thing is that it has already been installed by default so here you can see when we installed the other libraries pygame was also installed if it's not there of course you can go to add and you can write here pygame and you can install that so we will write here imports by game and then we will initialize a window for our pie game so initially the idea of this library is to create different games so whenever you want to detect a key press it has to be within a game window so we need to create that window so we will uh create a new function called inits we are going to initialize our pi game dots init and we will write that our window is equals to pi game dot display dot set mode and we are going to give in our size so this size can be anything that you want so let's say 400 by 400 and then we are just going to call this function so let's call it here so we are going to whenever you are creating a module you will write if double underscore name is equals to main now this just means that if i am running this file as the main file then do this otherwise no need to do this so we will write here in it so let's see if that works i have to write double equals okay so let's run this and there you go so you saw that this window came up and it went away so what we can do is we can initialize and then we will update our window and it will stay but for now we will move on so what we need to do is we need to get the key press and to do that we are going to write by the way you don't have to understand this whole part you can think of this as just a function to get the key presses so if you don't want to get into the details you can skip this part as well but it is recommended that you understand how we are getting the key press so we will write here for example get key this will be our function that we will call from a different script and when we are calling the getkey function we can for example write here the key name so we want the left key we want a we want s whatever key we want we can write here so we are going to have a final answer so we will call this and we will write it as false so by default it will be false and we are going to return this answer so if the key is pressed it will return true if it's not pressed it will return false now without going into too much detail we are going to write a few lines of code we will create a for loop and we will check the events so we are getting all the events and we are not specifically doing anything with them so we will just pass but then we are going to create our input so we will say key input is equals to now to actually check whether it's true or false it has to go in a certain format so we need to create that format so here we are going to write pi game dot key dot get pressed and then we will write here my key so this is that format i was mentioning so we will write get attribute and we will write pi game and it has to be in this format so it has to be k underscore and then within the brackets it has to be the value of that what do you call key so here we will write format key name so what this will do is it will write it like this so for example if we say that our key name is let's say left so it will write it like this so this is the format that we need so this is what we are doing so once we have the correct format we can now just check if key input my key then we will write answer is equals to true and what we will do okay this needs to be input okay and then here we will just update our pygame screen so we will say display dot update now the reason i'm not going into too much detail with this code is because this is not part of the programming for drones it is just acquiring the key strokes so there are multiple methods there are multiple libraries you can use to do this so i'm not going to go into the detail of how this works if you want to know more you can check out my other videos where i have explained these so this part here now we have created a module and if we call in our getkey function and we give in a key name it will tell us whether it has been pressed or not let me format this and what happens is that we can write a main function here to test it out so for example we can write main and inside that we can write for example if get key so this is how you can get the key or let me just print it out for you first get key let's say we want uh a and over here i'm going to print this so what we can do is we can write here while true while true run main okay so basically first of all the code will go here it will check if this is the main file we are running if we are doing this yes if that is the case it will initialize it will run this code and then it will create this loop and it will keep running main again and again so when it runs main it will check get key a it will do all of this processing it will check whether it's pressed or not and it will give us the answer back so it will be either true or false so let's print it out and see what happens so if i run this this is my pi game window and you can see now it's staying and right now it's saying false but if i press the a key if i press the a key nothing happens why ah yeah it has to be on this window so you have to click it and then you have to press now you can see when i press it actually changes to true so this is a common mistake sometimes i forget as well you have to be on this window when you press the button so this is the idea so here we can write for example if get key let's say is left uh left i think it's capital letters left then we can print left key pressed then we can write if gets key is let's say right then we can print right key pressed okay so let's run this so if i run it here right now nothing has been print so if i press the right key it will say right key pressed if i press the left one it says left key pressed so this is how it works so what we can do now this is a module so if we run it by itself this will run over here but we can use only this function and this function so what we can do is we can go to the telecourse we can create a new file we can go to python and over here let's call it keyboard control keyboard control so here what we can do is we can import this module so let me close the previous ones so this is our module so we can right here imports keypress module as kp so the reason we are writing it as kp is so that whenever we use it we don't have to write the whole spelling we can just write kp so what we can do right now is we can write kp kp dot init we will initialize it this is that init function and then we will write kp dot get key and let's say we want to get the key s so we can print this out so let's print it out so now if i run this part it will create that uh now the thing is it only ran once and it gave us false so what we have to do is we have to create a while loop if we want to run it again and again so we will write true and then we will put this forward and then we can run this there you go so now it's saying false and if i press the s key it says true so now you can see that we created that module and we are able to use that module in other scripts as well so oh the previous one is also running anyways so what we will do now is we will attach this to our tello and we will using these keys we are going to move it so let's write down the code for our telodrone first so we will write here from dgi teleport import hello and then we can write here from time import sleep okay and then we will write here me dot delo sorry me is equals to tello dot hello and then we will write me dot connect and of course we can check the battery it's always good to print out the battery to know everything is working fine okay so this is our initial part once that is done we are going to take off so we will write me dot take off and then then we are going to write our code inside here so let's remove that and we will write here that let's say me dot send rc control we are going to send in four values so for now we are going to put them as zero and later on we will add something to it okay so what we need to do now is we need to check if the button or the key has been pressed if it's pressed then we will change this values so based on which of the keys were pressed so we can write a function here to check the key presses so let's call it get keyboard input and here we are going to define our parameters so this is left and right then we have forwards and backwards then we have up and down and then we have your velocity right so we will put all of them as zero and then we will check each one of them so we will write if kp dot get key left okay it has to be capital left then lr lr which is your left right is equals to minus 50. so this is a speed that we are just giving so we can change this if we wanted so actually we can write a variable here speed is equals to 50. so instead of writing speed every time here we can just write it here so this negative means it will go towards the left side now if we wanted it to go towards the right we can simply copy this part and we can say elsif and we can change this to right again this has to be right and then we can remove this okay so this is how you can do for one of them and the same thing will happen for the rest of them so let's say this is for up and down so we will write here up and down and for up and down we have to use up which is the up and down okay it should not be u p it should be u d up and down so let's write it d over here so here when it's going up it will be positive speed when it's going down it will be negative speed so this is how you can do that and then we will write now we have left we are left with the forwards and backwards and we are left with the yaw velocity so what we can do is we can write the same thing here copy that paste it here the reason i don't like to write it like this is because this is very easy to read now the correct method or the standard you can say is like this but i like to keep it in one line if it's just one line function sorry one line if statement so i like to keep it like this so that it's very easy to read what is happening okay so then we will write here w and we will write here s so we will use the wasd keys so here we are going to use okay so i think up and down will be easier to be used if we are using it with forwards and backwards because normally with up down left right we are going forward backwards like a car so i think it will be better if we use here forwards and backwards and here we can use up and down so w is up s is down and then we can use the a and d so a is towards the left side and d is towards the right side so something like that so we i'm not sure about the rotation here which is clockwise which is anti-clockwise so if it's not the right one we can switch the values here like this so let's keep it to this and if it's not right we will change it later so all of these values uh yeah i forgot to change this this should be your velocity and this should be your velocity as well so all of these values are now ready and what we can do is we can send it back and we can write here return and we can return all these values so all of this will be in a single list and what we can do is when we call this function get keyboard input the output of this will be let's say values so this will be a list and we can just send the values of this list here so values at 0 values at 1 values at 2 and values at three so this will directly send all the commands and what we can do is we can add a sleep here just to make sure everything is stable nothing is happening too fast we can control this and what else one yeah one more thing about safety is that we should be able to land uh with a key press as well so let's keep it close so we can copy this line and we can paste it here and instead of a we can write q and if we press the q button we want to land so we will write here me dot land so this will allow our drone to land if we press the q button so we could add a button for takeoff as well but or should we add or should we not add let's add a button for takeoff as well so let's copy this and we can write here take off and we can remove the takeoff from here and let's say that takeoff button is e did we use e no so let's put it as e so e will take off q will land and then the rest of them will be used to move forwards and backwards so this being done now we can turn on our drone i think i need to replace the battery so let me place the battery and let's try that out so let's connect to telo it's already connected and i think it should be fine let's right click and run and there we go so it is sending the commands and the response we have 93 battery okay so i will run this now and let's see what happens so in the previous chapter we had a look at how we can control the drone with a keyboard now in this project we are going to create a spy drone or a surveillance drone in which we can control using our keyboard and whenever we require we can take a picture of that environment and save it to our computer so at any given time we will be able to see where we are flying and we can take any picture whenever we press a button so here we can right click and create a new project we will call this project let's call it projects and then we will write here keyboard control control what else should we write let's write image capture it's a long name but anyways so this is basically our idea fest we are going to import so we are going to import everything from our keyboard control so this is the code that we wrote in our previous chapter so we will copy all of that we will close it and we will paste it here so over here we are going to make some changes and what we will have to do is we have to merge these two codes the one we did for image capture and the one we did for keyboard control so we will go to image capture and here you can see that we have to first of all turn on the stream so we will copy that and we will go to our initializations and once we connect we will turn on the stream then in the while loop we have to capture the image so this is image is equals to get frame and then we are resizing it then we are displaying it so we'll copy all of this and we will go back and in our while loop we are going to paste it now this is pretty much doing the same thing as this so we are going to remove this and here it's giving us an error because we haven't imported the opencv library so we will write here import cv2 so this will import the cv2 and now we can see that if we run our drone it will display us the image as well so let's try that out okay this is the previous one we have to run this one i think it disconnected i need to connect it again okay so now it's connected and we can run it and there you go so this is the image that we're getting from the drone we can go back to our buy game window and over here if we press e it will take off and then i can go forward i can go backwards i can go left right up down i can rotate so all of the things that we did earlier we can do again but now we are able to see the actual footage so this is great so we can stop this now what we need to do is we need to store this image once we have pressed any button so what we can do is we can create a new if statement here so we can write if kp dot kp dot get key and we will write that let's say it is the z key normally it would be s for saving but we are already using s so we will use the z key so here we will write cv2 dot i am right so this is for saving the image so now we have to save the image with a different name every time if we don't do that it will overwrite the previous image and we don't want to do that so here we are going to write a string so we will write here let's let's actually put it in the resources folder we sources and then inside the resources we will put it in the images folder and now we are going to give it a unique name so how can we give it a unique name every single time the simplest method is using the timestamp or the current time because the date and time will never be actually the same even if you use maybe one second later two seconds later it will never be the same so this way we can write here time dot time so this will give us a unique name every time and then we can write dot jpg why is it giving an error let's check out because we are okay let's just say import time so hopefully that error will go away yeah there you go it's gone away and now we are getting this error so we can write here time time dot sleep okay so here we are getting our time and then we are storing it as jpg but what are we storing we have to define that we want to store our image so the image that we are using here we want to store that so in order to use this inside this function we have to make it global so we will write here global image so that should work fine so let's run this and see what happens so actually before we do that we need to create a folder here so we will right click we will create new directory so our directory name should be resources so we will write here resources and inside resources we need to create another directory called images so this is the directory we have to create and once that is done we can now run this so again it got disconnected uh let me connect it again okay so now that it is connected we can right click and we can run and there we have it so without actually running it again we can press the z button and if we go to our resources and images there you go you can see there's a lot of images saved and if i just right click and click on show and explorer there you go so these are the images saved now the problem is when i press the set button it clicked so many times and it saved so many times so what you can do to avoid this is you can add a delay so right after you press that you will say time dot sleep and let's say for 0.3 seconds so if i run this again let's press the set button and this time if we go back there's only one image so we can stop this and we can go here and there you go so this is the image that we get so let's fly it and see what happens and how we can get the images okay so let's take off okay we are getting some error i think we need to reset the teledrone let's try it one more time yeah it is unsuccessful so we need to reset the teledrone to actually run it again yeah the drone was actually turned off so we have to turn it on again and now let's connect and run it again okay so right click and run and there we have our pi game we can press the e button to take off we can go forward let's rotate a little bit let's okay let's rotate back let's take the image of this sofa then let's move towards the rights and let's take an image of this chair and let's move on the side let's take an image of this chair and let's go back and we can stop it so we will go to images and show in an explorer and we have the previous image that we took from the previous run and here we have the new images so this is the sofa this is one chair and this is the third tier so as you can see we can fly around and take pictures and they are getting one picture at a time instead of just having a lot of images at the same time okay so this is good so it's very simple as you can see you can gather together the different codes the one we did for keyboard control and the one we did for image capture you can merge them together to create an exciting project if you like the video so far give it a thumbs up and don't forget to subscribe in this chapter we are going to look at how we can map our drone now this is also known as odometry the idea here is that we are going to use the velocity information to find the distance and we will also use the angular speed to find how much angle the robot has or where is it headed and based on this information we are going to get the x and y coordinates so we will take the distance and the angle and we will convert it into cartesian coordinates so we can plot it in a graph so let's have a look at the explanation of this before we actually go into implementing it so let's say that this is a virtual grid that we have and every time we press the forward button or we rotate it can rotate only in these grids so whenever we press one time it will go forward and then we press it again it will go forward again now what is the distance that it will travel that we will calculate so let's say that the speed of our drone the one that we have been uh using in our previous chapters let's say that is 10 meters 10 centimeters per second so that means that for example if we take the time as two seconds then it will move 10 centimeters then again 10 centimeters so it will move 20 centimeters so our distance will be 20 centimeters so to convert this it's very simple we have 0 0 and then when it goes forward it has moved 20 centimeters so the coordinates will become 0 and 20. so that's how simple it is but let's have a look at what happens when we introduce an angle let's say that we do not move forward all we do is we rotate so let's say that we are just rotating so we are rotating like this so the idea is that we have our angular speed so let me bring that back so let's say that we have our angular speed as 45 degrees per second so it means if we rotate it twice so if we are here and if we rotate it twice so it will rotate one second 45 degrees another 1 second it will rotate 90 degrees so this is what will happen so now our coordinate is pretty much the same 0 0 but our heading is towards the east instead of north so what can we do to actually find the coordinates when we are pressing both the forward button and the right or maybe the left button so we are rotating and we are also moving forward or moving backwards so how can we calculate that so by the way the angular speed is pretty much the same way uh that you would calculate the distance so here if you are moving let's say for two seconds so 45 degrees per second for two seconds will give you an angle of 90 degrees so that will be your heading after two seconds so let's look at how we can use this and how we can find our coordinates so let's say we pressed the forward and we press the rotation button at the same time so now the drone starts moving but this time it will start rotating as well at the same time so at the same time it's moving forward at the same time it's uh turning so it will start moving like this so at the end of the day it has moved let's say two blocks and then it has rotated 90 degrees so how can we find these coordinates where it is now i'm just assuming that it is here but how can we actually find these coordinates so what we can do is let's say let's assume that these coordinates the final point is over here so what happens is that this is our traveling distance so this is basically it will be a curve it will not be uh going forward and then rotating it is happening at the same time so how can we find that well in this case uh let's say this is our final point and this is the this is after one second and this is after two seconds so let's say this is our curve so what we will do is we will find the coordinates after every unit of time so whatever distance we have kept constant or whatever time period we have set in this case we have one second so after one second we will look at what is our position so let's say let's say it is here let me place it a little bit nicer there you go so let's say that this position is over here so then what we will do is we will find first of all we have the distance so in the first interval our distance will be 10. after 1 second our distance will be 10 and our angle will be 45 so what we can do is we can write the equation where d is your distance then d multiplied by cosine of 45 so d here will be the value of 10. so distance multiplied by cosine of 45 and then we will have the sign so d here again will be 10. so this will be 10 multiplied by sine 45. so this will give us the y-coordinate and this will give us the x-coordinate so this way we will have the x and y-coordinate over here so the answer of this is about seven and the answer of this is also about seven so this point here will be seven and seven so if we just copy this we can write here that this is seven i think seven point zero seven and 7.07 so after what you call one second this will be our coordinate so we can place this dot uh in our actual diagram and then again we will if we are moving with the same speed or the distance and then we are moving at the same angle then it will also have the same effect so we will have 10 sine 45 and 10 cos 45 again so this calculation we will do every single time after every block of unit that we have traveled so it will be like this and then again we will have the same calculation but this time we will add our previous value to it so because uh earlier we had zero zero so we didn't do anything but this time we will add to it so it will be 7.07 plus 7.07 so that will give us 14.14 so this coordinate here will be 14.14 and then 14.14 so if we are moving at the speed of 10 meter per second and the angular speed of 45 degrees per second for two seconds after two seconds will be at 14.14 and 14.14 in our cartesian coordinates so this way after every unit of time we can measure and we can pinpoint exactly where we will be so this is what we are going to do so let's have a look at how we can implement this mapping method so we will go to our new file and we will name it what should we name it let's say mapping okay so the first thing we want to do a lot of this is related to the keyboard control because we will be the ones controlling on the keyboard and when the drone is moving we will plot it where exactly is it moving so we will take the keyboard control we will copy all of it and let's remove the other ones and we are going to paste it so the first thing we have to do here is to write down our parameters so as we have discussed earlier we need the speed and we also need the angular speed and from that we will calculate the distance and we will calculate uh how much angle do we have now first we are going to write the forward speed now the thing is that we already have the forward speed the one that we are sending so for example 50 means 50 centimeter per second but the thing is that when i experimented with it and tried to find out my own values to see if it's correct or not a lot of the times the values differed so the teledrone is not very repeatable and it is not that accurate but still we are trying to learn the concept and based on my testing i found out that uh if i move for example 10 seconds i was getting a distance of 117 so 117 divided by 10 this is basically our speed so i can write it here forward speed which is basically in centimeter per second so this is the centimeter how many centimeter it traveled and this is how many seconds it traveled so it traveled in 10 seconds 117 and i was giving the speed of actual 15 centimeter per second and you can see it's coming to about 11.7 around 12 centimeter per second so again doing some manual testing can help as well and again with the angular speed i tested it out as well so the a speed let's write it capital here as well the angular speed it took about 10 seconds to rotate 360 degrees so our speed will be so angular speed in degrees per second so this will be our angular speed in degrees per second now in the example we took the interval as one second so we were checking the value or we were plotting the value after one second now that is a very long time we don't want to do that we will reduce the interval to actually interval is equal to 0.25 so based on this information now we have the speed and we have the interval so every unit of our map will represent a certain amount of distance and a certain amount of angle so that distance and angle we can find we can write here that the distance interval interval is equals to forward speed multiplied by our interval and then the angular interval is equals to a speed multiplied by interval so this will give us the distance and the angle every time we move one unit okay so these are basically our parameters now you can play around with these you can try it yourself you can time it using a stopwatch you can write an actual code by yourself to test these different things out but these are the values that i found to be a little bit towards the better side so again this is not very accurate so don't expect it to be very very good but the idea is that if you understand this concept and if you come across a better drone then you can use the same methodology to create a more accurate model okay so that being said what we can do next is we can try to actually plot something to see how the opencv actually works so in order to plot let's actually remove this in order to plot we are going to create an image and we will create a black image so we will call it image is equals to and we will use the numpy library now we haven't used numpy before it's basically a mathematical library that allows all these different mathematical or matrix calculations so we are going to import that so we will say import imports numpy as np so we are going to use that so image is basically a matrix of pixels a matrix of numbers so that's why we can use the numpy library which is known for handling matrices so we can use that to call numpy.0s we want a matrix of zeros and we want it to be the size of one thousand by one thousand by three so uh the three is basically the channels so we wanted a colored image so that's why we have rgb channels and here we will say that the the values that are stored in these matrix they are basically unsigned unsigned integers so that's how you write unsigned integer and it should be 8 bit 8 bit means that we can have a value from it means 2 to the power of 8 2 to the power of 8 which is equals to 256 so therefore our values will range from 0 to 255 so these are the values that will be stored here okay so that being said let's draw something on it so let's create a function called draw draw points and we will call this function whenever we want to draw so right now we will just draw something simple uh let's just draw a circle so cv2 dot circle and we are going to give in our image so the image that we want to draw the circle on then we will give our points so let's say 300 and 500 and then we are going to say that our i think the next part is size let me check radius wait it's not it's not supposed to be radius what is happening is cp2 imported oh it's not imported that's why so import cv2 let's try now yeah there you go so now it's telling us what is required so the yeah the next thing will be radius then color then thickness so radius let's say 20 color let's say red so let's write 0 0 and 255 so it's bgr in opencv we don't use rgb we use bgr so this is blue green and red and then we are going to say thickness we want to fill it so we will write here filled so oh i wrote it wrong over here that's why okay so we can do a little bit of formatting here and okay let's run this and see what happens oh yeah we didn't actually show anything so we have to show this cb2 dot i am show and we will write the window name so let's say output and then we will write image and then we will write cv2 dot wait key to have a delay in between of one millisecond so let's run this and there you go so this is our output screen it is 1000 by 1000 and you can see the circle is drawn over here so we can make it a little bit smaller it's really big let's make it five let's see and there you go so now this is a dot and if we keep adding more dots uh then we can create our plot that we need okay so now what happens is that we have to find the value of how far we have traveled so let's call this these are our final values let's call them x and y so we will keep it very simple we will call them x and y so here we will say x and y both of them are 500 so actually let's write it at the top so the idea is that at any given point we want to plot this x and y so we can simply go here and we can say instead of this we can say y and we can say x that's it so if if we run it now it should plot right in the middle so this will be our starting point and then whenever we press a key this x and y value should change so what kind of change that we have to put here so let's have a look so let's look at the left first so here what we will do is we will add to our distance and then we have to add to our angle so at any given instance whenever we press the key we have to check the angle and the distance we have to update them so this is the final value in the x and y coordinate but there should be another value which is basically the distance and the angle and we will check every time what is happening to them so the angle we will keep the same so the angle should not be reset to zero every time because it will continue to add on but the distance we will reset every time so we will write d equals 0 here inside and a equals 0 outside okay so now we need to update this whenever we press a key so for the left we will update our interval so we will write here d is equals to our dp our d interval so normally whenever we are traveling we are traveling with this distance but we are not traveling in the forward direction we are traveling towards the west so we have to change our angle so we will put the angle as minus 180 degrees now for the right we will copy this for the right we are going to change our angle and the angle will be 180 degrees and our distance will be negative then we will go to up now for up our interval will be positive but our angle will be towards the north it will be 270. and then for down our angle so this is not actually up and down it is basically forward and backwards so for that we have to write that our interval is again negative and then our angle is minus 90. okay so this is basically simple because we are just moving left right forward and backwards but when it comes to actual your rotation then it gets a little bit tricky so we need to change the angle so here what we will do is we will check our previous angle and we will add to it so a will be that our yaw is equals to plus equals to our a interval so it will not just be a new value it will be added to our previous value and in the other direction we will remove it so we will write here minus a interval now we did not declare yaw so we need to declare that and we will write here your equals to zero and we can write here let's say global yaw okay so after doing that now we have updated all the values we need to update now we need to convert it into cartesian coordinates so here we are going to update our angle so angle will be plus equals the yaw and then we are going to get our x and y so finally we will write x is equals to actually plus equals to because we have to add it to our previous value initially it will be 0 0 then it will be added so we will write here integer and then we will write the distance multiplied by the math math dot cosign i think math is not imported so we need to import here import math and if we go down okay so math.cosine and then we will write math dot radians because this function actually uses radians so we have to convert it into radians so math.radians and then we can enter the value of a so there is a problem most probably because of brackets this is for this this is for this oh no it's the spelling okay so is that good there is still an error for some reason mad dot cosine math dot radians a and then okay the problem i think is because x is not global so we will put it as x and y there you go then we can copy this and we can paste it down here and we can say that this is y and this is sine instead of cosine so do we need to change anything else no so this seems fine so what we can do now is we can send this x and y over here now it's already been sent it's already been updated so we don't really need to worry about the points but we could actually write it from here we can send it back from here and then from here we can send let's say our image that we want to draw on and we can send the points so we can do this but it it should work automatically as well but this is a better way to do it so we can write here points and our points is basically points is equals to the values at four and the values at five okay so let's run this and see what happens point zero and then point one so let's run this and see what happens okay we get an error local variable a referenced before assignments where is a so this needs to go here let's try it again and there you go so this is our buy game we have to click here in order for it to run whoa that that goes really fast okay so it's going back and forward um and then it's is it rotating yeah kind of okay so it is working but there is an issue what is the issue uh let's try to find out okay so the first thing is that i forgot to add the delay so if the time is not accurate then all of our calculations will be wrong so we need to write here time dot sleep or i think we imported sleep directly so we can write here for example 0.25 so this has to be exactly the same as the interval so we will copy it from here and we will write it here so this has to be our interval and now it will be not that fast as before because of that interval so let's try it out yeah so now it's much slower and it seems much better and you can see now it's rotating as well when i press some keys but again the problem is that we cannot see the path so it's always printing this point but then we don't just want this point we want all the points that have been plotted since so what we can do is we can create um points we can go here and we can write points and we can put it as an empty list then we can go down and and instead of sending these values directly here what we can do is we can append so we can say points dot append and we want to append these values so this will keep adding to the points so let's and the other thing we have to do is here again we have to loop through these points and then we have to print them out or we have to draw them so we need to add a loop here so we are going to say for points in points we are going to write point at 0 and point at 1 or we could simply write point as well it should work so let's run that and see if it works we get an error argument is not a tuple okay so which line is it okay i think we just need to change the brackets here because those are for a list so hopefully this will work yep okay so there you go and now if we click and there you go so now if i move forward you can see i have kept the forward button pressed and now if i press d you can see it's rotating but it's rotating in the wrong direction okay so now if i press a you can see it's moving and if i press left it's going left if i press up it's going up and it will give us the estimate so we can simply print out the value of x and y and that will be the coordinates or that will be the distance traveled so we can we can print it here we can say put cv2.put we'll put it on the image and we can write the points so we will write here f and then inside that we are going to write a bracket it's here points we are referring to this and not this we are outside the loop so we will write points the last one so we want to put this text at the very end so we will take the last value and we will take the zero element of it which is the x and we will subtract 500 from it because our starting position is 500 so we will subtract 500 from it and then we will divide it by 100 so that we are giving the value in meters and not centimeters and then we have to write the other one so the other one will be pretty much the same thing so we will copy that we'll paste it here and we are going to write -1 and over here we will write 1 and then at the end we are going to write meters so that should be good so this is basically the distance or the x and the y coordinate printed out but now we have to mention where do we actually want to put this so we need to define the points uh the point where we want to put it so we don't want to put it exactly on the same point we want to shift it a little bit so that's why we will write points minus 1 and 0 and we will add 10 to it so that it goes a little bit forward and then we can copy this and paste it here and this time we will add let's say 30 so it is a little bit down and we will make this one so we can format this like that let me bring it here a little bit okay so that should be good enough and there was another issue that it was not rotating in the correct direction so when i was pressing a and d the the values were reversed so this is supposed to be plus and this is supposed to be minus okay so that is one thing now another thing that we did in the previous what you call the keyboard control chapter is that we wrote a single value here the speed as 15. but in this case we will have a different angular speed and we will have a different what do you call the forward speed so here we have to either we can write it here or we can change it down here so let's keep this as speed so we will keep it at 15 because that is the speed that i used for testing and the angular speed i used was 50. so i used 50 so we can write a speed is equals to 50. so we don't need to change this but we need to change this down over here so there you go so now if we run it okay something is missing oh yeah we didn't complete it we didn't complete this there is a font missing so we have to write the font cb2 dot font plane and then we have to write the thickness what else then we have to write the color we forgot a lot of things two five five zero two five five and let's put the scale as well so this is the scale and this is the thickness so let's put a comma there and let's run it there you go so right now it's saying 4.95 and 4.95 uh why is it saying that it should be zero zero at this point uh where did i make a mistake points at minus one um i think it's using zero zero maybe so we need to check here that if if the points minus 1 at 0 is not equals to values at 4 which means it is the previous value or the points at -1 one is not equals to values at five now this just makes makes sure that we are not using we are not plotting the same thing again and again then we will append otherwise we will not append okay why is this giving okay let's run it index is out of range indexes out of range okay so what we can do is we can give it some values in the beginning so it is not empty so we are just giving two values in the beginning so it can compare there you go but the value is still 495 why is that let's check our value here i think we need to add a bracket here maybe let's run this yep that was the problem okay so now we have zero zero and let's move it forward and there you go so now we are getting the distance as well as the coordinates in both directions and we are getting the map as well as i'm moving it i can see what is happening so i can press these different keys and it will keep rotating and at the end of the day it will tell me how far i have moved so what i can do at the end is i can create another circle so that it is for the heading so i can change the color for pg let's make it green and let's make it a little bit bigger so we will make it let's say 8 and we will use the last points so let's write again okay i'm missing something again it's points okay there you go so now it's green and if i move it there you go now it looks much better gives us the estimation of where the drone is so now let's test it out on the actual drone and see what happens [Music] [Music] so [Music] [Music] [Music] [Music] in this project we are going to look at face tracking so we will look at how we can make our drone to track a particular object or a face or even a full body so we will do it with face tracking but later on you can implement it with any of the other objects as well so first of all let's have a look at the concept now when we have to move the drone and we have to track our face we need two different things we need the motion of forward and backward and we need the motion of rotation so we need your and forward and backwards so let's have a look at forwards and backwards first so we will treat both of these individually and then we will combine the code for them so let's have a look at the concept of forwards and backwards so if the object or the face is looking small then it means we need to move forward and if the object or the face is looking quite big then we need to move backwards so we are too close we need to go back so if this is my drone and this red area is where it is too close and this red area is where it is too far so both of them the drone has to move this green region is where you do not have to move at all so let's say the person here she moves backward so now the drone will have to compensate and it will move forward so it will go up till that green zone and then it will stop now let's say she moves forward then the drone will have to counter the movement by going back and it has to go back until it reaches the green zone so this is the idea that there will be a green zone in which the drone will not move it will be stationary and if it's too close it will go back if it's too far away it will go forward so this zone we are going to define by area so let's say the area over here is 6200 and the area over here is 6800 so these are the values that i found after some testing so we will be using these values to actually put in our drone so this is the area so if you take the number of pixels so the width multiplied by the height so you will get this value so these are in pixels again you can change these values later on according to your needs if you're doing a full body the area will be different than if you are doing a face or you're doing another object so that being said let's have a look at the yaw angle so the idea here is that we are able to go forward and backwards but what if the person starts rotating the idea here is that we always want our object to be in the center of the image so let's say this frame is the image that we have and the person is not in the middle so then if they are on the left side you will move the drone you will rotate it and this will bring the person in the center and if for example if they are over here and then you can rotate and this will bring back to the center now the thing is we can give it a value for example let's say at this point our object is on the left hand side so we will say rotate anti-clockwise direction with the speed of 20 let's say so it starts moving at the speed of 20 and when it reaches the center it cannot stop immediately it will move a little bit forward because of the momentum think of this as a fan so if you turn off a fan it will not just stop immediately it will keep rotating and after a while it will stop so the same thing will happen here so if we tell it go 20 degrees or sorry go with the speed of 20 centimeter per uh second and when it reaches the center if we tell it to stop it will not just stop immediately it will go a little bit further this is called overshooting so the same thing will happen if let's say she's on the other side and then we do it here so what happens is when it overshoots the image will not be in the center it will go here then it will try to retrieve and it will go here so it will keep going back and forth and it will not stabilize so how can we make it better so we can make it better by reducing the speed as we reach our destination so let's say the speed here is 20 and once we are getting closer let's say the person is here now then we will reduce the speed let's say right now it's 10 then it will become 5 and by the time it reaches the center it becomes 0. so this way instead of having plus 20 and minus 20 it will have a value that will change based on how far are we from the distance or how far are we from the actual point so this method is known as pid i will not go into the detail of how pid works more than this i think this is a very basic explanation that will give you a basic understanding but if you want to learn more about pid it's basically proportional integral and derivative so these are the terms it's type of a controller that you can use you can relate two different entities together which are not relatable by default so these things are a little bit complicated if you haven't seen them before but for this course this explanation is more than enough that it is able to go back and forth and when it is reaching the center it has to slow down so we will be using this methodology in our implementation so let's have a look at the implementation now so we will first create a new file and we will call this face tracking so this project is more towards opencv so we will first look at the opencv part and then we will see the integration towards stello so let's import cv2 and then we will import numpy as np so what we will do first is we will run our webcam so i will be using my standard webcam rather than the teledrone and once everything is done we will integrate afterwards so i will write here cap is equals to cv2.video capture video capture and i have multiple webcams so i'm using the number one if you don't have multiple webcams most probably you will be using the number zero okay and then we will write while true true we are going to get our image from cap dot read and then we will say cv2 dot im show and we will say that this is our output let's say and this is our image and cv2 dot waits key and we will give a delay of one so this is the standard code for actually running the webcam with opencv so let's run this and see what happens so right click and run so here is my webcam so i will be using my image for the face tracking parts and once that is done we will try out with the telodrome okay so now the first thing we will do is we will track or we will detect the face first so we will create a new function and we will call it let's say find face and we want to find it in an image so we will pass that image that we want to find it in and to find these faces we are going to use a method uh proposed by viola jones so this is a very famous method of using har cascades so this this is basically a file that has all the parameters and information of a model that helps us detect different objects in this case the file that we have will help us detect faces so let me copy that file so this file is available on the official opencv website and it is also available on my website under this course so you can download it from either of the places so you can go to resources and we are going to paste it here so the name of this file is our cascade frontal face default actually we are going to click on rename and we will copy this name because it's too long and now we have to refer to this file so we will say that face cascade is equals to cv2 dot cascade classifier and then we are going to write in the image path sorry this file path now this file is inside resources so we have to write here resources and then we can write the link or the path of our file then we are going to convert this into grayscale so we can do by writing cb2 dot cvt color now if all of this is overwhelming for you what you can do is you can go back and check out my opencv course it has been very popular it has more than a million views now with 99 positive feedback so if you go through the opencv part this stuff will be very easy for you but if you don't have a lot of background with opencv then just take my word for it that for example this line is for converting to grayscale so then we will write image and then cv2 dot color underscore bgr to gray so this will convert it into gray and then we will detect all the faces so we will write faces is equals to face cascade dot detect multiscale and we are going to pass in our image so the gray image that we just did and then we are going to pass in some parameters now these parameters normally the first one is basically the scale factor and the second one is the minimum neighbors so what you can do is you can play around with these values the ones i found to be good are 1.2 and 8. so again you can read about these by going to their default function or you can check it on the online documentation i will not be going into that much detail here so then we are going to create a list of all the faces and then we will create a list of all the areas of these faces so there can be multiple faces so we need to find the biggest one and we will assume that that is the one we want to track because it is the closest one to the camera so we will detect that one so we will write here my face list is equals to empty and then we can write my face list area actually right let's write here c for center so this will have the information of our center point so cx and cy where is our face detected and this will have the area value so these will be two lists and then we will create a for loop so we are going to write 4 x y w and height so basically whenever it detects these faces it will return us these values so we can unpack these values within this faces so we can write x y z in faces and then we will write that we want to draw the rectangle around this face so we will write here cv2 dot rectangle and for the rectangle we have to send an image we have to give it the x y coordinate and then this is the the starting point and then we need the ending point so the ending point will be x plus width and then y plus height then we need the color so let's put it as red so it's bgr so 255 will be at the end and then we have the thickness let's put it as two okay so that is good now let's run this and see what happens let's try it out before we go any further and we need to call this function so we will call it by writing it here and let's run it so there you go so now you can see the face is being detected properly and it doesn't seem to be that bad even if i'm moving it's 30 and it stays there if you don't find good results you can always change these values to get something better out of it now once we have this what we will do is we will get our center x and center y so we will write here c x is equals to x plus width divided by 2 and then c y is equals to y plus height divided by two so once we have this we can also find the area area is equals to width multiplied by height as simple as that multiplied by height and then we are going to write uh that we want to append this but uh should we print okay let's let's just append it first so my face list c dot append we are going to append the value of cx and cy okay and then we are going to append my face list area dot append and we are going to append the area so once we do both of these things what will happen is okay let's let's print it first so we can draw a circle at this cx and cy position so that we see where exactly is the center so we can write here cv2 dot circle and we need an image what else do we need we need the center so the center will be cx cy and then we need the radius let's say radius is 5 then we need the color so let's say zero two five five and zero and then we need the thickness we will say filled cb2 dot fill we want it complete uh filled up from inside and that should be good so let's run this and see what happens and there you go so now you can see the center the value of the center is being displayed as well so this value is what we will return and but what we can do is because right now we only have one face so it's obvious we will return this value but the reason we did these two lists is because we want to find the maximum area and we only want to send that value back so here we will check if the length before we actually do any calculations on this we need to check if this is empty or is it filled so if it's empty there's no point of calculations so we will check first my faces we can pick any one of these my faces area is not equals to zero if it's uh it has some values inside then we will say i is equals to my face area dot index we want to find the index for it and then we want to find the index of the max value so the max value of our area my area list okay so this will give us the max value of our area and this will give us the index of that max value so once we have that index we will say return image and then also return the my face list at c at i and we also want to return the area because we want to use that so from that list we will use this index to find the maximum value of the area and from that area we will get the cx and cy values of that corresponding face but if this is not the case then we need to give it we need to return something else so we will say else return the image and then return 0 0 for the cx and cy and 0 for the area so this will we're pretty much returning nothing okay so now what we can do is we can get the values so we have image and then we have what else do we have we have the cx and cy we can call it collectively as info so this will be the info that we will get back and once we have it what we can do is we can print it out so we can write here info at let's say 1 this will give us the area so we can write here area area is info one so let's try that and there you go now you can see the area being detected so as we discussed in the beginning in the explanation if we go forward the area will increase if we go backwards the area will decrease so based on this we are going to move our drone okay so that is good and then we also have the center value so the center will be zero um should we print it all together maybe it will be easier to read so let's just print it here like that so let's run it again so there you go so now we are getting the center values and we are getting the area so the center values will be used to rotate and the area value will be used to go forwards and backwards so this is good now what we can do is we can introduce our drone and we can also introduce the range so let's let's do the tracking first so here we will define a function so we will say track face and what we need is we need the drone so the variable me and then we need the info and we also need the width and the pid and the p error now i'm going to explain all of what these are so don't worry about it at least for now so what happens is that we need to create a range so let's we are doing the forwards and backwards right now so we need to create a range so over here we are going to write that our forwards and backwards range is equals to uh as i mentioned before it was six to eight hundred sorry six two hundred and then six eight hundred so this these were the values for our range and then we are going to where is it and then we are going to write here that if our info or should we call it area let's call it area so it's easier to see what is happening area is equals to info at zero now it was at one so if area is greater than the front uh the forward and backward range at number two then we need to go back so the forward and backward speed will be minus 20 okay so this is that and then we can write else if the info or not the info the area is less than the forward and backwards range at number zero this should be 1. then we will say that forwards and backwards is equals to 20. so at this point it is saying that if it's too far then go forward if it's too close then move backwards now we need to add a safety factor here and the reason is that if we only say that the area is less than this value which is 6000 something then 0 is also less than 6 000 so if nothing is detected it will start going forward so we need to mention here that and area is not equals to 0. so if it's not equals to 0 and it is less than our range then move forward otherwise it will move backwards and we also need to add a condition where it will stay stationary so that should be the first priority that if it has to stay stationary stay stationary so we will write here if area is greater than our forward and backward range at zero and our area is less than our forward and backward range at number one so this is the idea and then we will write forward and backwards is equals to zero and we can make this else if so now we have these three conditions the first one is where we have the green zone the drone will not move at all the second one is where it is too close it is too big so it will move backwards and this one is where it is too small or too far and it will move forwards so now all we have to do is we have to send our command to the drone we can send this fb value forwards and backwards that's it now let's have a look at the pid for our angle so for the pid we are going to introduce the error so error is equals to let's let's unpack our x and y as well so let's remove this bring it here and then we can say that our x and y are basically info at zero so here we will say that our error is basically the x position so we will say that x minus the width divided by 2. so what is this width divided by 2 it is the center of our image so we want to find out how far away our object is from the center so this is the value of the object or the face and this is our center so when we subtract it we find the deviation how far is it once we have that so this w is basically the width of our image so we will define it when we are calling this function and then we will use the equation of pid now again you don't have to go too much into detail of what this does if you want just to know how it works you can copy the code otherwise if you want to go into detail i would highly recommend you check out any of the other videos on youtube that have plenty of information about pid so we are going to write here that our speed is equals to pid this is a constant value so we are basically changing the sensitivity of our error by this value so we will declare it here we will call it pid is equals to and we will give in the values of our pid we are going to use 0.4 0.4 and 0 in our case so this is proportional p this is integral and this is derivative so we are only going to use p and d we will not use i so here this is proportional this will be p and then what we will do is we will multiply this with the error and we will add the pid1 which is your derivative we will multiply it with the error minus the previous error so this will give us the speed and what we will do is we will clip this speed between minus 100 and 100 we don't want it to go above 100 we don't want it to go below 100 minus 100 so we will write here speed is equals to integer we will convert it into injured integer and we will say numpy dot clip and we will clip the speed between minus 100 and 100 so this is our new speed for our yaw that we can send in to our drone so if you remember for the drone we can send in the value something like this so we can say me dot send underscore rc underscore control control and then we send in uh all the four values so the first one will be zero the second one will be forwards and backwards so it will be this and the third one will be zero and the fourth one will be the yaw which is our speed so this is how we can send our command to the drone now one more thing that we have to do we have to return this value return the error because we need to use the previous error so this error when it comes back here it will come back as the previous error okay so here we are going to write error uh is error global no it's not global but we can write e error as global so yeah we can do that and then when we call it so let's call this function so after we detect that we are going to send all of this and we will get back the p error which we will send in in the next one okay so here we have to include the code for the drone we will still wait for that we are sending in the info which is good then we need to send in the width so where can we define the width we can define it up here we can say that our width and height is equals to let's say 360 by 240. as we did earlier as well we can resize our image based on these values so we can write here image is equal to cv2 dot resize and we will give given our image and then we will say that we want it as width and height so this will be the value and then we can send this and we already have pid at the top declared so that should be fine now again you can change these values if you don't find a good results you can change these values and try out different parameters i did not put much effort on tuning of this because tuning usually takes a long time so i'm showing you the basic method and later you can tune it yourself and see how it works okay so that being said what else is missing do we have anything in the track face so in the track face one more thing that we have to do is that we need to define that if we do not get anything then we have to stop so we have already defined it for this forwards and backwards but we didn't define it for speed so just to make sure we will write here if our let's say x is equals to zero then our speed is equals to zero and our error is equals to zero so our previous error will be zero okay so that is good enough and now we can bring in the code for our or let's print out actually let's remove this and let's print out these values so let's remove this and let's remove this and we will print out the value of the speed and the forwards and backwards so we will write here print and error and fb so let's run this okay so we have an error so fb is not declared so let's go back up and is it changing every time so we can write here fb is equals to zero okay there you go so let's look at the webcam and there you go so we are getting the center value and this is uh let's let's remove the other prints because it is annoying okay let's try it out again so as you can see the first one we are getting so you can see here we are getting forward uh so we are getting backwards because it is too close and then if we go back uh i think it is too big to actually display here because the 6000 value is too small so we can try it out later but for the angle we can see that if we move left or right you can see that the speed changes and when it's at the center the speed will be close to zero and then we start moving away the speed will increase uh further so that is the basic idea but here we are seeing that it is going to negative 180 so let's see why that is happening so here our speed oh because we are displaying the error we should display the speed and not the error okay so now if when it's zero it's displaying -100 should display zero yeah because we need to print it down here so this is right before it goes to our drone and there you go the speed is zero so once it's detected there you go and yeah it works fine so now we can test it out in the actual environment but before we do that we need to add the part for our drone so let's go to image capture and we will copy the code from there so we will copy all of this and we also need to copy dgi telepi so we will paste it here and we will remove the import cv2 and then we will go to image capture again and we will copy these two lines of code or maybe just this one and copy it from here and if we go down we can remove this we can comment it and we can paste this one here and that should give us the frame and we can comment this and we can uncomment this part that will send the values to the drone so let's not print out anything and then let's try it out and see what happens now we need to add a safety feature as well so when we press the q button it will land and we need to add the takeoff part as well so here we will write me dot take off and what i have observed is that when it takes off it's at a lower level than the average height so what we can do is we can uh make it go a little bit higher so that it's easier to detect the faces rather than going down and bending so we can write here that actually let's copy it from down where is that there you go so we can copy it from here and we can paste it here and we can say that we will not go in the forward backwards or yaw instead we want to go up and down so we will go up at the speed of 25 and we will do it for let's say time dot sleep we will do it for two point two seconds so it depends if it goes a little bit higher then you can reduce it if it's lower you can increase it and then for the landing we can write something here now this will be a little bit confusing but just understand that this part of the code when we press the q button it will stop it will land so we are using the cv2 library to actually do this so we will write here if cv2 weight key 1 and 0 xf equals ord we will write q so if the q key is pressed me dot land and then we will break break means we will finish the while loop so it will end so that is the idea and that should work so let's try it out [Music] [Music] [Music] [Music] if you like the video so far give it a thumbs up and don't forget to subscribe in this project we are going to create a line following drone now the idea behind line following drone comes from the line follower robot so let's try to understand how that works and then we can implement the same methodology on our drone so let's say that we have a line following drone which looks something like this if we google the results you will see something like this so you can see we have a lot of different types most of them they have a long chassis and then at the end they have sensors so it can be three it can be 16 it can be eight so a lot of different types you can see here so this is let's say the exact same robot it has these two wheels and we are looking from the top and these are the three sensors that can be either on or off so if let's say the robot is on the black line then it will be on if it's not on the black line then it will be off so at this point these two will be off and this one will be on so let's have a look at the concept so let's say i want to move forward now here this middle one will be on and the left and the right one will be off so the robot will move forward now once it reaches this point now both the right and the middle one are on because the black line is both available on the right and on the middle sensor so then we can tell it to rotate a little bit now if it keeps going further and it comes to a point where only this one is left it means that it needs even more rotation so and in that case we will rotate even further to come back to the middle part until we get this point then it will keep moving forward and at some point you will see that maybe this one and the center one start detecting the black line then at that point you have to go backwards in the anticlockwise direction so this way if we have this sensor on we need to rotate in the anticlockwise if we have this one we need to move clockwise so we are always moving towards the center of our line now if we talk about binaries now if we talk about numbers if there is no uh sensor with the line available then we can say it is 0 0 0. now if one of them is on then we can write it then we can write it as 0 1 0. so we can have all these different conditions where we can say if it's 0 1 0 then go forward then let's say that we have this in that case it will be 0 0 1. so if we have 0 0 1 we can go right so we will rotate towards right and if we have let's say 1 0 0 which will be this case then we can say go left okay so this one is straightforward but because we have three different sensors the possibilities are two to the power of three which means eight so we have a total of eight different possibilities so we have written down three of them already so here we can say it is 2 to the power of 3 which is equals to 8. so we have written 3 then we can also have 0 one and one so we will say slides right instead of going right when it is like this so when it's in the middle it will have zero one and one so this way both of them are on so we only have to rotate a little bit earlier it was zero 0 1 we had to rotate a lot now it is 0 1 1 so we have to rotate a little bit so that is slight right so the same way we will have 1 1 and 0 so this will be slight left and then we will have zero zero and zero and then we will have one one and one so a lot of the times uh maybe one of them is not possible like for example one one one maybe it's not possible and zero zero zero maybe it's not possible if it doesn't go out but at some cases they can be possible so we can write for example stop and stop for these cases so we have now one two three four five six seven cases and the last one is one zero and one so again this should not be possible so we will write stop or we don't need to define it so here we have eight different possibilities that can happen so based on this we can move our robot to follow this line so it will rotate then it will go forward it will rotate a little bit and then it will keep going forward so this is the main idea so let's see how we can implement this to the drone so here we have the rotation of the drone so let's say that our drone is up here and it is moving forward and the camera is facing downwards so we are looking directly downwards now we are going to use a clip to actually attach a mirror so that it can look downwards this clip is available on thingiverse now if you cannot print it or you don't have a 3d printer you can always glue a mirror on the tele drone itself i have tried it with the glue gun and it works perfectly fine but a neater way a cleaner way is to use the thingiverse file and this file here is provided by works of clay so thanks to the guy who actually built this and you can see it can view from the top if you add emitter there you go i think this is the guy and yeah so you can print this it will print quite quickly and then you can use this to view downwards so this is what we are doing we are viewing downwards and what we will get is we will get an image so what we will do is we will split this image into three different parts so we will have left center and right so this will be exactly the same method of our line follower so like this line follower had these three what you call sensors the same way we will have three regions where we will detect our black line so when we are getting the image if the image is in the middle then we will say that go forward then if it's on the right then we will say rotate towards the right so the drone will rotate towards the right and then it will come back to this position and then when it goes a little bit here then it will move towards left and it will travel like this so this way it will follow the line completely so the idea is exactly the same as the line follower robot all we are doing is we are converting our image to three different sensor zones so right middle and left now because these are virtual zones what we can do is we can split them into more regions as well we can make them five we can make them ten but the problem is if we have more than three let's say we have five then we will have two to the power of five different combinations so that is a lot of combinations to handle so i would suggest to go with three and if you want to try it out you can try with five as well now this is part of rotation now because in a line follower robot we can only rotate so it is a bit easier and that should be enough for the line following but in a drone apart from rotation we can also translate so maybe the drone is not here maybe it's here then it's not actually viewing the line properly maybe it's like this or maybe it's like this so we have to make sure that the drone is always at the center and then when it's at the center then it will be able to tell when to rotate properly otherwise if it's here it will not be able to tell how to rotate or how much to rotate so we also have the problem of translation so this is the idea of translation that when the drone is moving at any given point we will try to find the center of our black line and where the drone is so if is if the image or the line detected is right in the center we will keep going forward if it's not if it's deviated we will add some translation back towards the center so the same thing if it's going this way let's say it is like this then we will reduce it back to this so at any given point we will measure so it will keep coming back like this so this is the idea of translation so let's get started with the implementation so the first thing we will do is to right click and go to python file and we will create the line for lower so we will open that up and here we are mostly going to use the opencv part so we will be doing image processing a lot of the times so what i will do is i will use my webcam and once everything is done we can implement it on the drone so the idea here is that we are going to lay down the a4 papers as our path because it is very simple to find the a4 papers and everyone has them so this will be a very easy way to create a path so we will be using a4 paper but you can use anything that you want one good thing about the a4 paper is that it is plain white so we can use color detection to actually detect our white color so let's get started with that so what we will do is we will write down here import cv2 then we will add our webcam so we will write cap is equals to cv2 dot video capture and we will put it as one you can put it as zero if you have only one webcam i have multiple so i'm using one then we will write while true and then we are going to write underscore image is equals to cap dot read and that should give us our image from our webcam and then we are going to write here cb2 dot uh weight key one and we also need to show so we will write cb2 dot i am show and we will say output and then image so this should give us our image and also we can resize it so that it is not too big so we can write here image is equals to cv2 dot resize and we will give in our image and then we will resize it to let's say 480 by 360. again you can use other values as well if you wish now the thing is when we will be using the the telodrone the image will be flipped vertically because we are using a mirror to actually find the image so it will be flipped so we need to flip it back so we can write here image is equals to cv2 dot flip and we want to flip our image in the vertical direction so that's how you can do it but for now we are going to remove this and later on when we are implementing it to the drone then we will add this again okay so by the way for now i have these images so i have created these simple images we are going to test on that so this represents the path the a4 paper and we have a straight and we have a curve so we can use the webcam and these images to actually try them out i will put these images on my website as well so you can download it from there by the way everything will be available on my website for free you can download the code and all the resources from there so the next step would be to find our path so to do that we are going to use the color method we will create a thresholding function thresholding and we will give it an image that we want to find the color from so we will copy this and we will write here thresholding and we are going to use the hsv color space to find our color now if you are not familiar with this again you can go back to my opencv course and you can check that out i highly recommend you do that but if you don't want to do that then just keep along and i will explain as much as i can so we are going to convert it into hsv so this will be our hsv image so we will write here cv2 dot cvt color and we will write that we want to change this image to hsv the reason we are converting it into hsv which is hue saturation value is because it's easier to find colors in that color space so we are converting from rgb to hsv actually bgr to hsp because opencv uses pgr so we will write here cv2.bgr sorry color to bgr color to color bgr to hsv so this is what we will use and then we will give in our limits so we have the hue minimum and hue maximum then we have the saturation minimum and saturation maximum and the value minimum and the value maximum so we have to input these values but how can we get these values well i have written down a script again this is from my opencv course so this script basically helps us find these six values so let me create a new file here color picker we will create a new file and we will paste it so this is the code again this will be available on my websites so what we can do is we can read it from our capture device so we can uncomment this and we will comment this so again this color picker you can use with the drone as well i have added the code for the drone as well so when you want to tune for the code for the drone then you will use this as well so right now we are using it for the images so let's run this so there we go so here you can see how i think it's flipped let me remove the flip so here because of the drone flip i have added the flip here as well so it is a little bit confusing so we can run it without that and there you go so now you can see my screen there's black and white so we need to find this white so what we have to do is we have to move these values and we have to keep the white color and we have to remove you can look over here you have to keep the white color and you have to remove the black there you go now it's removing and there you go now we are getting only the white so you can see here the white that was here is now gone then we can increase this it's getting better and we can play with this as well and there you go so these are the values that we got what we will do is we will stop this and then we are going to go and copy our color from here so we will copy all of this and we will go back to our line follower and here we are going to declare a new variable and we are going to call this the hsv values so here we will call it hsv values or let's say vals is equals to this so now we can use these values to actually create the minimum and the maximum so here we will call this the lower lower whites or let's just call it lower and then we will say numpy.array so here we have to import numpy as np so we will say np dot array and then we will add our hsv values then we have upper is equals to numpy dot array and then again we have to add our hsv values so we can write here hsv values at zero then hsv values at 1 and then we have hsv values at 2. so the same thing we will copy from here and we will paste here and these values are basically 3 4 and 5. so now if we want to create that image the mask image so we can write here mask is equals to cv2 dot in range and we will write hsv and we want to use the lower and the upper so this will be the image that will only contain the paths the white parts that we have and nothing else so we can return this return mask return mask and then we can go back here and we can get this mask mask is equals to this so what we can do now is actually let's call it image threshold so what we can do is we can display this image threshold so we can say path so let's run this so there you go so now only the white part is showing as you can see and if we go to the curved one you can see even the curved one uh the only part showing is the curved one so it's detecting only the white portions so here you can see setting the white portions and now for the white paper for a4 paper it will be a little bit different so you will might have to tune it again actually let me put down a paper and show you what it looks like so here you can see that this is the white paper the a4 paper now again as i've mentioned before once we are done with the code we will tune it on the drone from the camera of the drone so that it will have a little bit of different colors so we will tune it then but for now it looks good but as you can see my carpet is gray and it's detecting that a little bit as well so once we tune it it should be fine okay so now that we have the image of our path we still don't know where the center is or how much curve is present so to do that we need to convert this into a box and from that box we will get the center point so we will create a bounding box around this path and we will get the center point of it and based on that center point we can translate so if it's towards the right we will go back towards the center again if it's towards the left we will go back towards the center so we will try to keep it at the center so to do this we will create a new function and we will call it get contours and we will send in our image threshold and we will also send in an our actual image on which we want to draw the bounding box so let's create this function get contours and we will have image threshold and we will have image okay so the first thing we will do we will get the contours so we will use the function cv2 dot find contours now contours are basically edges so we are finding the edges in our image and based on those edges we will find the path or the bounding box so here we will write that we want to find it on our image again if you want to go into the detail of how this works you can check out my opencv course cv2 dot we are going to use the external method that will give us the outer region and cv2 dot chain approximation again you can use simple or none let's use none and what this will return is it will return contours con tours and it will return us in hierarchy now we are not interested in the hierarchy but because it's going to return us we are just going to unpack it over here okay so then we have to find the biggest contour from this because that will be our path we are assuming that the biggest contour or the biggest region you find will be our path so here we will say biggest is equals to by the way these contours are basically the dots or the points actually let me draw the contours so that you can see what this is let's remove this so let me draw so cv2 dot drop contours and we will give in our image we want to draw on that and then we will draw all the contours and this one minus one means you have to draw all of them and if it's just one or two it will draw the first one or the second one and so on then you have to define the color so let's put it as purple purple and let's define the thickness as seven so let's run this and see what happens we are getting an error let's see what that is uh see we do find contours something is missing i think um oh okay uh here it should not be image it should be image threshold so we need to find it from the image threshold that we just got the path from here let's run it again and there you go so now you can see that the contour you are getting the contour like this now the problem is that my computer is showing the white regions on the edges as well so that is a little bit problematic otherwise if i zoom in you can see it's getting the edges or let me try the other one there you go so now you can see it's detecting the path uh in the correct way so if i go here or here if i go back it will get the bigger one as well because i have white regions elsewhere but if it's zoomed in it should be fine so while we are using it with the drone it should be the background should be a different color so it will be easily detectable so now that we have these contours we need to get the biggest one so as you saw we can get the biggest one by writing biggest is equals to max of contours and what is the key that we want to use the key will be area so we will write cv2 dot contour contour area so this will give us the biggest contour the points and then we will convert all of these points because it will have lots of points we don't want that what we want is the bounding box the x y and the width and height so we will change all of these points into just a bounding box or a rectangle so we will say that x x y then width and height is equals to cv2 dot bounding rect and we want to change the biggest one so this will give us the x y width and height so we can use the rectangle method to draw that and we can also find the center of this so the center x will be x plus width divided by 2 and the center y will be y plus h divided by 2. so these will be our points and as we did earlier we can draw these points cb2 dot what do you call circle we will draw it on our image we want the center to be cx and cy we want the radius to be let's say 10 then we want the color to be let's say green and then we want the thickness to be cv2 dot filled so that should work what else and we can change this instead of drawing all of them we can only draw the biggest one so there you go so now we are getting the center as well so if we zoom in properly then you can see the center is here so if we are getting something like this we will say that our main center is here somewhere the center of the image so we have to go back until it's in the center so it will be like this so the drone will move like this so using this let's check the other one there you go you can see we are getting the center so using this method we can find the deviation and from that deviation we can tell the robot to go left or right so do we want to draw this yeah why not let's let's keep it there you can also draw the rectangle using these points cv2 dot rectangle you can use that to draw that so all we need from this function is the cx so we will return cx and we will go back here and we will get the cx value over here so now we have done the first part which is basically for translation so i can write here this is for translation okay now we need something for rotation so as we discussed the three sensor method we are going to use that so we will call this as get sensor output okay we will create a function called get sensor output we will send in our image and we will send in the number of sensors we want to use so the image will be the same image threshold and the number of sensors will be three so now you can change those as well but for now we are going to keep it at three let's copy this and we will create a new function here so now the idea is that we want to split our image into three different portions so if we have three sensors we want to split it into three different sections so we can use the numpy library because these are matrices we can use the numpy liability to actually split it so there is a very good function for that let's say our images are called images is equals to numpy dot horizontal split because we want to split it horizontally we will given our image which is the image threshold and then we are going to given how many numbers do we want so we want sensors which is three we want to split it three ways now one thing you have to make sure that when you are splitting the the size of the image here the width of the image should be divisible by three if it's not divisible by 3 it will not be able to split it into equal portions so you have to make sure of that so 480 is divisible so it should not be a problem so what we can do is to actually check if it's working we can print out cv2 dot i am show and we can write let's say we can create a for loop here for image or let's say i am in images we want to we want to show the image and we have to give a unique name so we can enumerate it enumerates and we can get the value of x so here for the name we can put the value of x we can convert it into a string and we can write here i am so every time it will loop it will create a new image so we will have three images to show so let's run this and see what happens there you go so now you can see uh so this is our main image and then we are getting these three images uh converted so let's say that let me change this screen let's say that it is in the center or let's say it's on the right uh it's on the left a little bit so then the right one will be zero the left one will be one the middle one will be uh sorry this is left one and this is the middle one so it will be like this so right now our sensor value is 1 1 0 so right now here we have a lot of values if we had very little values maybe something like this then we can neglect it we can say if it's more than twenty percent then we will count it as one and if we have it like this then it's one one one so it will go forward if we have it like this it will be 0 1 0 sorry 0 1 1 and so on so this way now we can check each of these images so we need to count how many pixels here are white and based on that we can say whether it's on or off so to do the counting we are going to write here pixel count is equals to cv2 dot count non-zero from our image and then we can write here that if our pixel count is greater than the threshold value multiplied by total pixels total pixels so what is this threshold value well we can define it up here let's say this is 20 so if it's more than 20 of the total pixels then it means it is one so we can multiply it here and uh wait what happened here what is total pixels so total pixels is basically total pixels is basically so this will be image dot shape we have to take the width so the width is element number one divided by the sensors so for example our width was 480 it will divide it by three sensors so it will divide it by three and then it will give us the width of our single image so the one that we have split and then we will multiply it with the image dot shape at zero so this is the height so we are multiplying the width by height and that will give us the total number of pixels but because we are splitting it up into three pieces we are dividing the width by three first so this will give us the total pixels and then we can check if the total pixel count multiplied by threshold is greater than is less than the pixel count then it means it is one so we will say that send out so this will be our let's say sensor values dot append we are going to append one so we can declare it here our sensor out so these will be the values so if the sensor is 1 1 0 it will have the value 1 1 0. so based on every sensor is it on or off so here will be else we will say sent out dot append at zero so this is the idea and we can show this for now but later on we can remove this so let me print this out and show you how it looks like so print let's say our sensor out and we can return our sensor out and we can unpack it here like this okay so let's run this and see if it works fine what are the values that we get there you go so this one is the middle one yeah this one is the first one and this one is the third one so let me show you how it will work so here okay what happened so i think it was unable to find the biggest contour that's why it gave an error so we can fix that later on we can say do all the calculation if something has been found otherwise don't do the calculations so here you can see that at this point it should be 0 1 1. so this is 0 this is 1 this is 1. and then if i go here it is one one zero if i zoom in or if i zoom out then the middle one no yeah so the middle one right now is one the rest of them are zero zero so i think this is clear enough so here you can see one one zero then one one one and then zero one one so if twenty percent of the pixels are present it will consider that as one so we do not need to display this all we need is the sensor values of zeros and ones so once we have that the last thing we can do is we can send out all these values to our drone so how can we do that let's have a look so we will create a new function and we will call it send send commands and we will say that we are going to given the sensor out and we are also going to give the cx so these are the two values this one is for translation we got the cx and this one is for rotation so we get the sensor out so based on these values we can rotate or translate our drone so here we are going to write that our function is send commands and the first thing we will do here is we are going to do the translation so let's look at translation first trans translation so the translation is actually very simple so we will get the value of left and right we will call its lr so this is the left and right value so this will be basically our cx minus the width divided by 2. so the width is not declared so we need to declare the width so we can write here width and height is equals to 480 and 360. so we can write that here as well width and the height okay so width divided by two and we can introduce a sensitivity elements so we don't want to send in the exact difference we can divide it by something to make the the sensitivity a little bit diluted so this value will control the sensitivity so i can write here sensitivity stay with c and we can copy it here so if the sensitivity the number is high if number is high less sensitive okay so that is the idea and then we can send this value to our drone so how can we do that now we need to make sure of one thing uh which is basically that we don't want to give it a lot of value so we don't want it to give a very high speed so maybe the speed is coming to 200 or maybe it's coming to 50 or 70. so that is a very high value and it will be very fast so we need to restrict our translation to a certain value so we can use the numpy function here again so we can say that our lr is equals to integer numpy dot clip and we will clip our lr from -10 to 10 so we don't want to go above that and the last thing we want to do is we want to add a value where the translation will be zero because at any given time there will be some value of deviation but what we want to do is we want to give it some um room for error so that it's not always moving so we can write it that if our lr is less than 2 and our lr is greater than 2 which means it's between minus 2 and 2 then our lr is equals to 0. so at that point it's not going to do anything so you can write it here and you can combine all of this and now we can simply send it by saying me dot send underscore rc underscore control and we can send in our left right and the rest we can keep it at zero for now and we can comment this now the second part is your rotation so as we discussed in the rotation we will have these sensor values so whether the sensor is on or off we have to check so we will say that if our sensor out is equals to 1 0 and 0 then we will have a curve the curve value is equals to let's say 30. so the idea here is that if the value is 1 0 0 then it should turn more if it's 1 1 0 it should turn less so we can call them as weights so our weights are basically let's say -25 then minus 15 then 0 then 15 and then 25. so the idea here is that if we have 0 1 0 which is the center then this is the weight so you don't have any curve zero curve then if the value is 1 1 0 then the curve will be minus 15. if the curve is 1 0 0 then the value will be minus 25 on the other side if it's 0 1 1 then it will be 15 and if it's 0 0 1 it will be 25 so we can take the weights from here so we can go down where is it here and instead of 30 we can write weights and this is the very harsh on the left so this will be the first one so it will be zero so now we can copy this and we can write the other statements so here we will write else if the send out let let's put it like this so it's facing the same line and here it will be one one zero and the weight will be one then we can copy this line several times uh how many times do we okay if we need we will remove so it will be one one one zero then it will be zero one zero then it will be 0 1 1 then it will be 0 0 and 1 then we can also have 0 0 0 and we can also have one one one and we can also have one zero one so in the last three cases we will not do anything to the curve or we can put it as 0 to not go any further so it is up to us so 0 1 2 2 will be 0 so here this value is 0 1 and 2 so this is 0 so that's why we are using 0 over here so if it's any of these it will be 0 and then we have one we can change it to two three and four so these are our weights do we need to change anything else um doesn't seem so so the only thing we have to do we have to declare the curve so we will declare it here that the curve is equals to zero and we can write here global curve so that should be fine and the last thing we have to do again we have to send it so here in the forward uh speed we will have a constant speed so we will have forward speed and for the curve we will write here curve so this forward speed we can define so i recommend to keep it at a slow speed don't go too fast but as you keep improving then you can keep increasing your speed so that is pretty much it so let's try it out and see what happens so what you can do is you can try out one part of it so you can try out only translation and then you can try out only rotation and then if you want to tune it you can tune it and then you can open both of them and work them at the same time so here if you want to just run the translation you will remove the curve you will make it zero and vice versa you can make this zero as well and use only the curve so the last thing we have to add all the things for our drone so we will go to the image capture and in the image capture we will copy all of this code from the top and we will paste it here we are repeating import cv2 so we will remove that and then oh did we repeat anything else doesn't seem so and we also need to take off so let's take it off over here so me dot take off and then we have to write over here so in the image capture we are going to copy it from here and then we will go back and we will paste it here so we can remove this and we can put this and we can remove this as well so this will be the flipped one so that our image is straight so that is pretty much it so let's go ahead and test it out so this is basically the drone and then this is the clip or the mirror that we have and what we can do is we can simply snap it here so it's a pretty good and light design now it is a bit fragile so just take care not to break it so this is how you can attach it and now you can see the camera from right here so there you go as you can see the camera and now it will view down so let's run it and see how it works now the thing is that it might actually view by the way i got this from a store and let me let me check the size for this of this is about 25 by 25 mm yeah so it's 25 by 25 so you can get this or you can cut it yourself now the first time i tried this i cut i took a mirror i cut it myself with a cutter and i just glued it over here with an angle i just turned on the camera i looked at the angle i held it while the hot glue was drying and it worked out fine as well so you don't have to get this fancy stuff if you don't have a 3d printer but if you do then this is much better as you can remove it and you can add it very easily so let's run this now and see if we can find the the angle correct so i will remove the take off for now so it doesn't fly on me and let's connect it and see what happens so there we go so now we are getting the uh the live feed from our drone and yeah as i was mentioning before you might get get this problem so as you can see on the very edge you will see there is a black uh area and this is because of the mirror so what you can do is if you press it hard enough so you can see this is the mirror showing you now if you press it hard enough towards the upper side then it will stay but if it doesn't stay what you can do is you can take a scotch tape and then you can tighten it from there so something like this [Music] and you can pull it back you can put it here and you can pull it back and then tape it down here so now it should not give you uh any of the edge that you were getting before so let's try it out so there you go so there's no edge now and it is fine so the first thing we have to do we have to find the color so again we are getting this error for the biggest contour so what we can do is we can go down and the the problem is where max it cannot find max if it didn't detect anything so let's go to the max so here we can write if the length of contours is not equals to zero so we can write that hopefully this will work well and we can define the cx as 0 so that if it has to it can return just this okay let's try it out again okay so now what we have to do we have to first fall find the color so i will lay down the a4 papers and then we will find the color of using the drone camera so let's go to the color picker and we will run this okay we need to change it to the drone camera instead of this so we will remove that and let's run it again so there you go okay again it's not flipped so we need to flip it here let's run it again there you go so now i can move this to find the optimum value for the whites so we want to keep the white area so remember that there you go it's getting nice and good you can add a little bit of this there you go and then maybe a little bit of this okay so these are the values that we have found and it should be fine so we can stop it we can copy it from here and we can go to the line follower and we can paste it in the hsv over here if we run this now there you go so now we are getting our path this is the center of our path and as you can see we have a little bit of right and left and then we have uh the forward speed there is no curve but if we rotate it if we rotate it we should get some curve so now you can see that we have a curve if i go on the other side you can see the curve value is changing the last one over here 15. so if i go back and i think the battery died okay anyways the idea is this that everything is working fine all we have to do now is to try it out and we will take off and run it and see what happens [Music] [Music] you