what's going on everyone welcome to part two of our brand new project which is a live emoji and if you don't know about this project let me show you a quick intro so it is predicting smile good luck nope hello smile so how's it pretty interesting so we have already covered our part one which is the data collection part if you haven't seen that i would recommend you to go and watch that but it is still okay if you want to follow along with this part uh you can actually watch that part after watching this video so let's start this video quickly and in this video let me tell you we are going to finalize this project which means that we already have collected our data and now we just have to load that trade data train that data uh and then run the inference on that so we are going to create two python scripts in this video first is for training and the other one is for doing the inferencing so let's start shall we all right so here we have our data collection part uh so let's create our first script which is data training i will just call this data training dot py and we will also need another script which is inference dot py we are going to code this later on but first we will focus on the data in dot py let's open this with whatever text editor you use let me just take it to my this screen all right so here we are now let's do that uh before starting this video we actually need some data actually let's just run the data collection dot pi file two or three times and collect some data so i'll just open the terminal in the same directory and i'll say python data collection dot pi and i'll just run it and let's give let's say i want the i want to give the data for hello emoji so i'll just say this alright so we have collected data for hello let's do for good luck let's give the name good luck and let's just give for let's say nope all right so i guess this should be enough let's close this terminal now we have our good luck dot npy hello dot npy and nope dot npy we have three data over here now let's start so first of all we are going to use os module for uh getting the files in the uh in this script and then i guess we also going to use numpy numpy as np let's just import cv2 also in case we need that i guess that's all we need for now and now let's do this so first of all we are going to search for all these files which is uh good luck dot npi hello dot and py node dot antipi and npy in the same directory so for that we have the function called list directory in the os module let's just see for this uh return and we'll say for inos dot lister print i so let's just open the terminal in the same directory and run this script which is called python data collection sorry training so you see that this is returning data training all the files in the same directory but we are interesting interested in only the files which end with the extension.npy so it is very easy to filter out those files will say if i dot split at dot and this is going to return a list and we are interested in the last element of the list and that if that last element is dot uh npy if it is npy then we're gonna say that uh this is the file we are interested in right so what to do with that file of course we want to load it by using the numpy so how to load it now there are some ways of loading the file first one uh we will uh create our x okay just leave it we'll just say init is in it we'll just initialize it as false so initially we are going to initialize the x and y uh data variable or to create the array once the initialization is completed we are going to just concatenate the new files the upcoming files into that previous loaded file let me just quickly show you what do i mean by that so first of all we're gonna say we're gonna say if not is initialized we'll say is initialized is equal to true now we are going to initialize it so we gonna need x array which is going to be our input data so we will be gonna say x is goes to np dot load that file particular file very easy nothing too complicated in it now we also need y so what will be the y y will be uh obviously going to be the label associated to that data for example if we are uh if we are talking of nope dot npy or good luck dot npy the all the data inside goodluck.npy will going to be my x data which will be input to my model and for that a particular input data i want my model to predict something and what i want my model to predict is good luck obviously you agree with that or not so we are going to have that uh data inside the y variable so how to do that we are very easy we'll say np dot array now how much the size of the data the same size we want of y so we'll create a variable let's say size initially we will set it to -1 then we will initialize it over here size will be x dot shape and we are interested in the zeroth part meaning the number of rows we have right so the size will be uh x dot shape zero and now in this we will create a array of uh i dot split let me just first write that code i dot split at zero and the zeroth uh part which means we are going to split at uh dot which is good luck dot np y now the list will contain good luck and npy we are interested in the zeroth element which is good luck right now this is a list of only one variable we are going to multiply it by the size whatever the size of x will be the same size of y will be okay i guess this is enough yeah so once the initialization is completed meaning we have loaded first loaded our first data sample now we are just going to concatenate uh concatenate in this uh x and y which is which is going to be the same process after all we'll say uh np dot concatenate concatenate uh x concatenate x and the new data we are getting which is obviously np dot load i yeah do you agree with that or not similarly we'll say y is equals to np dot concatenate con catenate y comma uh np dot array the same thing over here i hope this should work if we get any error we'll just resolve it yeah uh i guess everything looks fine over here now we'll just to make sure everything is loaded perfectly we'll just print the shape of our x and y so we'll say print extra shape y dot shape let's just save this and quickly run this we got an error uh line five integer scalar let me just see yeah so uh this need to be uh in passed in tuple in np dot concatenate we here we are getting the error yeah let's run this once again we have the size is 300 rows and one zero two zero column of our x and three hundred rows of y uh let's do one thing let's reshape this to uh minus one comma 1 and similarly dot reshape this 2 minus 1 comma 1 let's run this once again uh np dot array wait a minute this have to be over here let's run this once again yeah so everything looks fine over here um so we had our okay so over here we have loaded the data everything looks fine let's just try printing our data x and printing our data y let's see how the data looks like so x looks good over here right we can directly send x as the input but y doesn't look good over here you see that we can't directly send the y as it is this is string but the model needs the data to be in integer or float the model can't work with string so we need some way to encode this convert this string to uh some sort of integer if we can which is very easy to do we will just create a list of labels labels over here let's just call this label and let's just create a dictionary also you will see why we need that and let's create a flag variable c to zero so what this dictionary will do every word or data sample which means good luck or nope and hello so every word will have a unique uh integer associated to it so for example hello will be zero and nope will be one and good luck will be one uh two and later on we will just convert those strings to to their particular identity unique uh integer associated to them and if required we can just re-encode them when we when we are making the prediction from our model so that's why we are using this label over here so i hope everything makes sense to you let's just come over here here we'll say label dot append what we want to append just try to pause the video and make sure you can guess what you want to open obviously we want to append this so we want to append this and similarly let's do the same with dictionary dictionary this particular word will have c variable and we will increment the c 2 plus 1 yeah that's it now let's try printing our x sorry dictionary and similarly label see if it is doing or not yeah good luck have zero nope has one and hello has to and this label is also the same okay so now we have the dictionary over here and now we need a way to convert this particular where is this yeah we want to convert this y to the integer so which is very easy to do we'll just iterate in uh for i in range the length the size of our white data so we'll say y dot shape 0 and now we want to convert that particular y so we will say y and that i throw at zeroth column obviously which will be equals to dictionary uh the key to that dictionary will be y i zero okay so let's try printing before uh encoding and after encoding yeah we have done it successfully you see that but the only problem i see that it is a string we want it to convert to integer which is very easy we'll say y is equals to np dot array y and we'll just change the data type to int 32 or whatever you want let's save this and run this once again yeah that's it yeah i think everything looks fine but still there is one thing we want to do what is that thing uh let me just explain you real quick over here let's just do the commenting over here okay so you see that we have encoded let's say hello to 0 and let's say nope to 1 but in if we talk of integer 1 is greater than 0 yeah you agree or not but nope is not greater than hello in any way these are two different words having two different meanings but you can't compare them with any quantity so we can't say that nope is one and hello is zero when we train our model for our model this might have uh more precedence or this might have the lower precedence or vice versa but that we really don't want that so how what can we do what can we do to solve this problem very easy you see when we are pointing to zero we will create a 2d array when uh the the index is 0 the value is 0 will say 1 comma 0 when the 1 is there we will turn the first index to 1 like that so how to do that very easy there is a function in tensorflow dot keras dot utils import to categorical still it doesn't make sense just just wait a minute i'll just show you what do i mean by that so over here let's just print the y dot shape let's get rid of these print statements okay so now over here we'll say y is equals to 2 categorical y yeah that's it and now we'll again print the y dot shape like this let's run this you see what did i meant by that okay so this is what i was talking about you see that initially the y shape was just single column and later on we converted that single column to three columns yeah do you see that what did i meant by that okay let's just try printing the before y and after y let's save this and run this once again okay so you see that over here when we have the index as 2 then let's talk about last so the second index is turned 1 and everything else is 0. similarly when the index is 1 the first index is turned 1 and everything else is 0 0 when the value was 0 first zeroth index was one and everything else was zero so that is what i was talking of now everything looks fine we can just directly send our data to the model and we can create the model obviously first of all then we can send this our x and y to our model and we can clean our model so what do we gonna need for making our model obviously we're gonna need some layers let's quickly import them from keras dot layers import input and we're gonna need a dense layer that's it and also we're gonna use functional api from keras dot models import model that's it let's create very simple model since the data is not pretty big and the work is also very simple so we're gonna just need very simple uh model so we're gonna say uh input first layer will be the input layer ip we'll call this ip input it's gonna need the shape the or the shape is going to be x dot shape and it doesn't have to do anything with with the number of rows we have it only have to do with the number of columns we have so x dot shape one let's create the middle layers will say uh what do we call it let's call this m dense and number of units let's say 512 activation let's give it a activation of value let's connect it with the previous one which is ip let's copy this and it have to be connected with the previous one which is m and change to 256 neurons and let's create the output layer which is going to be again dense layer the number of units over here we have this is uh important just makes just a 10 just look at over here so you see that in the output we want three neurons meaning uh output neuron which is making prediction associated with the zeroth index first index and the third index so over here what do you think we should pass obviously why dot shape this time the first winning the number of columns we have the activation can you guess which activation gonna use soft max why softnext because sorry so because we want every neuron having the probability associated to it probability of being zeroth class probability of being first class probability of being second class and when all of these three summed up they're gonna uh sum up to one so that's what we are interested in and we're gonna connect this output with the previous uh middle layer which is m okay now we'll finally create the model we'll say model is goes to model inputs will be uh ip and outputs will be op very simple and straightforward model nothing much have to do in it now we'll compile our model quickly with the optimizer as rms prop let's just leave it over here and loss will be categorical cross entropy my categorical cross entropy simple because we have the categorical data and finally uh we have to give the matrix we'll just give accuracy acc uh yeah now we'll just fit our model model dot fit on x and y and number of epochs let's just give 50 epoch and yeah let's just save it and see if everything is working fine or not let's just train the model and fingers crossed and see if everything works fine or not yeah yeah everything is working pretty fine and look at the accuracy and the loss it is reducing yeah everything looks good but there is one thing i want to talk about let's just you know i'll just cut this out for a moment model dot fit statement i'll just comment it out and i want to show you something if i print my y let's just print from here so you see that uh when we are training our model first of all it is trained on the zeroth class continuously than the first class continuously than the last class continuously so i i think that this creates some problem while training our model our model first learned about hello hello hello hello then our model learned about nope nope nope and then our model learned about bye bye bye bye and so on so this is just creating some type of cluster type of thing i would say so it would be nice if we just shuffle our data you know if we shuffle our data then it would be our model would be able to predict the things very effectively so how can we do that very very very easy and tricky uh method i have so we'll just uh create the copy of our x and y we'll just say x dot copy extra copy and similarly i'll just create my new and i'll say why so we will have a uh let's just create a counter also you'll see why we need that so let's just create a cnt yeah so we'll create a cnt uh this is going to be uh np dot arrange x dot shape meaning the number of rows in the x will be equal to the number of rows in y so you can use whatever over here x dot shape zeros the number of rows in x so what this function returned let me just show you quickly real quick numpy as np and what i am doing i am saying np dot arrange let's say 50 so you see that this function returns a list of uh a list of all the counting from 0 all the way to 49 so that's what we are interested in we are storing in cnt and then later on we are saying np dot random dot shuffle that's cnt so this uh function will shuffle function will going to shuffle all these values from 0 up to 49 shuffle all the values from their places now over here we'll just iterate in for i in c and t we'll say x new at that counter position meaning at zeroth position will be x i that shuffled particular shuffled value yeah i hope you are getting what i what i was doing over here why new again at that counter position because we don't want to lose our data by new at that counter position we will say y i and then we will update the counter i really want that python please add the support of counter plus plus directly uh doing such in cc plus but we have to do this counter plus one yeah so that is what i was talking of so now if i print my y and print my y new then you will see that we have our shuffled data let's just quickly run this okay so now you over here you see that uh this was my previous y which is the continuous and now we have the random samples yeah that is what i wanted to be now our model will be pretty much very accurate to predict the things let's just save our model real quick by saying model dot safe so that we can load in the inference part we'll call it model dot h5 and one thing i really want to say is np dot lib oh sorry np dot save you will see why we need that np dot save labels dot npy and the array will be entry dot array label you will see why we are saving these labels but you should you might know that you might you can guess why we are saving this labels.npy let's just save this and run this real quick all right so the training is completed the loss is this the accuracy is this and let's see if yeah model.h5 is saved labels.npy is saved that is what we wanted now let's quickly load this inference dot pi file i also want this data collection pi file i now at this point i would recommend you to open this uh part one and try understanding this because i'm going to use the same code from there because i don't want to repeat myself writing again and again the same code you can watch the explanation of this part over here because already this video is being so long uh just leave it i'll explain you don't worry about that okay so how to start from here look we have the model so we'll just import the cv2 we're gonna need that i know we're gonna also need numpy s and p we also going to use media pi as mp and we want to load the model obviously from keras start models import load model load model yeah that's it let's first quickly load the model so we'll say and model equals load model and pass the name and load the labels also so we'll say labels equals to i'll just call it label uh np dot load labels dot npy yeah that's it now as i said we are first going to use this the same code uh i'll just quickly copy this don't worry i will explain to you so what is this this is uh again if you are if you want if you want line by line character by character explanation of this code please go ahead and watch the part one i'm just going to be real quick explaining you this so over here we are using the holistic solution for uh the prediction of the post prediction given to us by media pipe very easy nothing too complicated in it and over here we are uh creating the object of this holistic class from this holistic uh the type of reference we made over here this is the hands solution provided by media pi for the landmark prediction of hand this is for uh creating the visuals the solution provided to us by media wipe again now i will just going to use this cv2 dot video capture to capture from my webcam directly save this and let's just do this magic over here i'll just copy all of this okay so this is the true loop this is my true loop this is my list lst just list and over here i am reading from my camera which is i have created over here i have flipped the frame i don't want that mirror look and then i am going to use this hollis object which i have created initialized from this holistic class this hollis object can process the frame but that need to be in rgb format so i have used cvt color to do that since cv2 reads in vgr format now i am checking if if this hollis object or media pipe is able to find any face in the frame which is checked by using this line then follow along with this code and then else do nothing so if there is someone in the frame just just uh iterating it so we can iterate result dot face landmark dot landmark this is going to return a list and i am iterating it in it and then storing my x and y value inside the list not directly but by subtracting from the reference point i have already explained in part one you can go and watch that part one again i am checking if the left hand is in the frame just store it in the same way we have stored for face otherwise just store zero in place of that similarly if right hand is in the frame just store it by subtracting from the reference point again why we are subtracting i have explained in part one you and watch that part one otherwise just store zero zero instead of that okay so we don't need this line over here worry about that we also don't need this line over here that's it and over here we are just using this drawing over here which we have created there we are using this to draw on frame what we want to draw is are the predictions the result framework and then over here we are uh we can also remove this line we don't need that yeah that's it that's all we wanted um now it's just matter of converting that lsd to our array so we'll say lsa is goes to np dot array lst and we're gonna reshape it to 2d array since this is a 1d list so we to reshape we'll say 1 comma minus 1. if you want to know what this line mean np dot reshape 1 comma minus one minus com one comma one you can watch my previous video i have explained in that i will link that in i button so this is my lsd now it is just matter of doing model dot predict model dot predict lsd that's it let's try printing the predictions over here though it will not going to make any sense we'll just use the labels which we have saved over there we are going to use labels to convert that to human readable thing so we'll say python inference inference dot pi hope we get no error yeah we got error data size o okay so we want to remove this v since we have copied from the previous script obviously we are going to face some error yeah it is running perfectly fine and over here we are getting the predictions from our uh model let's escape it to get the max prediction we are going to use a function given to us by numpy np.org is very easy and straightforward nothing to explain so this is going to be my maximum uh index with the prediction and i can uh convert that to to a string by using my label list which label array which i have saved from my previous script right if you remember so yeah this prediction will contain that let's run this now so yeah this is returning good luck obviously it have to return anything so let's try giving hello yeah it is returning hello let's give good luck good luck let's give no no everything is working fine it's just let's just do one thing let's use the put text to put the prediction on the frame instead of reading from the terminal let's copy and save label we'll just use directly prediction prediction let's save this now and run this hello good luck nope yeah you can add as many expressions you want to add okay so yeah you can add whatever expression you want to add let's just why won't we do that so again i'll just add new expression let's say python data collection this time i'm going to add the expression for smile okay so we have added the expression let's run python data training dot pi let's run this yeah i guess we're gonna get an error yeah so you see that we are getting the error why we are getting the error because over here we have labels dot npy this is not our data this is a labels dot npy so somehow we need a way to ignore this label sort mpy which is not too much difficult to do so we'll just say we'll come in data training part and we're gonna say if the file is npy and the file is not zeroth index obviously and the file is not uh labels save this and run this now so it is going to do the training part yeah training is completed let's run the inference so it is predicting smile good luck nope hello smile everything is working pretty fine and i would like to end this video over here if you still have any queries any doubt regarding this project you can watch direct you can directly ask me in from down comment section if i know answer for that i will reply if anybody else no answer for that please reply to that comment made by that person over there so that's it for this video if you really enjoying my work please make sure to give this video a thumb up and stay tuned for the more upcoming videos because i have planned so amazing content for you similar to this so that's it bye i will catch you in the next video