hi this video is a part of deep learning with tensorflow playlist in this playlist i quickly explain the logic and implementation of deep learning algorithms and application using tensorflow note that i do not quote along rather i make use of pen for explaining the tricky parts of code so that you people can read the notebooks and re-implement these algorithms yourself code files for each video will be given it's in the description so i'm expecting my audience to be well aware of deep learning algorithms and i'll be just implementing them using tensorflow and i'm expecting that you people just need hints for these implementations in this video i will talk about mri image segmentation using unit mri images are basically 3d images using 3d volumes known as voxel what is generally segmentation what are those mri images and what are those images specifications how do they look and what is unit architecture and what is basically dissimilarity metric that we will be using in this segmentation task and how will we code all of this in python let's jump to the whiteboard and the notebook to understand all this okay so we will begin with the task so mainly our task is to find regions of brain images containing tumor now these brain images are what we call mri okay so what is basically segmentation now in segmentation we classify each of the pixel so this is like classifying each pixel okay but if we look at it what is classification generally classification is given an input predict the output so in simple classification we have a single output but in segmentation we have output per pixel so for each pixel we have separate output so segmentation is more difficult than simple classification okay once we understand this basic concept then let's look at the mri images usually images contain this height this width and these are their channels so these are a three dimensional image usually so if we have the image which is black and white so it means the channel is equal to only one so it will be kind of a 2d image uh we will be there there will be no information of channel but if the image is colorful then the channels will be basically three one is red then blue then green so we'll be having total three different uh height and width values uh for red and green and blue okay then in color image what we see is basically a combination of all these red green and blue intensities i hope that you people know about these images and these basic concepts okay once you have this understanding then let's come to this mri image what this mri image will contain is this x y z and this fourth dimension first of all this x and y and z are not simple image dimensions having height and width and channel this is different because our brain is is in three dimensions so it's a volume so what we see is that this brain this this the mri is having uh this brain uh captured in in a in this volume having x axis value y axis value and z axis value so this is what we call 3d volume and a technical term for that is voxel so we will be having this 3d volume information so that where we are where at which point in this 3d mri image in this mri image which point are we looking and this fourth dimension will be actually information about different sequences uh the zero is for flare then t1 w t1 gd t2 t2w these are technical terms in medicine so those who are in medicine so if you are if you are interested in this subject then of course you must know these four different values but our main task is that given this mri image given this mri image we want to pixel we want to predict uh the tumor intensity and tumor for each of x y z for each of point in the 3d mri volume in 3d mri image in our brain so we want to find the points in our brain which contain tumor which contain this disease okay so the output for training purpose will be having this xyz values a 3d array where uh the the array the 3d array will contain the value of 0 1 or 2 or 3 this integer value where 0 is just background one means edema this is one condition then a two means a non-enhancing tumor and three means enhancing tumor so this is what we will basically see so this xyz will be having integer value for each voxel for each of the voxel means for each of x and y and x y and z uh point okay what does that mean it means we can visualize each of the voxel with the color if consider this is a first voxel second voxel third voxel this wax will mean xyz value and in this voxel we have this edema uh so in this voxel we will be having this value of one and for non-enhancing it will be zero and zero so in a way uh this can be captured uh as a red green and blue so in this case the red is one the green is zero the blue is zero so somehow what i'm trying to say is that uh what is the condition for each voxel we will be able to see that uh using these colors and the color red green or blue will represent the class of tumor that is present and at the end we will be predicting these class of tumors using these voxels as an input okay once we are clear here then i want to talk one more thing when it comes to visualization so if you want to visualize these 3d images 3d mri images so in technical terms in in medical terms and doctors know about that so of course uh there are different planes so consider three planes one is the sagittal and the coronal and transverse so this is a representation of those plans i hope so it is more better seen here so one of the so so it's y z plan then it's y x plane coronal and x z plane this transverse is x z plane once you know about these planes these labels these three different labels and their color that we will be showing you in this voxels and each of this voxel if you know all of this still now then perfect we are in the perfect shape to go to move forward let's move forward and let's look at this another thing what we are calling generate sub volumes so what we will do is that we won't give a complete mri image for training because a complete mri image is very very huge and it's computationally uh and memory uh memory wise very expensive task so if you're not having that much of a ram then we then we can go out of resources and that is what i'll be showing you inside the notebook uh that when we try to give a complete volume what happens uh the notebook that that sell in notebook the resources are completely exhausted but uh once we we divide we take random sub volumes we take random sub volumes and in those sub volumes what we do is we standardize data first of all we are having we make the mean zero and standard deviation one uh once we do that for each of the sub volume i'll be also showing that in in in notebook then we will apply this unit uh model for segmentation purpose unit architecture somehow looks like that in which we go downward to level four level one two level two level three level four where we are doing convolutions and once we reach there here then we go up where we legal wall so along with deacon evolution we also uh share we also uh give this information from each of the level share this information for from each of the level uh to its corresponding uh up level so this is how it will work i'll show how how we have how we will encode it in python from scratch uh but few things that you need to know one is that we are dealing with uh four levels here and in each level we are having this uh convolutional on three cross tree after that we apply relu uh so this is within one level these three things are happening and then this red line shows the pooling max pooling of two cross two and then uh this uh con and booting happens again and again and again after that then what we do is uh we do up sampling and in up sampling what we do is first of all we up convolve so this is deconvolving it to make the size uh bigger and then what we do is we then again go up for each of the level but along with going up we share these level information which is the output of level three we also uh share it here so i'll be uh i'll be just coding this from scratch and you'll be having a better idea there once we code it and we predict the output given the input image that uh that given the input volume instead sub volume if you have a if you have a good ram and you can you can use this whole volume it's perfect but uh generating sub volumes is also a good idea given we have limited resources then what we will do is we will look at the error metric what we want to minimize we want perfect segmentation it means suppose we have uh this prediction the a is prediction suppose a is prediction and b is actual actual values at each pixel so what we are interested in is that we want this similarity between these two uh between these two images contain consider these images so this is what we are calling dice similarity coefficient how will we find it we will find the intersection between them and we will divide it by this a and b once we do that then this will give us an integer value uh this this this will work first of all with integer values a and b both must have integer values but we are dealing here with the probabilities because this unit architecture will output uh probability so how to how to like calculate this die similarity uh for probabilities i'll be also showing that but uh let's just uh let me just show you this formula for simple die similarity and a coefficient where this is the prediction part and this is uh basically the actual values and we multiply it this is kind of encoding this a intersection b part and this is again prediction and uh this b and this is kind of a denominator denominator part okay for multiple classes we have to just divide it by three and dc one two and three add it and they kind of take mean for each of the class once we do that then again it is good for intuition but contains zero or one uh and or model outwards probability so we will use soft dice similarity coefficient and what is it i'll show the formula for that and you can look at it uh inside google then we will even wait our model uh these are two important evaluation matrix sensitivity and specificity uh i'm expecting that you know about that uh if you don't know about it so then it is basically true positive divided by true positive plus false negative specificity is true negative divided by two negative plus false positives uh now true positive false negatives i'm expecting that you know it because these are very very basics of classification evaluation matrix and these are also used in precision and recall and confusion matrix and all of all of the details i'm expecting that you know that okay once we know all of this then we are good to go we can directly jump to a notebook but let me revise what i just said so first of all segmentation task i explained this segmentation task where we are predicting each of the pixel then i explain this input image that we are having and this voxel uh the each point in this volume so these are this is actually not an image this is kind of a volume that we are giving so this is like volume class volume segmentation then uh we have talked about the output or where for each voxel we are having either edema non-enhancing tumor or enhancing tumor then we could visualize those those uh edema and non-enhancing and enhancing tumor using rgb and we will be also looking at the voxel uh at each of the volume from different sections so these are different sections of the 3d volume uh which doctors know and are interested in usually and then how will we train the whole model we will generate these sub volumes we'll stand right standardize the data we will we will first of all draw this unit model uh this will define this unit model and compile it after that we will just run it to predict after prediction we will define this error metric with dice similarity coefficient once we have that then we will finally evaluate it using sensitivity and specificity let's jump to the notebook so we have this segmentation task at hand first of all we will just import all the libraries uh and all the uh relevant uh layers and all the things that we will be needing then then i have explained this mri dataset here uh this is where we are we'll be exploring this data set so this first section is all about exploring the data set and this is i have explained in detail so we have data stored in our directory then we load the data and this is the shape where 242 4155 is kind of a volume and these four are different sequences and this is these are the labels uh for each of the voxel okay in this volume then uh these colors are actually corresponding to enema and all of the things that i've already explained then what we will do is we will uh normalize it using cv2 okay once actually we don't need that cv2 because this is generally uh min max normalization uh just subtracting the minimum and then dividing by max minus main so that all the values are in between zero one okay once we do that then what we will do is uh let me show you uh then we will generate different planes as i have already explained this is like generating this coronal plane uh we are adding this another one axis in this labeled image so this is like i'm just creating this labeled image after having this labeled image then what we are doing is how do we create this labeled image uh we have actually this uh this um uh red rgb edema non-enhancing keyboard and enhancing tumor labels here and we just append it with the with the with the original image so we have this three another dimension with this image 242 4155 this volume basically so this is the volume 242 4155 and these three are actually the classes uh and then what we will do is we will add one another dimension to it why do we add that because we want different planes so for coronal plane we will transpose and readjust this basic uh basic dimensions into this new dimension and for a transversal plane we we change this into this new dimension and for our sagittal plane we change it to this new dimension once we have all of these three dimensions then we will look at uh the the input data with labels so it this this is the coronal plane this is the transversal plane and this is the sagittal plane of the brain containing uh this disease containing this red rgb uh and then r means something and g means something green means something red means something and blue means something as i have already explained here and uh red is it demo green is non-enhancing tumor and blue is enhancing tumor so this is how we see it i hope that you understand this uh part if you have understood this part it means you have understood this data once you have understood this data then i think so it's much easier to work uh work further and understanding this data let me remind you these are not simple images these are volumes so that's why all of these images are having x y and z component to it uh once we have these three components then we are having uh three different classes so this is like depth uh depth that we add to uh our input image uh containing volumes of xyz once we have that we can simply visualize it uh by by dividing into different planes that we want okay once we have that this is just uh for creating gif uh once we have this diff so you can just visualize it uh in different planes uh three different planes are are seen here you can see all the three different planes and and their corresponding outputs here okay once we have that then we pre-process it pre-process these patches of data where we standardize and and we generate these sub volume so this is where we are calling this function gets a volume so this is this is randomly generating a sub volume of 160 160 come of 16 okay so it has nothing to do uh it has nothing to do much it has nothing much to do with the logic it's all about numpy manipulations okay once we have done that then this is test case to see if we have or if our implementation is right or not okay once we are done with all these tests which we see that they are right then what we will do is we will load our file our data then we will look at this application of this function get sub volume so if you look at the shape then of course we have just achieved it and we have retrieved this 160 160 gamma 16 volume sub volume out of that whole mri volume mri image volume then this is the output 160 160 16 comma 3 so these three are for three different classes and this is again a visualizing patch so this is this was actually one of the sub volume uh this is a single sub volume and this is the output for that sub volume uh for channel two and channel two containing enhancing tumor if you if you want to change it to another value into another channel we could also see the result for for that what we call uh a non-enhancing tumor i have stored in one and uh edema is stored in y0 in the first dimension of y okay once we have that then what we'll do is we'll standardize it having a zero mean and one standard deviation and then once we have normalized it then we will call this function we will define this architecture this is kind of a bit tricky part but not not too much of a tricky part and then we will also talk about this nice similarity coefficient but and this is for a single class and this is just looking at the implementation of our implementation is right or not once we have done that then we will go for multiple classes and this is for multiple class once we have done that then again we test it if it is right or not once we have tested it then this is soft dice loss and this is the formula for soft dice loss uh you can check it out in the google but the main idea is same as i have already explained in dice loss and uh once uh this is for multi-class dice of dice loss and this is actually the implementation of the numerator and denominator part and we combine it to create this dice loss function once we have this dice last function then we will first of all check it whether the implementation is right or not this is test case 2 this is test case three once we are completely sure that it is right after five different six different test cases then uh we will we will focus on our unit model uh okay uh in unit model we will be having these two different blocks one is convolutional block and another one is this get up convolution this is deconvolution why we are calling the function for that although we are just doing simple 3d convolution here so this is a 3d convolutional layer because this is a function why let me just go and let me show you this unit model here if we see so we have an input image using this input image this is a convolutional layer again convolutional layer and then uh we have a relu uh radio activation function and uh given this uh relu activation function we are the output of that is uh then max pool then again uh convolutional convolutional relu max pooled convolution convolution and then again uh we have a kind of a reload then pulling and then convolution then again convolution then cooling and then convolution and convolution and then up sampling okay first of all this from this up sampling we go upward so first of all we have we we have seen the pattern in two things one it goes up to this depth so this is the depth of unit model okay up to this depth we have this convolutional block containing this we have this convolutional layers applied again and again in the multiples of two and for each time we have different dimensions of convolutional layer if you look at it for each time we have different dimensions so instead of writing this convolutional layer again and again we have defined a function which we will be which we will be applying uh given the input parameters as different dimension as explained in this unit model that's why we have defined this conversion function and inside we have we are just applying this con 3d layer okay and in this getup convolution deconvolution this same thing is happening uh as this uh green line is up convolution this is a gain of convolution of convolution of convolution so for each of these up convolutions we have just called this function and we will be using the input parameters to generate different kind of convolutional layers okay let's combine all of this as i have already explained the depth we will be storing the levels what are levels as i have already explained maybe i guess this single block of convolution convolution and this uh relu is called level one and then we have level two level three level four each of this level will be used for uh during up sampling so we will store this information in this levels list okay then for up to the depth i mean while going downwards this is going downward this this this for loop is for going downwards what we do is we first of all create convolution and then we again create convolution and then we max pooled given if we have not reached yet depth once we reach depth then we we don't max pool it instead we concatenate this layer one and we we just call this a current layer this layer two why let me show you that so this is two times convolution for the first level and then max pooling then for the second this is like first layer first uh level this is level one of depth then during this second level we have layer one for convolution layer two of convolution then again max pooling then again max pooling then again max pooling but uh up to that up to death up to uh this depth we have this this uh max pooling but after that we we have this up sampling for this last layer why because we have this convolution and this convolution and there is no red red sign of max pooling instead we have this green sign up of up convolution so that's why we called up to the last last level of depth uh we call this current layer max pooling and that's that current layer will be used as an input for this convolutional block in the in going forward however uh in the last uh in the last level of this step uh we called layer two as a current layer and uh what we do is we we convolve up we do up convolution from this down down level to the up level so we go uh go in the upward direction uh from from the lowest depth to the upward side in the unit architecture and in upward side what we do is we up convol after that we concatenate and then we again store this convolution block and this convolution block so these four things are seen if you look at it so first of all we up convolve then we have two times convolution and then we again upgrade two times convolution up convolve two times convolution up convolve these two times convolution okay once we have done that then we have a final convolution of 3d and then we apply this final activation function and then we uh after deploying this equation function and this final convolution uh then we have a model which we which we actually define from input to output and then we compile it and this is what it is we will call the function where this model has been combined and returned and then we will look at this model that we have just uh created while looking at this model uh this is in vertical form so that's why it might not show this u kind of a shape but this is basically this same unit that we are talking about here uh you can just confirm it uh after saving and and rotating this this model okay then training on the large data set as i've already explained is a problem but let's say we are initializing it uh and we are storing all of that and uh these are like uh these are like uh fitting with the complete data set uh once we fit it with the complete data set then the problem is that uh we have resource exhausted error that is what i that is what i have uh at least confronted even though my laptop is having 32 gb of ram so my laptop is really powerful but still it is like there are a lot of resources uh but still i am getting this resources exhausted error for that reason uh we will write this function of volume data generator uh where we will be randomly uh getting out this volume uh and in the volume data generator uh class we have defined basically what we have done is uh that this class is actually inheriting from keras.utilities.sequence class so within this class if you are inheriting from it so this is what what it is doing so on each epoch end and uh and and this this function of get item what it will be doing is that randomly it will be sub sampling or randomly it will be generating this sub volume and will be giving for training okay once we have that we will we will uh we will create the instances for these classes the for training data and validation data and then what we will do is we will fit with this generator class and it will work and it has worked then we will show save our weights and once we have saved our weights then we'll load our weights and then evaluation part comes in the evaluation part we will do patch level prediction so as i have already we have already seen this patch this sub volume uh we will remove this batch batch batch what we call we don't need batch we just need this single image so that's why we are we are giving zero uh inside this dimension of an array uh so we are looking at basically non-enhancing tumor that's why it's true if you want to look at the actual enhancing tumor then we could also look at this y1 once we have done that then what we will do is then we will basically predict we will first of all normalize it and then predict it so this is the prediction part this is the actual part so our prediction is not really good for this non-enhancing tumor for this second class for this third class uh however it can be improved uh if we play with the if we run it for longer and we play with some hyper parameters let's look at the sensitivity and specificity where we are just calling this true positive or we are just computing these two positive two negatives false positives false negatives and finally sensitivity and specificity so this is just test case to see if our implementation is right and once done then we will just calculate it for each of the class uh then running on entire scans this is like getting the label image then we predict and visualize so this is how it looks so this is the ground truth for different sections and these are the prediction so the prediction seems good but transversal section is not predicting really cool and citadel is good but the green class is not not not prediction the prediction for green glass is not good so we could visualize also that uh now i hope so this makes sense and i hope so it is much more clear now uh as these videos are short videos are kind of videos for machine learning professionals who and deep learning engineers who really know about deep learning and the tasks that are at hand so i have just gone through all the process in a way in that way uh i will be creating a separate playlist where i'll be a live coding and i'll be giving a detailed lecture about all of these topics and all of these uh short video uh i'll be expanding on those short videos so just stay tuned and i'll be working on that and i'll be also explaining that so if you have if you have gotten the complete explanation perfect if not then i cannot make this video longer enough i cannot make this video more long so for that reason i'll be creating separate video