oriole i'm going to go over the multivariate normal distribution so the multi so this is going to be the first tutorial and hopefully many others are kind of going over clustering algorithms so what i want to go over is the multivariate normal distribution so the multivariate normal distribution is described described by probability of x given these parameters mu and sigma and then you got sort of this kind of long formula right here mu is in a k space and then sigma is in k by k space and this is the covariance matrix and then you can also just recall that the covariance is given by this so if you have an x and y covariance is just the x minus its expected value y minus expected value over n minus 1. so let's get into this so i want to create a class ultimately that basically given the data set x will predict this mu and sigma and then from there given any other x i can predict the probability so the first thing we're going to do is we're going to import numpy and then import matpot lib so the first thing i'm going to do like as always i'm just going to kind of generate a toy data set so i'm going to say x is equal to numpy or random not normal let's say sorry multivariate normal i'm just going to go ahead and give these just a centers of zero zero so this will be our mu and then our sigma here i'm going to give them a variance of one for x1 a uh covariance between the two of 0.5 and then on 0.5 again and then 1 for x2 and then i want to give let's start with 100 data points so let's see we have x and as you see looks like we got what we expected 100 by 2. so let's get started on this class here someone call this multivariate normal and then on the initialization here uh we don't actually have any parameters any init but i'm just going to go ahead and just uh give my mu naught mu just set it to none and then also set the sigma to none now let's define our fit method and then we need it passes some data x so here on the first thing we need to do is we need to get the mu so the mu is just going to be the uh basically the means so we'll say self dot u under var and actually the first thing i want to do is i want to check uh ultimately what i'm going to need to do is um these are right here will make more sense that these are passed in by uh column vectors and so actually i want to recast this x as a basically n number of column vectors so i'm going to say if x dot and if x dot dim is equal to two actually we'll do this we'll set x is equal to x comma new axes if x dot dim equal to else x okay we have that and now let's just go ahead and uh sort of make sure whoops this needs to be a plus make sure that little line of code works so i'm going to say my model is a multivariate normal and then let's go and fit it to our x and we get an error has no dimension dim so let's see x dot end in that's what we're looking for and um so there we go we get that so that should be good uh so let's see now we want to set our self dot mu under bar we're going to set this equal to x dot mean along the zeroth axes so let's do that again okay that works and then let's just check our mu it should be roughly zero and uh whoops did we not do that along the correct axis there looks like we did not oh we were supposed to do that on the first axis since i just created that we broadcasted that oh actually let me see here actually this is apologize this should actually be uh to make this a a bunch of column vectors this actually needs to be added along the last axis there and so this should be in fact the 0th axes and there we go we get a 2 a column vector that's approximately equal to zero as we would expect alright so now to get the uh covariance so what i'm going to do here is now i need to subtract my x from u so i will do uh x minus self dot u and actually what i'm going to do is do a uh numpy dot find sum i'm going to do the uh i j k so that's going to be x minus mu and the second one will be i but instead of j and k i want these to be uh inverted so k and j and we want this to go to a a j by k and let's pass that in so x minus self dot u under bar and then pass it in again self dot u under bar and this should hopefully give us our uh our covariance matrix so self dot sig under bar is equal and if all goes correct here the last thing we need to do is divide by n minus one so let's do that let's do uh x dot shape along these zeroth axes and then subtract one and if all goes well this should hopefully fit sigma to our toy data set so we get 0.74 so let's see let's actually make a few more numbers let's make this 10 000. and there we go we see it approaches one one point five point five as we would expect let's get rid of that make those there so there we go so now we've properly fitted our model so next thing we want to do is we want to get the probabilities of it and so as always we'll pass self and then we want it to pass in some data x so now we need to implement this formula right here so now that we do have our mu and our sigma we can get the probability of x so i'm going to divide this up and i'm just going to call this portion right here factor one and then this portion right here factor 2 just to kind of clean up this function right here so we'll do factor 1 will be equal to 2 by pi up to the minus self dot mu under bar oops dot shape and we want this to be along the first axis because our mu mod.mu dot shape is a two by one so that should actually be our zeroth axes since it's a a k column vector k will be two in this case and then we'll divide that by two and then we need to multiply by the determinant of our self.sigma and then multiply that to the minus half power so there's our first factor and now we need to implement the second factor so here we need the exponential a minus half and then we need to uh implement this portion right here and the easiest way to probably do this is again use this ein sun function so we'll say numpy dot ein sum and we want the first one this x minus u we're gonna make that i j k like last time uh the sigma right here we want to multiply that uh by this uh by the j since this is inverted right here we're gonna multiply that by the j and then i'm going to give another axis called l and on this last one i want to call this one i l and um k so that on this one we'll multiply the x this axes by this axis of of our second one and then from that we want to go ahead and again we'll just keep it as a column i number of column vectors so we'll just say i by k or each or i guess a one by a one by one column vector n number of one by one column vectors we'll just keep it in that format so now we'll pass in our x minus self dot u under bar uh then we need to take the inverse dot inverse of our sigma self dot sig under bar so there's that and then again we need to pass in x minus self dot u under bar and then if this all works out correctly hopefully this will work for us factor one by factor two so let's try that so say [Music] mod.prob and pass in x and we get an error 100 operands cannot be broadcast together shape 100 by 2 so there's something here on this factor so it's probably on the sign sum so let's go and see if we can kind of troubleshoot this here so let's see x minus mod dot u under bar those cannot be broadcast together oh that's because i forgot to do this here i need to re-broadcast it so i'm just gonna go ahead and put this here real quick and really we should refactor this since we're repeating this twice so i'm just gonna call a static method here method uh we'll just call this uh uh redim the x pass it a x and let's just go and yank this guy out here and so there we can say x is equal to read max and then again yank this out here put him here delete this guy and let's see that should hopefully that should hopefully work out for us redem is not oh redem x x and then right here same thing x let's see name read them x is not defined it's because it's not in the correct space so we need to do self dot read mx self dot read x so there we go so let's try this here 100 by two and two by one cannot be broadcast together all right so let's see what did we do here mod.underbar so that's the correct form that we want oops and as you can see it looks like our function is doing something so these appear to be probabilities uh they're between zero and one so uh let's go and see if we can kind of kind of plot how well our function does so i'm going to create a mesh grid to kind of plot these probabilities over so i'm going to do a line space uh we'll go from -3 to 3 let's do about 100 times and then we want to repeat this twice along the zeroth axes and then pass this into a mesh grid it looks like we need to unpack this along these 0th axes and there we go we get two and then let's make this into an array actually so for plotting purposes we'll want to unpack these into an x x and x y but in order to pass this into my function i need a i can't really pass this in as 2 100 by 100 uh matrices so let's see if we can reorganize these guys that works there and now we want to let's see the shape of this guy is 2 by 100 by 100 and i want the 2 to be uh along uh the the last dimension here so i'm going to say transpose transpose the sky we want this to be one two zero so that should get us the shape we want and then if we reshape this guy to minus one by two by one we get the form that we need so that looks like that's going to work out for us so i'm going to say this is x new now let's get our probabilities is equal to mod dot prob of x new so we got those now the probs here are going to be of the wrong shape so in order to pass this back into this contour i'm going to need to reshape this guy to be 100 by 100 like the xx and x y so now we should be able to uh plot the contour of this x x x y we've got our probabilities let's see hopefully this plot something here looks like we got something that's kind of gaussian distribution looking let's make this a little bit bigger there we go we can see it a little bit better uh let's give it some labels on the uh the lines here so we can kind of see some values awesome we got some labels and then now for the final test let's just kind of uh make a scatter plot of our original data that we had here uh so star x and that should hopefully make us scatter whoops transpose star that should hopefully scatter our data points so there's our 100 original data points that we fitted to our gaussian distribution and uh appears like it looks like a pretty good fit to me um so i think we've hopefully successfully implemented this uh if you liked the video leave a comment leave a like if there's something uh you had a question about or something that doesn't look right just let me know thank you bye