come on I'll give you a quick one minute intro to the office welcome to nutrify HQ I'm Daniel Burke machine learning engineer and co-founder of neutraly which is an iOS application to help you learn about Whole Foods this is Josh hello he's our head iOS engineer so anything you can feel and touch on the IOS app on nutrify Josh has probably built that here's our planning station not sure if there's confidential stuff on there but we got two whiteboards this is our data labeling station so the whole idea with machine learning is that you draw boxes around food and you try to get a model to replicate that our model is image classification right now but we want to upgrade it to object detection and this is my office which by Hoodie cuz it's been getting a bit cold lately I have two computers upstairs I'll show you them later we got an article what we learned from the year of building llms shout out to her this is my technical bookshelf and one of my favorite books designing machine learning systems by chip Hillen probably the the most important Book for a machine learning engineer to read anyway that's been a quick intro welcome to a day in the life of a machine learning engineer nutrify office is a converted Lounge room in my house and my brother lives about 1 km down the road so we usually start between 7: and 8:00 a.m. I often get up and make my girlfriend an IT and cook breakfast for the three of us we eat quite simply mostly Whole Foods not too much this morning's breakfast was three rice cakes and three three eggs with a pinch of salt we have our morning meetings outside in the sun and here we catch up on what we did on the weekend and discuss what we'd like to do over the next couple of days after breakfast around 8:15 is we get into it the first thing I usually do is write down the top three to five things I'd like to get done in the day on a notepad and for the first half an hour to an hour or so I read an article on machine learning or a technical paper to break things up during the day I do some form of exercise snack which may be body weight movements every 45 minutes or so or a 10 to 15 minute walk every couple of hours today was five pull-ups 12 push-ups 15 squats every 45 minutes it takes about a minute or two and adds up nicely by the end of the day from around 9: to 10:30 I spent just over an hour manually labeling food images for the new nutrify food Vision model I like to spend about 1 hour per week manually labeling data so I know it's of high quality and I'm familiar with the data set we're working with after data labeling it was time for another exercise snack this time I swapped the squats for squat jumps round two of movement done model training time so this is how I train models I got nutrify food Vision I got a script called train up high can add some notes that's about the experiment that I'm running so I labeled about 300 maybe 400 manual images there's the model name this is from the Tim library on hugging face train the body means I want to fine tune the entire model and 50 EPO so the WB note stands for weights and biases all this information here if we hit the magic key as long as I've got no timeos this is going to upload so gcp is connected weights and biases logged so this is weights and biases is going to track all the metadata the model artifact uh metrics all that sort of jazz there we go view project run here so that'll appear down on my MacBook and training is going to happen on this PC here so there we go downloading artifact all the files and we'll see how this runs in a few minutes so it turns out that this training run actually failed okay I think I found the issue dimensionality issue here when I calculate the test test accuracy Dimension out of range expected to be in range of this so there's a a dimension issue I don't think I need this torch ARG Max maybe we'll go find out [Music] oh my goodness these roasted sweet potatoes and change your life so we got lunch time at nutrify HQ naturally being a food app we like to cook all our own food so today we've got some ground beef little bit of butter seasoning got roasted sweet potato guacamole and roasted broccolini stalks and we have two special guest today have Joey who's none other than our residential food extraordinaire Joy's helping us turn nutrify from image classification to object detection and we have none other than pizza dad oh hello hello hello hello good gra you all piz out you may have recognized you from some of my courses um helps with computer vision teaching and but today your sandwich day oh thank you so while the food's been eaten models of training and Bone appetite family thank you cheers duck in Duck in Che oh [Music] jeez during the afternoon session we had Joey on the food box labeling Josh working through the IOS app and me refining the data pipeline if I could recommend a read that most mimics what I do day to-day it would have to be the 2019 Apple machine learning paper called Overton you see that person that's me someone who monitors the data the models and sees which part of the data needs to be upgraded in order to improve the production system and as I've said before we're trying to go from single Foods per image to multifoods so my main goal now is building an automatic data labeling engine to identify and label multiple foods and images in other words copying Tesla's data engine for self-driving cars but for food images we still rolling are we in 4k it's been a brilliant afternoon at neutal HQ so uh fun thing is is a machine learning engineer is there's not actually that much model building a lot of it is once you build a model you kind of just refine the data set and so this is my workspace for weights and biases where I track all of the experimentation and as you can see each one of these runs is a mini experiment so this is the one that's running right now upstairs I'll go show you that in a second but there's the loss curves that are going down which is good accuracy curves going up exactly what we want and there's the model I got all the hyper parameters saved here so basically if you're running machine learning model experiments you pretty much want to be using something like weights and biases there's a lot of experiment trackers out there I just started using weights and biases and I love it the model is from hugging face but we fine-tune it on our custom food image data off to Josh Josh what do you doing I've just been chilling no I'm working on text detection what do you mean there's just Daniel changed the names of all the foods so so I can better interact with these model so now it's just ruined a lot of matching that I had image matching that's what I'm doing text and images so we have two models that run in nutrify if you take a photo of something there's the food Vision model which will identify what food is in the image and that works on image classification but some things aren't image classified they have text so we also run a text identifier over the top and eventually we'll probably look into something like combine those two into one model so right now there's two models eventually combine that in something like a generative AI VM type thing but that's future we come over here speaking of model upgrades our in-house grocery boxer Joey Joey what are you working on mate I'm just going through about 10,000 images of food doing the manual labor here of trying to to Source out what is individually seen on here so the model can pick up instead of just one foot at a time it object detection and figure out that hey for this example we've got fries uh tomato chutney or salsa we' got a chicken pmy and some rocket on the screen so instead of picking up just one it will be able to detect everything on the page at once multi food multi food now come upstairs I'll show you where nutrify the brain lives this is really cool you know what we realize today is that this house has solar panels and solar gets translated into electricity and so you could say that the computer here is being powered by the same energy as food cuz sunlight hits a plant grows food circle of life like the lon KY but nutrify food Vision model is training on here you know those loss curves I showed you on weights and biases before that's what's happening here so sunlight into model weights how cool is that and so we've got some training logs here um testing Epoch 7 training Epoch 8 I could optimize this by maybe only evaluating every five EPO or so like that but I like seeing training and testing results and so we got the Titan RTX this computer is about four uh Nvidia Titan RTX that is this computer's about 4 years old maybe closer to five now but it still runs like a gem all of neutr models uh that have run on a device are trained here and if you look down here we've got an even better deep loing PC this is quite new it has an Nvidia RTX 49 and so this is the newer Generation all the specs have been bumped up uh compared to my original Deep lining PC and you might be wondering why don't I train the models on here well I've already tried that and the really cool thing is that neutralized food Vision model takes about 22 hours to train on there in the current training scheme that we have takes about 10 hours on here and so Daniel why are you training on the Titan RTX but not the RTX 490 well I'm going to get to that eventually but the 490 is currently doing the very important work of inference so it's going over an extremely large Text data set of about 50 million rows that are sort of sentences related to food about a billion words maybe and running a named entity recognition I almost forgot I know it's just n n model over all of those rows of text to extract the food terms and why are we doing that well we want to create an inverted list of term occurrence so say the word Apple occurs 10,000 times we want to make sure nutrify takes care of the top 10,000 most popular foods as judged by the internet because that's what we want it to be a food app and so this is a really cool thing about having two computers is that one can be training the model and one can be performing inference to enhance a data set and eventually we want to follow this pattern for every process we want one to be training a model at all times and another one to be forming inference to upgrade the data set and we keep that flywheel going so data in label Feed that to the model training occur reoccur all the time revolutions you follow revolutions keep the clients on the feris wheel and it goes the Park is open 247 365 every decade every goddamn Century that's it now I got to go back downstairs and monit monitor some more training there's a timer I'll see you soon machine learning is a lot like building muscle you want to make gains sets and wamps you want your model be better run a th000 experiments a lot of them will fail or crash but that's how you figure out the ones that work like dazzling dream 1348 1384 okay let's round out this video so what's happening is we got a model training as always finishing the day model training overnight so we'll leave that run out and then what's going to happen after that very important step in any machine learning workflow is is to predict and evaluate so uh that's the last prediction one and in here is where you'll see I go through all of the predictions and see how the model did on each of our different classes and make sure that it works okay in the real world or on our test images because otherwise if we ship a model that doesn't work that's not a great experience but what you'll see is throughout the video we've been testing it ourselves and I use nutrify every day so I know how it works on real life Imes of food and I actually found a really cool term today in the article I was reading about llms which is geni Jutsu go and see for yourself that's from Toyota's production system apparently and I translated this geni gutsu is real things real places same as nutrify using the app in everyday scenarios that mimic actual use cases repeat after me the real test set is real life peace are you waiting for the be yep okay when it beep I go [Music] y goodbye Josh thank you for today another beautiful day how would you describe working at a startup Josh fun