yes okay cool yeah yeah thank you so much everyone for joining this course and this talk I'm super excited to be here uh my name is Sophia young I lead developer relations at mol so uh yeah sorry it's June already uh the online of this talk this talk is we're going to talk about uh an overview of m models we're going to talk about our fine tun API that we just released today and also we have a open source fine-tuning code base that you can play with and then I will show you some demos uh some brief history about mrol I hope all of you knows about mrol but if you don't we are a Paris based team of more than 50 people um we uh we were founded uh about a year ago and in September last year we released our first model Mr 7B and then December we released mixol a time SB uh with a lot of good feedback on both models and uh we also released our first commercial model Mr medium and also the platform where you can use our model on the platform through our API uh and then February this year we released Mr small and Mr Large Mr large is our Flagship model with Advanced reasoning U multilingual capabilities function calling all the good things and Lead is our uh conversational AI interface is completely free so if you like to talk to our models um please sign up and uh use l and in April this year we released uh at the time it was the best open source model eight times 22b uh it was it was really good and just last week we released craw which is a spe specialized model train down 80 plus languages uh programming languages and you can use um with various uh vs code plugins to uh to generate code and talk with your code and then here's another view of our U model offerings we have three open source models aach two license which means you can use it uh freely for your inter your personal use cases or commercial use cases we have two Enterprise grade models Mr small and Mr Large um yeah so for your most sopis needs Mr Large is really really good uh it supports multilingual function calling uh really good at Rag and uh and now we support fine tuning um Mr small and Mr 7B and and again we have special specialized postol for for coding and we have an embedding model so we really care about uh customization a lot of people asking about like how do you find tun mol a lot of people want to have a recipe or want to use our API to find TR model so uh about two weeks ago we released um Mr fine tune uh which is the fine-tuning code base everyone can use to fine tune our open source models um and then just two hours ago we announced uh fine tun API so you can customize your model uh customize your mile model using our fine tuning API directly and uh yeah so uh the technology we use is Laura fine tuning uh you since this is probably the end of the course you probably already know a lot about Laura uh it's it's very efficient it's very performant uh so we did some analysis on the comparison between Mr laara fine tuning and a full fine fine tuning on Mr 7B and Mr small so they have really similar performance as you can see here Mr 7B um yeah Laura fine tuning uh on this Benchmark is is 0.9 and and the full fine tuning is 0.91 so very very close uh note that this Benchmark is an internal Benchmark normalized uh to U this M Small number so um just some note there um and then before I show you some demos just uh some thoughts on prompting and fine-tuning uh you probably already learned about this so before you start F tuning you should always think about maybe you can just do prompting uh so with prompting your model can work on out of the box doesn't require any data or training to make it work so it's it's really easy um and it can be easily updated for new workflows or prototyping uh with fine-tuning uh sometimes it works a lot better than prompting uh it it can work better than a larger model faster and cheaper um because it doesn't require a very long prompt so for specific use cases with a large model if you have a sophisticated prompt you can make it work but with um small fine tune models you can just find un it on specific behavior that doesn't require a long prompt so uh yeah again it's better alignment with the task of interest because it's specifically trained on different tasks and uh you can teach it new facts and information okay so now I want to show you some demos basically um how to use our fune API and also how to use Excuse me our uh Mr fine tune the open source codebase uh specifically uh I want to show you four demos uh demo of how can you use a fine tune model uh that maybe fine tuned by yourself or by someone else um demo of some developer examples like real world use cases and also the API work through and the uh Mr fine tune work through so okay let me move this bar okay so in this notebook uh first of all we need to install uh mol API I this is the latest version um 0.4.0 that was released uh today so if you want to use uh or finding new features make sure you have the latest version um and in this notebook I'm actually using uh three finetune models shared by my coworker so if you are in an organization you want to use uh models created by your coworker or you want to share the Lin model youself yourself cre ated uh you can you can sh you can use this the model from the same organization um with different different sets of options so it's really easy for me to use um a model that my CER has fine-tuned uh today so for for the first example um it was trained on title abstract pair from archive so basically if you input a um a title of a research paper uh this model will generate a abstract for you so if I input point two is all you need uh it will give us some similarly okay abstract for for this paper title uh and then just another fun example the croissant convolution uh we made it up obviously so uh so the abstract is like we propos novel convolution ler uh the cisson convolution uh so so it's just a fun example for people to see how the fine tuning could work um for to play with and then I have another example mroy is all you need and again you will output the abstract for a research paper because we trained um on the title and Abstract uh from from the research papers uh and another example uh here's another model uh note that the model name always have this structure uh it's fine tune um it's fine tuned on open mol 7B our 7B model so it will always have have this um this name here and it will have some random strings um for for the fine version um so for the medical Chat bar we actually trained on this honey face data set of uh a medical cha data and then here as you can see we ask it some questions and and will answer uh like it was a chatbot uh and another example so those two examples are from actual data so here the data we get from archive and here's the data we got from hunking face but what if you don't have the data you can actually generate um data from a larger model you sometimes we want to mimic the behavior of a larger model like Mr Large because we know Mr Large behaves really well uh and we want to do some knowledge desolation or knowledge transfer from from a larger model uh so in this case um if you want to see how exactly we generated the data uh you can go to our docs and in this example uh we have uh this prompt you are a news article stylist following the economist style guide so basically we want to rewrite the news as the e Economist uh news article and then we use Mr Large to generate the uh the revised news or like different revisions of how the news can be revised and we give it some guidelines basically so so in this example we use Mr Large to generate our data and then we give it a news article right and then we use the F tune data um it can generate a new piece of news article that may sound more like Economist news um and then if you give it give it some prompting it can also generate a set of revisions it propos for you to to change how you can change the style of the news to make it sound sound better or more like Economist uh yeah so that's that's um some quick demonstrations of how the fine tune model can look like um in our docs we have some developer examples of real world use cases that has been using our fun API uh for example we have um FF they they are startup using uh our function model uh for rack for internet retrieval and you can see the details here uh we have examples of rack for medical domain uh Financial conversation assistant legal co-pilot uh so they are all using fine-tuned Mr model with our fine tun API and in in these examples you can see the specific data um you can see the evaluation training uh steps and how how The Benchmark uh results look like um yeah so the fine tune Mr small is better than the not fine tune Mr small not surprising um yeah and and then different results so yeah if you're interested uh you can check out the our de examples for some real world use cases of U fine tuned model so yeah so in this next section uh next demo there are so many demos I want to show you today um because it's very exciting uh in this next next section I want to show you an endtoend example of how you can use mral finetuning API um on your own and for your own data set so um of course we need to install mol AI um make sure is it's uh 0 point uh 4.0 uh or or some numbers larger than that after probably after after today and we also need to install pendas the first step is of course we need to prepare our data set um yeah so whenever you want to do some fine tuning the first step is always the data uh in the simple example we read data from a paret file that's hosted on hungy face is the ultra chat data U we only re we are only reading one parket file because this data is quite large and we don't want to spend too much money on it so um so you can feel free to change the data to your own use cases uh we split the data into training and evaluation here and then we save the data locally uh into the DL format uh this format is actually needed for using our API uh and note that uh here are the sizes for the training data and evaluation data so there is a size limit here uh for our API for training data is limited at 512 megabytes um so you can have multiple files feeding into your training your fine tuning pip line um but each file needs to be under 500 12 megabytes for your evaluation data set it needs to be less than one uh one one megabytes um yeah so a lot of times the data um hugging face is not not greatly formatted for the training purposes so um we have a a script to re to help you reformat your data um this script is not a maybe not um robust in in all of the use cases but in our use cases works so if your use case is more sophisticated you might want to change the script to your to your own uh use case accordingly uh so in this example we basically just skipped several examples because they're not not right formatted uh so if we take a look at one of the examples one of the issue here is the assistant message is empty so we can't really train uh based on this kind of data uh so we need we need to make sure the data is Right formatted uh otherwise it's it's going to be difficult I have I have a quick question about this is there like a format validator that y'all have I see that you have a thing to convert it to format but curious about validation that's a great question we do have a validator script as well that was in my next notebook oh sorry about that no no no that's a great question so we do have a validation um data script from the MR find to uh repo um also whenever you upload the model to yeah whenever you upload upload the model um to our server uh you will validate the data for you so if your data is not right it will tell you why it's not right and ask you to change it so excellent yeah thanks for the question yeah so yeah so the next step is to to upload upload the data set uh with the files create function and then you can Define um the file name and then the actual file here and we're uploading the file um and then we can see the ID of the file the the purpose of the file uh the purpose right now is just fine tune maybe there will be some other purposes later but right now it just fine to and then to create a fine tuning job you will need the IDs of the files we just uploaded uh and of course you need to define the model you want to fine-tune uh right now we only support Mr 7B and mral small but in the future we might add more models here uh and then you can you can try different hyper parameters uh and then in this example I only ran 10 steps um just to have it faster um but if you want to increase your steps AC you probably should increase your steps if you're doing something serious um yeah so so make sure you can change those different configurations here and then we can we find the job ID and and then we can use uh and then we can like do different uh different things like listing the jobs we have I have quite a few jobs that I ran um and uh we can retrieve a job based on the job ID to see if it's successful or not um and then you can see we have some metrics here training loss validation loss token accuracy uh because we only ran 10 steps you only see this once but if you run run it 100 steps you will see it 10 times so every 10 step um or every 10% of the step you will see the the metrics um so you can see if you're making progress or not making making progress and finally with the fine tuned model when the model is um the job is successful uh you will see the fine tune model get populated uh from your retrieved jobs and then you can call this model and then ask any questions you want so so that's that's how you can use a fine model it's exactly the same syntax as uh what if you if you are using our normal nonf finding models okay and finally uh if you want to use weight and biases uh you can add your weight weight and biases uh credentials here um yeah you will need your API key and everything uh just to show you how that might look like basically you can check your your losses your proxity score uh your learning rate and everything so yep so that's that's how you can R the uh fine tuning through our API I hope it's it's clear any questions Okay cool so this last demo I want to show you is uh what if you want to just use our open source codebase to F tune uh Mr 7B or other other Mr models uh I'm actually running this in collab um Pro Plus account so it's possible to to find two model uh in collab um so in this example we I just get clone this repo the Mr finding repo because it's not a package it's a repository and in this repo we need to install all the required packages um and of course we need to download our model because we're we're doing everything uh basically locally uh and we're downloading this model from from the mro website and we're downloading uh 7 BB3 model uh here and then same thing as we have seen before we're preparing the data set uh we're using the exactly the same data as before reading the parket file splitting into training and evaluation save the data locally and then same as before we REM reformat our data everything uh the evaluation data looks pretty good and afterwards you can verify if your data is uh correctly formatted or not with our evaluation data script so yep so it will this this actually will take some time because it's evaluating um yeah each record and then for and then you can start training the the important thing here is actually this config file right this this basically is telling the model uh telling the uh the Laura how you want to fine-tune your model um and and different where where's the path of everything so we have the data file you want to define the path of your your instruct data your evaluation data this is the training data your evaluation data we want to define the path of your model um we want to you you might need to define or just it as default the um uh those hyper parameters uh we recommend using a sequence lens of um 32k but because I'm using a collab as the memory is limited i r to out a memory issue a lot so I I decrease this number to 8,000 um but if you have a have a better GPU you you should use uh 82 uh thousand and then you can you can Define all the other font stuff right okay yeah and then with this one line of code we can start training and finding our model um and then in uh the result as a result you can see the checkpoint of um of your your Lura result here this is where we can use um to uh to in our inference um as our fine Unity model and to run our inference we have a package called Mr inference to help you run all of our open source models and also all the fine tuning models uh so basically in this uh in mral inference you need to define the tokenizer which is um in the which you are downloading from the myal um mistal fine tun file we use the V3 tokenizer because it's a V3 model uh and then we need to Define uh the model that's reading from the models folder we downloaded and then we need to load Laura from the checkpoint that we have we just saved from the P fine-tuning process um and then we can run check completions and get the result so that's basically how you can run uh Mr fine tuning uh with Mr fine tune uh the code code base uh and finally I want to share here some exciting news that uh since we just released our fine tun API we are hosting a fine tun hackathon starting today to Janu to June 30th um yeah so please feel free to check out our hackathon and you can submit from this Google form and yeah very exciting looking forward to see what people are building thank you so much