hello guys in this video we will talk about tf idf and how it should be calculated with a very simple example so let's start a very popular representation for text is the product of term frequency and inverse document frequency commonly referred to as tf-idf the tf idf value of a term t in a given document d is like this note that a tf idf value is specific to a single document d where idf depends on the entire corpus systems employing the back of words representation typically go through steps of stemming and stop words elimination before doing term counts term counts within the document from the tf values for each term and the document counts across the corpus from the idf followers each document thus becomes a feature vector and the corpus is set of these feature vectors this can set a be used in a data mining algorithm for classification clustering or retrieval that was a quick introduction to dfidf and now let's go to a simple task let's have a situation we have some sentences the first one is a quick brown fox jump over the lazy dog what a fox the second one a quick brown fox jumps over the lazy fox what a fox please keep in mind that all the sentences in our corpus are defined as small d so we have d1 and d2 based on this rule our corpus is defined as the big d now we have some data this data led us to shape a main question and a question is like this how word fox is relevant to corpus d documents remember we have documents d1 and d2 let's go to the solution part let's start in here with some definitions what is dfidf in the first part of this calculation we need to clarify that df is the frequency of any term in a given document we need to calculate df for document number one and document number two by a given argument award fox so let's calculate for document number one we have a 12 words in total in this context we have a word fox accurate two times knowing this information we can calculate df like this 2 dividing by 12 equal 0.17 in the same way we calculate df for document number 2. in this case we have fox occurred three times in this document so the calculation will be as follow 3 dividing by 12 equal to 0.25 keeping in mind that d1 and d2 has the same number of total words the first part of calculation is done now we have to move to the second part we need to calculate idf idf is constant per corpus and account for the ratio of documents that include data specific term we need to calculate idf for full corpus that we have for this we are using a logarithm in this equation at the upper side we need to look at how many documents at our corpus a given word fox is accurate now we see that award fox is awkward in document number one and in document number two so at the upper side of our equation we need to input a value of 2. on the lower side in this equation we have to input a value of total documents that our corpus is consists of and that's mean we have to input value of 2 because we have two documents in total in our corpus that resulting into logarithm 2 dividing by 2 and it's equal to 0. now we have enough information to calculate tf idf for all documents in our corpus d we have document number one and document number two so we will calculate tf idf separately for each document in our corpus for the first document tf idf equal to 0.17 from the first part of our calculation multiplied by zero it is from second part of our calculation and it's equal to zero for the second document we calculate tf idf in the same rules that we applied for the first document let's do like this 0.25 from the first part of our calculation and multiplied by zero from the second part of our calculations and it's equal to zero now we have calculated tf idf for all documents in our corpus and for document number one tf idf equal to zero for the second document in our corpus we have calculated tf idf and it's equal to zero again now we have calculated tf idfs for all the documents in our corpus and that means that now we can answer to the main question in this task how a word fox is relevant to corpus d documents and the answer is using tf idf that we have calculated just before the word fox is equally relevant for both documents d1 and document d2 because we have the same values of tf idf it's zero this calculation can be applied in any amount of documents that you have in your corpus and one more time again what is tf and idf tf is a simple choice to use the raw count of a term in a document and the idf in word document frequency is a measurement of how much information the world provides in our corpus by say in corpus i mean across all the documents that we are having so by summarizing this video tf idf is a statistical measurement that evaluates how relevant a word is to a document in a collection of documents this is done by multiplying two metrics how many times a word appears in a document and in inverse document frequency of the words across the set of documents it has many use most importantly in automated text analysis and it's very useful for scoring words in machine learning algorithms for nlp tf idf was invented for document search and information retrieval i hope that this video was useful for you and i wish you never stop learning if you like this one please subscribe me and you will get more similar useful videos in future so see you there