hello legends in this video i'm going to show you how to scrape linkedin profiles to generate new leads without paying for any apis whatsoever so that you can create spreadsheets of linkedin leads like this that have the lead's name the title the link to their profile and a snippet of information from their profile and each of these links actually goes to that person's profile that you can then use to either add them or dm them or to run some other automation to find information about their company and uh before i actually get into the automation i just want to show you the method that we're using here is actually we're basically doing a google search where we're searching the linkedin site specifically for uh people that fit this profile so we're looking for ceos in the automotive industry uh in united states and if i just copy this i want to show you what what we're actually doing here so i'm just pasting that same search string directly into google and if i hit enter you can see here that all of these results are the same results that we're pulling out from our n scenario so the first one was jim farley which is the ceo of ford motor company and then we've got them over here jim farley ceo of ford motor company so really what we're doing is we're actually using the google search api to just search for people on linkedin and then we're returning all these results we're scraping them and adding them to our google sheets all right so let's demonstrate how this works i'm just going to delete all of these search results so we can see that we're doing that we're starting from a fresh screen and back in nan i'm going to open up my chat i'm just going to say hi and now we're triggering off and we're actually executing our sequence so we're hitting that search api to receive all the results we're extracting all the results um we're waiting for a little bit so we actually don't hit any api limits for the um uh google search api calls that we're doing or for the add to the google sheets uh api calls just going to stop this here because the list is going to be built anyway um that's what the weight block is for uh so yeah after we hit the http uh request we get all the results which are actually 10 per request then we're adding them to the google sheets and then we're just checking ah are there any more results from that same you know query that we initially hit and if there are more results uh we're basically looping back into this module over here and we're just going to page two now every api that you hit for the google search api you only get 10 results back so uh technically if you want to get more and more results you have to hit that api again all right guys just quickly i have a weight list for an ai community that i'm going to be building at a high level these are the things that i'm going to be offering so like weekly calls uh behind the scenes of my nan make.com builds um any customcoded stuff that i do any other interesting things that i work on um i'm actually running this community with my brother chris who has uh run a lead generation uh like a lead genen agency he also set up dual.com's full sales and marketing pipelines in the team so he's very experienced with sales and marketing i'm very good with uh ai and automation so together i think we have a blend like a a unique blend of an offering um and then the starting price is going to be at 55 bucks so if you fill out the questions below it's basically your your name your email and then some information about like how long have you been following my channel what do you enjoy the most and then what's a must have in a community that will make it worth the money for you thank you very much guys let's get back into the video but now over in our google sheets you can see that we got the same results as before so we had the first 10 from before but now we did the page nation which is we're basically going to page two and getting another 10 results and then another 10 results and another 10 results so i stopped ours prematurely but as you can see here we're able to generate a list of leads extremely quickly and then we have links to all the leads that we want to contact all right so let's break down each module so we can understand how this works uh let's dig into the set fields so in the set fields we're actually setting two variables one is called current star index we're setting that to one and then the other one is called max pages and we're setting it to 10 now each api call to the google search api brings us back 10 search results or 10 linkedin profiles um i'm just setting us a limit so we have up to 10 api calls to that one search string that we're doing so we can get 100 leads that are ceos in the automotive industry in the united states uh but you can set this higher if you do want and then you can just generate even more leads uh for that specific query so zooming out the very next thing we're doing is we're actually hitting that search api so this is where we actually have to uh create some credentials so we can actually hit the search api but at a high level we're just making a get request to the google apis uh forward/custom search endpoint which basically allows us to make a google search using a specific search string and then we're setting a bunch of query parameters so these query parameters are actually appended into this url but to make it easy for you guys to duplicate this flow i've just broken them down like this so we have the api key so we can actually hit the api then we have the cx value from another uh setting we have to create and i've got both of those links at the start of this uh workflow so just give me a little bit and i'll get to there in a second and then we have the actual search string so q is the name for the search string and we're basically searching for ceos in the automotive industry in united states and then at the very bottom we have our uh start value so the start value represents which page do we want to start on when we're hitting the api call for this search request because technically we could just start at page 10 and then we're good to go but what we really want to do is we want this to be dynamic for the very first request we want to start on page one but then every time we make an iteration through we actually want to pull out that we want to go to page two page three page four so the rest of the code nodes in here actually do that for us automatically um but we have to run this uh simple expression so we're asking ourselves is the run index equal to zero which is is this the very first time we're running this api call and if it is then we're basically taking that value from the set fields block so the set fields block we're taking that current start index value which is set to one and then we're making this api call but as soon as we go one rotation around and now we're on run two this statement is no longer true we're not actually on the first run we're on the second run so we now go into this uh expression and this expression is the very last module that we have in this workflow and it basically checks oh is there another page of results if there is let's loop back around until we hit 10 full pages cuz sometimes you might have a very specific very niche uh search that you're looking for and it might only have three search like three pages of results not 10 like this is probably a very common string but you might have whatever you're looking for might be very very specific and you might only have three pages so you don't want to be wasting the you know fourth fifth sixth api calls for no reason needlessly uh this flow will actually just stop you um but yeah this is what this equation is doing um so then for the second third fourth api call that we're that we're running through we're just using the page two page three page four and that's what we're setting um from the actual node later on so for the very first one we're actually using the set fields which is set to one um for this since we've actually fired up a few times uh the actual value is 31 in this case and now to show you the output of the api call on the right hand side you can see if i zoom into here we have previous page we had a count 10 which is 10 results that we had and the start index was 21 so essentially going um when it when this value equals to one we're actually looking for the very first search result so we're starting on page one for the very first search result and then we're plusing 10 so that means when we go to here uh when this equals 11 it's actually the start of page two when it equals 21 it's the start of page uh three and then so on and so forth so we're actually plusing an extra 10 each time simply because there are 10 results per api call uh but anyway um i'm just going to zoom out of here and then we have uh all the api results so we have uh praka which is the president and ceo of bendix commercial vehicle then we have all the information information associated with him and you can kind of keep scrolling down and then you have uh the rest of the 10 results now to set up your credentials i've got some information over here so there's actually two things that you need to do the first is you need to go to your google cloud console and then you want to create yourself a new project so i'm just going to click on my project then i'm going to go to new project i'm going to call this linkedin-2 because i've already got linkedin one all right and google's a little bit tricky so even though we created the project it's not going to go into that project for us straight away so we have to just select our project once you've selected your project let's go back to nan we want to search for and enable the custom search api so let's just type that in custom search api let's click on it awesome let's go enable all right awesome it's turned on and now we want to go to credentials and then create an api key and then copy our api key value so i'm going to go credentials i'm going to go create credentials api keys i'm going to copy this value i'm going to close this and now on linkedin scraper i'm going to go back into our http module and over here where it has key and then the value i'm just going to backspace this paste this one in and now i'm just going to leave that to save and the second thing we need to do is create a programmable search engine so let's click on this link okay okay so our instructions are create a name leave it as search entire web and then copy the cx value from the field that is created okay so i'm going to go google- uh search and then uh i'm going to go search entire web you can technically just put the linkedin domain over here but if you do want to use this method for other websites then you know just leave it search the entire web i'm going to go on i'm not a robot click create let's just zoom out and once this is ready we have our cx value which is this little bit over here at the end of this uh url so i'm going to copy that ctrl + c go back into linkedin double click on this and where it has cx value i'm going to backspace and then paste this in all right awesome and now one more thing i want to speak about quickly so what we're doing is we're using the free tier for the google search api which gives us 100 requests per day now each of those requests returns 10 search results which means that 100 requests per day uh by 10 search results is 1,000 um lead well essentially 1,000 search results or 1,000 linkedin leads per day that you can run at no cost now you can upgrade your tier and just pay a little bit more and then you can do essentially unlimited like whatever you want to pay for you can just amp it up but this means that you can do 7,000 new leads fresh from linkedin for free uh per week uh which i think is absolutely fantastic for this method so that's one thing to keep in mind um we'll come back into here and actually test it out but let's go into the extract values now okay so what's happening in this code node uh you can go through this code you can put put it into chat gpt to explain it for you but essentially we're going through this api response which again is like full of some stuff that we probably don't need um and it's pulling out each of those 10 linkedin profiles and then pulling out their name um their snippet the actual url to their linkedin profile and that information that we're essentially adding to google sheets and then it's yeah taking this it's cleaning it up with the code and then it's putting the search results here so you can see we have p over here we have the title the link the snippet the image if i just zoom in and then we have the next person is the next result edward we have his title link snippet image everything else and then this is just for our sake when we're doing our page nation checks so we're seeing what start index are we at and is there any more results on the very next page uh then like i described before we essentially have this function which is just waiting for 5 seconds that's because we don't want to hit any api limits for the http call that we're making for the google search api or when we're adding uh information into the google sheets as well so this is 5 seconds i find that it works well for me but if you run into any issues you maybe just need to pump it up by another 5 seconds to 10 uh then for our google sheets module uh we're essentially just taking all the information uh bit by bit from the previous node which is the code block and then we're adding it and we're mapping it into the um yeah into the google sheets value by value all right so let's go into the next block then we have our page pagionation over here so uh this is pretty simple we're just going through the results over here we're just double checking that okay uh firstly what index are we at now so we know what what index we need to be at for the next http call to the google search api and we're just confirming all right are there any other search results after this page so in our case we had these search results after we run it through the code then we had uh this information brought back to us so we had continue loop equals true which means yep let's keep going right around and then we had start index equals 31 which means that all right now we need to plug 31 into the http call so we know we need to start on the very next page and get that very first result so that's why we have over here if that condition is true if we actually do have more results we go back into this http call and then if i scroll down over here and i zoom in to this start value you can see that we have 31 just like what we had in that previous uh check node as well so we had 31 here because it was 31 there okay awesome so let me just delete the results that we had over here and let's just confirm that the um google authentication method that we ran through just before is working so this is a blank sheet uh let's go to refresh this reset i'm just going to say hi hit enter and now we're shooting off and then getting our yeah 10 results each time we're waiting let's just confirm that they're added to the sheet appropriately so let me just pause this now we should have our first 10 results and there we go our first 10 results are here awesome all right guys thank you very much for watching i hope you enjoyed this video if you did enjoy the video can you please drop a comment below like the video as well it's going to help the algorithm and it's going to help my channel grow thank you