what's going on everybody Nick here welcome to the most comprehensive video on scraping inm make.com currently available on the internet I'm going to teach you everything you need to know about web scraping in make to make a big splash either in your own business or in the businesses of the companies that you're working with so you don't need to be a web developer to know how a web scrape instead you're going to watch this video you're going to learn the 8020 AK you're going to learn the 80% of stuff in 20% of the time that will allow you to scrape the vast vast majority of all static and dynamic websites out there both inm make.com and then using something called headless uh browser scraping which I'll show you in a minute the real value here is that you'll be able to connect like any website any scraping Source anything really to your make.com flows and if you combine it with what I've taught you in previous videos You'll essentially be able to make every system known to mankind in make which is quite incredible in order to do this I need to show you a couple of things and now I don't want this to be a very detailed technical extraordinarily nerdy sort of video I want this to be something that's accessible to the vast majority of you regardless of where you are in your make.com journey but in order to really get it and to really get that 8020 we're going to have to cover a couple of computer science Concepts so what I'm going to do is we're going to start there in a gamma slideshow and I'm going to explain these Concepts then while I'm explaining and after we're going to move back over to make.com and I'm actually going to run you through them all uh in this scenario editor here so the things you need to know in practice from top to bottom you need to know about the two different types of websites static and dynamic and the differences there're in you need to know a little bit about parsing and how it works and then from then on out I'm just going to scrape the websites live in front of you both static and dynamic and I'm just going to do it so many times that you'll have no choice but to learn how to do it this is how I learned how to do this I just looked over the shoulder of a friend of mine that was quite good at this stuff I watched a bunch of YouTube videos and I basically hammered my brain into submission until I knew everything that I needed to know about scraping I pattern matched it essentially and that's what we're going to with you too so let's start with the two types of websites that start with this little bit of theory essentially some websites are statically generated and others are dynamically generated I'm going to talk about those just in a few seconds but just keep those in mind they're static websites they're Dynamic websites but we take one step further back uh what the hell is a website anyway well websites are made up of HTML H being hyper text hyper T being text uh M being markup and then L being language but what the the hell does that mean essentially this hypertext markup language is just a convention that a bunch of people happen to agree on about 30 years ago bunch of people got together shook hands and I think the guy's name was Tim something which I mentioned in a previous video um but he invented this hypertext markup language just as a shared set of symbols that everybody from then on out if they wanted to put something on the Internet would use in order for it to be rendered nicely if you think about it logically why do you need one unified language well it's because because if you want to build infrastructure um that'll just work on every computer work on every browser you need to make sure that the data is being represented in the same way right so sort of makes sense now because the web Builds on itself we've used HTML ever since when HTML was made all of a sudden you know a couple of months or weeks or years later whatever uh some person would make some change to it or they'd create their own coding library that would add on to HTML do something cooler um and so since HTML was the beginning basically everything else has just been building on top of it and so now we have all these crazy Frameworks and we have all these styling languages and all of these different approaches but at the core these approaches just output HTML because HTML is what everything else runs on now HTML is the code but code is for machines it's not what you see I'll give you a quick example if we go back to our scenario editor if I just I don't know I'm just going to make an HTTP module here the get a request and then I'm just going to right click here and then I'm going to press inspect this is going to open up a little window on my right hand side here called the Chrome developer tool conso and the real value in this isn't like I'm not going to run you through this because this that pretty complicated but uh the point that I'm making here is just that everything on this website like literally everything on this page is actually HTML and if I scroll through it on the right hand side here you'll see that um as I scroll through some portions you'll see that portions get illuminated other portions don't and that's because the code here actually like is referring to a specific element on the page this code div class equals IMT I panel space in style whatever this is actually like being rendered by my browser right now as we speak and it's adjusting how the page looks if I change this little style parameter here say I want it to be 150 you see how it has some Downstream effect on the module itself now that's the code it's not for people what what's for people is this right the stuff we see on the left hand side the right is code that's for machines the left is for us so in order to convert that stuff on the right into the stuff on the left um your code the HTML needs to be rendered by your browser there needs to be some process that takes place and that's really all browsers do browsers will take in HTML they'll take in code and then they will do some process on it and then they will convert that into what we see on the left hand side here so the job of chrome Safari Netscape Internet Explorer if uh y'all are still with me is to read the HTML the web page and then to turn it into something pretty for human beings so that's that now we need to talk a little bit about clients and servers and I've mentioned this in a previous video on HTTP requests if you guys watch that you'll already know all about this but just to reiterate your computer is a client okay in order to access a website it needs to send a request over to a server this is an HTTP request a hypertext transfer protocol request and what happens is it sends a request to the server and inside of that request it says hey how's it going I want to get the full HTML of this website can you send it to me can you send it to this web address and the server just because it they speak in a very specific protocol and all this stuff's been worked out many many years ago um says okay great and then it sends the HTML back to your computer then it's rendered in your browser as we mentioned earlier and then because your computer screen is made up a bunch of pixels and there's some relationship between the pixels on the screen and the area and placement of the HTML on your browser um that is what allows you to visualize them and you know see elements on the page and drag things around and exit out of them and that sort of deal so we need to understand client server terminology because it's used all the time in web scraping and I just want to make it clear that there's nothing complex about it your computer's the client the server is just some resource out there on the cloud really it's just some big rack which is running your website more or less let's talk a little bit about the static versus Dynamic websites because now we know everything that we need in order to make this work static websites are very easy to scrape I scraped them in my previous video I've scraped them hundreds of thousands of times by now um and I can build a scraper for a static website in less than 30 seconds flat which I'll show you in a moment Dynamic websites are a bit harder and make.com doesn't support them natively okay you can't natively scrape a dynamic website using make.com you need to use a third-party platform luckily most of these thirdparty platforms work with make.com so you can just drag a little module in connect it to that third party platform and basically the rest of your flow is the exact same in this video I'm going to show you how to do both because both are important now in terms of what static and dynamic actually mean in this context all a static website is is it just means that there is literally a file somewhere on a hard disk somewhere right on that server with all of the HTML of the website it's static because it doesn't change it just exists and what happens is when you send the request to the server the server just says oh okay great we got this file just take the file here you go it's the same file every time basically which is why it's static it's not changing these are easy to scrape as you can imagine because all you have to do is you just type in the URL stands for Universal resource locator you pump it into your make.com request module then it goes and it grabs the website so if I were to do this live in front of you I said 30 seconds right I got a couple of other sections here but let me zoom in if I wanted to get the data of my own website all I would do is choose change this to Raw change this to Jason leave everything else same then I right click and I press run and if I zoom in here and go to data I have all of the HTML the website available to me now to do stuff with I wasn't kidding when I said it was easy to scrape a website with make.com especially if it's static if it's a dynamic website it's no it's no longer just one HTML file somewhere in a server what it is is the website is a set of rules it's usually rules in a different programming language and what happens is you will send your request your request will include some information about who you are um the type of browser you're using some information you're requesting whatever and then the server will take that information and then it will apply it to the big set of rules and then it will generate HTML a new essentially and then send it back to you now because the HTML isn't there because it you know it's something that that first of all takes a little bit of time although there's a big debate on what's faster um server side or or server generated or static generated um but first of all it needs to be generated um but essentially what that means is when you make the request with something like make um you don't actually get the the whole you don't get all of the data uh you get a little thing that says JavaScript is not enabled for this browser or you know there's nothing at this resource or you just get like sort of an empty website and so you can't actually scrape those in make.com I can't just go back and do that for you in like half a second unfortunately but I'll still show you a way to do so and to solve that problem for you so that's the difference between static and dynamic if you know that you know more than 99% of people do about web scraping on the next thing I want to talk about and really the last thing I want to talk about before we can actually jump into this puppy and start scraping let me just move these around a little bit it's bothering me uh is parsing so when you scrape a website notice what we did back there we typed in my website left click. a then we pressed run and then what did we get we got this this is just a bunch of HTML now HTML is nice and all but as you mentioned it's not for people it's for machines right it's for browser it's to structure the website and all that stuff realistically if I'm going to left click. which is right over here my website and I want to scrape something I don't actually care about scraping the HTML what I want to scrape is I want to scrape the the page links I want to scrape the description I want to scrape the the testimonials I want to scrape the pricing right I want to scrape the case studies that's what I care about I don't care about the HTML the thing is it's not what I wanted to do let's go oh man we got a lot of tabs over so in order to do that you need to pull out the data from the HTML you need to turn it into some type of structured object or something and that's what parsing is that's what parsing allows you to do parsing just refers to the process of breaking down the HTML into its tiny little parts and then pulling out the that you want so I want the text I'm pulling it out I want the pricing I'm looking for stuff with a number sign pulling it out or I'm looking for a specific element or tag or div or or whatnot now in the past as you can imagine this is pretty difficult right like you used to have to create a ton of specific rules for literally every element you had to you know if you wanted to get like an email address you had to like type an email address identifier if you wanted to grab the pricing you needed to like go and find the specific element or the division and then pump that in these days it's actually really easy and you can do it 99% of the time just with AI you literally just say hey extract this element I'm going to show you in a second hey extract this element um output only this element here's a whole website and then it'll go and it'll find it for you there's also a bunch of platforms out there specifically one called data for SEO that make this a lot easier as well they structure the outputs for you um and I'll I'll walk through all of this over the course the next little bit but first i'm going to drink my tea because I'm realizing that it's getting quite cold o that is a zingy tabag all right so now that we've covered all of this let's actually go into scraping some sites live let me actually show you what this looks like what we're going to do is we're going to use the HTTP module just like I just did and then we're going to do three examples the first is we're going to scrape contact details from our website so you are you have a big list of websites and you're trying to pull the email address of people at every single one of these websites well you can actually do that yourself just by creating a little web scraper that takes you less than 5 minutes to do which I'll show you how to do in a minute the next thing we're going do is we're going to feed a website into um our make.com flow and we're going to have our flow tell us something about the website it's going to convert this big site into structured data using artificial intelligence the value there is you can use the structured data in literally any other flow that you want you can convert this into whatever type of structure data that you want you could have it tell you the number of pages you could have it tell you what the website is about you could have it tell you the average price of the service what the testimonials are about what the testimonials say what what they're like what sort of images are on the website you could have it tell you all this stuff basically then you could feed that into a flow maybe to personalize some Outreach to them or maybe to do a competitor analysis and give you a quick quick lay of the land in 5 seconds or something like that the last thing I'm going to do is I'm just going to grab a bunch of Fiverr listings Fiverr is a very popular freelance platform and I just want to show you how you can apply everything I'm going to show you today in make.com to like actually go out there and then scrape a bunch of fiber listings which is a very like large tabular data uh which then we can use to do cool stuff with like maybe you wanted to Market to each of these Fiverr guys or maybe you're making a new Fiverr profile and you wanted to I don't know like very quickly click a few buttons and then get a whole lay of the land again so without further Ado let's jump back into our make.com scenario editor as you see I was playing around with this just before the video and it's kind of messy and dirty and I don't really like it but I do need a few of these modules what I'm going to do here first I'm just going to delete this I'm just going to delete that and then that as well and then um our first example is going to be scraping contact details so I'm going to call this one scraping contact details then our second example here is going to be oops it's not what I wanted to do um take extracting structured data right which I've already done and then the third one let's just call this uh grabbing Fiverr listings that looks good it's just called a scraping Fiverr and then after I'm done this I'm actually going to save this blueprint and give it all to y'all so you know everything that I do and you can just use this as like a training playground if you want to all right so what websites in particular are we going to scrape well I was looking for a website that had contact details on it and I just typed in Calgary Plumbing Calgary is the name of the city that Elon and uh I found like 20 websites very quickly using Google Maps I literally just opened up the top five and like sorry top the top 20 and something like uh 15 out of the 20 of them had contact details on them so that's great if you wanted toate a scraper you can imagine how you could scrape the phone number really easily or you could scrape the email address really easily obviously we could do this manually you could pay somebody in a lower cost of living country a few dollars an hour to go out and do this for you but you could do it 100 times faster and probably with 100 times better data quality honestly if you just do it automatically not to mention it would be like basically free you know be the cost of Ops which in our case would probably be like 50 cents or something so let me show you how not to just scrape data from this website but how to scrape data from any website that you feed in that's static that fulfills uh the parameters that I mentioned earlier all right so I'm going to go to our make scenario here called scraping contact details and let me just exit out of this um full screen mode so I can actually copy the URL the copy the URL of this website is icon Plumbing so I'm going to go over here paste that in the very first thing we're going to do is we're just going to run this the reason why is because you should always run your HTTP request once to see what the the data looks like see if there any gotas see if there's anything weird a lot of the time you'll see whether it's a static or dynamic website by doing this um and we can obviously put in rules later on that will like do this for us but um this looks like a very simple static website you see there's tons of data here there's meta data right there's the title of the website right over here in these title tags if we were to scroll all the way down there'd be a bunch of text and that sort of thing now what we want is we want the email address right so basically we want to take all this and we want to feed this in then we want to grab this the problem though is right now this is a bunch of HTML and there are two issues with this one it's really long like this is 2451 characters so if we wanted to use this in some flow we'd have to consume 2451 characters worth of that flow this isn't a problem if it's like a a parser which I'll show you quickly uh in in a moment but if it's an AI then that's a problem because it's 24,000 characters it's going to cost us a fair amount every time we run this because of token usage so the first thing that we want to do is we basically just want to take all the HTML away and in order to do that I'm going to go click on this Orange module here and then click HTML to text this is going to take in all of this ugly HTML which again is not for humans it's for machines and it's just going to convert this into a bunch of text and now basically what it's going to do is it's just going to give us a bunch of links and these links are going to be in these brackets here and then it's just going to give us a bunch of text our team read our FAQs giving back Plumbing if we were to scroll all the way down here you'll see that there's all of the text on the page and most importantly if we scroll even further down you'll see that we have our email address and the email address is put in twice um just because they added a link to the email so you just give it a click and then open up your email then they also have the email address same thing with the phone number so the question now becomes how do I extract that well there's a really simple and easy way the first is AI you could do this yourself in a few seconds again I'm literally going to be building on a general purpose scrape before you that works on any static website to extract an email address and I'm going to do it live you could do this with AI literally just by clicking on your AI of choice so I'm I'm using anthropic Cloud right now because I find the quality is a little bit higher personally these days this may be different by the time you're watching this maybe gp4 is back or maybe there's some new model um I go to create a prompt over here then what I do is I'm just going to connect this I pick Claude 3.5 I make the max tokens 4096 and then all I do is I say hey let me move this over here extract the email address from the following website data output only the email address your only token should be their email address an at symbol and then their domain you don't have to use this prompt I just wanted to get this prompt right the first time because I'm not giving it any examples of what I'm looking for and then all I'm going to do is I'm going to feed in the text the website the system prompt is your a helpful intelligent web scraping assistant you find emails and don't put out you don't output anything else and that's it okay so this is going to be extract email I'm going to give this a run and let's see what happened you'll notice that this is the longest taking module here but essentially we started with our HTML right it's super long and ugly and not really meaningful to us then we went to this text parser and we made it a little bit nicer to look at but it's still not that great what we want is we just want the email and what you see is there's this text response but um variable in the output which has the email that we're looking for service iplumb doca so what we could do is we could run this and then what we could do is we could have let's say a big list of um I don't know instead of Plum let's just let's just call this like websites and then we could output the email address of every single one of these so what I'm going to do is I'm just going to use this Google sheet just for um example purposes for two or three of these is I'm going to feed in instead of me doing this on a one by one basis right this is just for testing I'm going to feed in a list of these and I'm going to have um gp4 sorry uh Claud um find the the contact details and then I'm going to have it update that same spreadsheet so the way to do this is I'm going to use a search rows module then I'm just going to map this to my specific Google sheet which I'm going to call big list of websites very fancy big list of websites there we go we're going to go sheet one and then that's that and delete the filter just click okay now if I right click on this press one sorry I think there's an error cuz I closed a little too fast I'm a little um too fast with my fingers here make sometimes needs a second there you go so I'm going to right click press run this model only and then oh sorry I didn't actually put a a website there that's my B I'm going to go back over here put a URL in and then why don't I go Calgary Plumbing again let's just see what other websites there were because I need I need at least two websites right otherwise it's going to be sort of meaningless let's go Calgary Plumbing and then why don't I go to maps and why don't I just pick like the first three plumbers let's go here we'll go website website website website okay so this is icon right okay we have another one here Mr Mike's Plumbing let's do that so I'm just going to do this in the case of two but you can imagine how you could easily scale this up to the case of 30 or 40 you can also imagine how you could easily get another scraper to um populate this and then use this to pump into our our other scraper and then you know get us all the emails so I just gave this a quick run just to make sure I could get the specific one I'm looking for and now I'm going to change the maximum number of return rows to two and if I do this it will then go through every single one of them now the flow right now is going to scrape the contact details parse the text extract the email but what I want to do is I want to update my row basically I want to take the specific row and then I want to feed that email back in so I'm going to go over here and then I'm going to call this big list of websites I'm going to map the sheet as per usual looks like it's going to rain soon that's unfortunate it's goingon to go for a lovely walk that ain't happening uh we're going to put the row number in here we're going to grab this row number from the first row that way if it gets a row for Icon Plumbing when we update it it's going to update the email address column and then we're going to grab the text response from Claude we're not going to change URL we don't need to update that okay now we're going to run this twice okay just ran once updated the column and if we go back here we'll see that why is it doing service at Icon oh sorry I hardcoded the URL my bad my bad um let's feed in the URL classic all right and we're going to do this one more time all right icon plumbing and then General Mr Mike's Plumbing all right these areca because I'm Canada but yeah these works for for any sort of website out there now imagine if we failed in 100 of these maybe we'd get like 50 um email addresses uh you've essentially just built something that people on the internet charge hundreds or maybe thousands of dollars for if I'm being honest and as you see this is something that realistically you can build in three minutes or so very very simple and straightforward so I mentioned that this was Ai and this was kind of costly it was very easy to do and to set up but it sort of costs a fair amount of money how can we do this in the more General case well I'll show you the way that you do it in the more General case is instead of using AI right maybe you use AI really quickly to determine sorry let me just unlink this to determine whether or not the flow works and then after you actually build a dedicated infrastructure for it but the way that you do so is you use um what's called uh a reg a regex I was about to say regx but the last video where I talked about Rex I got shat on people like it's regex man I'm like I'm sorry sorry I'm Canadian uh anyway so um our reg X is just like a uh it's a pattern matcher and what it'll do is it will take in this whole string here and it will simply apply a couple of rules and it'll look for emails that are formatted like this something an at sign something else dot something else now I have one here um that I've just validated works for the vast majority of domains I've tried it on I've tried it on over 100 domains I use this one basically every time I want to do this statically um so I'm just going to copy this and put this in the description as the email regx but basically what's going to happen is I'm going to feed in the text from the previous module and let's why don't we just run this as an example first before we update the row and then it'll go through and it will do the exact same thing that we did before it's just going to go one step further and it'll also give us the name and then the domain and the reason why I split this up in a name and domain in in the regx and you'll do it too you'll get this too if you just use the same module um is by doing this uh if this is a first name I can pass this into Ai and and have it say hey check to see if this is somebody's first name if it is update the first name variable so this way I don't just get the email address I also get one additional piece of data which is their first name because a lot of people will sign off with their first name Josh at Icon Plumbing Peter left click. that sort of thing then when I put this into an email sequence now I can say hey Peter I don't have to say hello dear hiring manager whatever think you get my point so if you wanted to map this full flow out I would just feed in this to the email address module then I'd run this thing one more time and I mean you know I didn't delete the email address so it's not going to update but I think you guys get my point we just ran through the scenario one more time now in order to do this magic module you got to go to this Orange um section again and then what you need is match pattern this is the module that I selected up here and this is the one that you'll need in order to make work and basically your regex or this little code string um this goes up at the very top and then the textt that you supply goes down to the bottom if you wanted to look for multiple instances of email addresses let's say there's like five email addresses on a domain then just go Global match and turn that on I'm I was only looking for one and I was comfortable stopping after the first so that's where I stopped all right if you're still with me at this point we've done something pretty miraculous and incredible in make.com and it really took us just five or 10 minutes um I've shown you how to take a giant list of websites excuse me iterate over those websites to grab the URLs and then scrape those website HTML scrape the HTML of those websites rather feed that HTML into an HTML detex module and then extract only the thing that we want from the this system here is something that if you go on upwork or Fiverr and you say build me a system that scrapes company emails or or whatever this is basically the exact same system they're going to be building for you they might just do it in code or they might do it in make.com but it will not look as sexy as this so we did something pretty cool here but this is just the beginning of scraping because this is the simplest case and this is one of the simplest things that we could do which is to scrape contact details okay why don't we kick this up a notch as Gordon Ramsey says why don't we just add a little bit of olive oil and why don't we get some structured data from these websites instead so what do I mean by structured data well why don't we just actually use this Google sheet I wasn't planning on doing it but that'd be pretty cool iterating over stuff is way better than just like hard coding it in because this immediately shows you guys that you can apply this to other data sources which is pretty sick okay so what we're going to do in this instance is we're going to scrape the contact sorry we're going to scrape the HTML the website instead now what we're going to do is instead of getting the contact details we're going to try and get something about the site we're going to have them tell us about them we're going to have them give us a brief little summary let's say what they do and we can use that brief little summary later on and some sort of later on flow to maybe personalize some Outreach to tell us something about our competitors that sort of deal so how are we going to do this well we're going to follow the exact same structure we did up here where we have the HTML the website then we have a text parser to turn that into text and then we're going to do something else with then we're going to dump it back in our sheet so I'm going to go over to my AI module is somewhere around here I guess I deleted it let's just use the same one we did to extract the emails instead of extract email I'll say gener right summary and then if we're going to um do this we need to change the prompt so I'm just going to dive in now what I'm going to say is the following is a website is text from a website scrape in Json return a summary of the website um well I mean we're probably not going to be able to do who because most websites don't have that name but maybe like what the website does okay why don't we do this we'll say what the website does and their key value props return in this format we're going to say um description uh and then we'll go double quotes if you haven't seen my previous tutorial on Json I recommend it then we're going to do um maybe we'll just do description and then we'll do value props then I'll just say value props go here and then description goes here the reason why I'm doing this um is because I don't want it to like output a bunch of sub things like a bunch of sub objects or whatnot I'll say don't output anything but the Json okay great and then I'm just going to feed in if you think about it um I'm just going to feed in the oh jeez you know I forgot to parse the text my bad let me go over here grab this paste that in you guys are probably watching for the last minute with like a palm in your face okay let's turn that into text first let's feed that in here okay great and then I'll say your helpful intelligent web scraping assistant you extract data and only output Json awesome okay now let's see how this works on our first example we got to do a couple more things to make this flow functional but I'm going to walk you through regardless okay as we see here um we got a pretty intense or deep description of what's going on and this looks good to me looks like they have eight different Valley props honestly a lot but whatever the thing is sorry for zooming in and out in so fast um this is in Json format and we want to convert this into a format that we can actually use so to do this what you need to do is click on this little button type Json find that and then use parse Json what you want to feed in as the input is this text response this will parse it and turn it into mappable fields that we can use later on I'll show you what I mean in a second let me unlink this let me click run now instead of our output being text our output is going to be a bunch of nicely structured Json that we can massage and do whatever the heck we want with see this there was that nicely formatted Json I was mentioning now what we can do is we can feed that back into our sheet and then I don't know maybe instead of email or why don't we do this uh we'll do description and then we'll do value propositions right and what we're going to do is we're going to take the same data we're going to update the row number which I grab here and then we're not going to update email address what we're going to update is we're going to grab description and then value props note that these don't have headers because I need to update the module but I know that it's still column C and column d right column C column D um so the scenario is still going to work regardless and what I want to do is I want to go back here and change the maximum number of return rows to two because I wanted to iterate over all of my data okay let's give this a quick little run looks like we got number one done we got number two done wonderful now I'm not going to stretch these out super wide but I hope it's clear that we got descriptions of all these companies and we also got the value propositions and you know we could do some better training of the model or better examples to avoid the fact that this one has a one in it and this one doesn't right this one's listing them sort of 1 2 3 four 5 this one doesn't this one isn't but uh it's not that big of a deal as of right now I think you guys get where I'm coming from and that's the most important thing all right so that is number two why don't we do number three now which is to grab a bunch of Fiverr listings now Fiverr seems pretty int ating for a lot of people right I mean oh I guess I already got it right over here um because it's pretty big and pretty jam-packed freaking website if I'm honest a lot of people think that make doesn't work for this well I'll show you that it does when I go to digital marketing here just going to click off and let's just hypothetically say we want to scrape this digital marketing page um why don't we go to social media marketing specifically and we just want to like I don't know get a bunch of these and then do something with them so I'm going to give this a copy and then I'm going to go back to scraping Fiverr I'm just going to feed this in as the URL and then I'm going to scrape this text and then I'm going to use Ai and we're not using parsers here because I just want to be able to do this quickly you can imagine now you could build parsers to extract the various pieces of information you wanted to and let's see what sort of data we have here I think we have the name I think we have the I don't know if we'll be able to grab the level can we grab the level I don't think so because the this level two is it a design or oh no no no okay no we we actually can't grab level so let's grab the name the level and let's grab the title and then the cost let's do that so let's scrape all these name the level of the freelancer the title of the job and the cost return in this format we'll go name name goes here we'll go level we'll go level goes here we'll go uh title we'll go title goes here and then we'll go cost and then we'll go cost goes here I think my Jason's right I don't know for sure so I'm going to go to Json formatter give this thing a little paste and then I'm going to click format okay looks good to me go back here and just make it look nice and pretty and then as input I'm going to feed in this text parser so this is just sort of a a little test we're going to see how this works first um and I'm not going to attach it to a Google sheet or anything all I'm going to do is I'm going to grab the parsejson module attach it and I'll just show you guys that you can do cool stuff with this dat and actually why don't we also do one more thing let's set set a variable to we'll put that at the very end for now I'm just going to say x is equal to Y okay um looks like we oh sorry about that I did not map the correct data because I haven't actually used this yet and I mapped that I think no I didn't need to map the text response here awesome let's give this k a little run through let's see what happens so we just scraped Fiverr we got a 200 request and we got a ton of HTML it's very very long right odds are when I see HTML that's this long I got what I needed okay it looks like there was an issue here but that's all right um the next thing is we converted all that to text it's still hella long the next thing is we generated a summary of that text it looks like we only got one one person which kind of sucks I think it's because the input was way too long yeah so I think the input was a little too long um and we probably need to train this a little bit if we want to 36,000 tokens I'm be paying that bill pretty soon uh it was so long that it actually screwed up and provided the Json so yeah that's not very good okay so when we scrape a data resource that's this long what we basically need to do is we need to find a way to slice this data into subsections and so that's what I'm going to do here I'm going to find a way that I can slice this data which is super crazy long into subsections and then feed in that subsection um iteratively into uh the thing that I'm I'm looking for okay so let's see here we got a bunch of categories okay seller details all right sort by best okay so first of all we don't need anything above uh yeah we don't need anything above sort by bestselling so the very first thing I'm going to do is I'm I'm going to grab this and then I'm going to add another module here Zoom this out a bit I'll use the set variable module uh and you know what let's actually do two set variable modules here and first I'm just going to call this split um header I'm going to feed this in and then I'm just going to split this entire thing by this text sort by bestselling so I'm going to feed in oh sorry the name should not be this uh this should just be let's just say Sellers and then I'm going to split this by the text sort by bestselling I know this looks kind of weird right now but in order to test this I'm going to download the output bundles copy all of this then I'm just going to run this paste in all of that text and I'll just see if I can turn this into two okay great I can turn this into too notice how the second part is a lot easier wonderful um that looks pretty good to me now that I've done this what we need to do and let me actually just split into sellers is I'm going to wrap this in a get I'm only going to grab the second result of that array and then I'm just going to add a different a second variable here just for Simplicity in reality I would not actually normally do this I would just do it all in in one module but what I'm going to do now that I have all of this is I'm going to look at the text I'm going to look for a way that I could scrape this sorry that I could separate this it looks like there are two new lines here and it looks like if I were to look for these new lines there are 21 on this page so I bet you if I were to split this whole thing sellers by new line new line I would get 21 different um results then for each of those 21 results I would feed that into Jason and then we should be able to get exactly what we're looking for here okay let's give this a quick little try and I actually don't need this okay give this a run and we'll see how our wonderful Fiverr scraper which I developed in five minutes is five minutes for Fiverr looks like the source is not valid Jason there's probably a reason why H okay no the splitting didn't didn't work actually uh why didn't the splitting work that's interesting we grabbed the data which looks like it did work oh I know why it's because when you feed in new lines um it actually into a set variable module it actually parses them this is a a very much like a gotcha in make uh so you need to instead of splitting by the back slash n where the heck is this you actually need to split by the presence of an Elon and I think I need to split by two I'm not sure we're going to give it a try I wonder how many times I could scrape fiber before they rate limit me okay I had to take a little break there just to work out the logic but uh just finish it up going to give it a quick run here and we're just going to grab the first record um so that we we got an output uh okay so essentially what I did was nothing changed to the scraping module same thing nothing changed to the text parsing module same thing um what I'm doing here is we're getting splitting the text sorting by best selling so nothing changed there I'm then splitting these by the presence of a new line plus a new line you don't actually need this plus I find I think make just like inserted that for me um and then what I did is I changed the prompt here um to a couple of things one I said do not output anything but Jason your first token should be this I also said of there multiple records return the first this is just sort of a quick and dirty hack um you could also add a pre-step here where you preformat multiple records into one record and then just iterate through them and then I just gave it like one example of what I wanted to do uh basically if you want a a to work for you just give it an example it'll eliminate 99% of the BS so I gave it an example um then feeding in the split text and then I changed the system prompt so it includes your first character is always that what you end up with that is um you'll feed in the the text and then you'll get an output um you know in Json as a result which is quite nice so we have it right there and then when you feed that into the parse Json module right just the the pure text response what you get get is you get these variables which then you can later use to you know fill out a spreadsheet or something like that so uh yeah you know I can follow the exact same flow that I've done here with the Google Sheets I'm not going to just for brevity and I want to make sure this video isn't like 17 hours long but I think you guys get my point you can scrape more or less any data source that you want as long as it's static um sorry you can scrape any data source you want with the HTTP request module as long as it's static and it doesn't really matter how complex it is um we saw examples from getting like one piece of data from one website to getting you know multiple pieces of data um from you know like a more grandiose uh typically like secure website like Fiverr which if you sort of think about it like logically speaking should be one of the highest security websites just given the volume and that sort of thing but that is static websites if you made it up until this point congrad you freaking relations we're now going to dive into Dynamic websites which are similar they're really not that different um all we have to do is that initial scraping step where we get the HTML like right over here we just have to replace that module essentially or we have to fill it in with some type of like API um call to a service that does this for us so you know I mentioned that the majority of websites are static nowadays and that's just because static websites are faster to load and typically less complicated for small to midsize businesses if you are starting I don't know like a plumbing store for instance you're probably not going to go with some big Dynamic single page application you're not going to need a bunch of database logic you're not going to need to do any of that stuff um there are Frameworks out there like View and react that do all this crap for you sorry um there are Frameworks out there like nextjs and n .js um which sort of just like do the static Generation stuff for you if you have a coder doing it um if you're doing like WordPress nowadays I'm pretty sure the vast majority of Wordpress uh websites are rendered to static Pages uh you know basically any other major um website platform is going to be doing that which is quite nice the only real instance in which you need Dynamic websites are usually when you're developing like some type of Enterprise or big or like multi-million dollar application one where you really need to format the data that's coming in you need to like have the rules of that HTML I'll be different maybe for different users depending on different places in the world or depending on like very short-term updates you want to make to your application with it displays and that sort of thing so that's a mouthful but essentially I just mean it's harder but it's certainly doable um your data source instead of being the HTP request module is just going to be um apify which is a thirdparty scraping tool which I talk about quite often on this platform I love it it's amazing they also give you $5 of like free Tri credits uh which last for me anyway for like the last couple of months or you can use hidden apis I've talked about hidden apis in a previous video I'm just going to like Drop That and Link that down below um I'm also going to mention this really quickly um in the next module in the next page so um you know if you guys want more information on if you want like a full guide just dedicated to Hidden apis then check that out any who the rest of the flow is going to be the same I got two examples here similar to the three examples previously um where we had the icon Plumbing my own website and then Fiverr what I'm going to be doing here is I'm going to be scraping these two cryptoart and billybilly so I'm just going to open up crypto art over here and I think you guys will kind of get the idea it's an nftr collection that tells the history of ethereum very cool painting 9,893 all these blocks like look it's like a super deep art piece here and you can I guess buy a bunch of these abstract um geometric shapes and and that sort of thing I personally uh did not get into the nft craze I mean calling it a craze but I know a ton of people did make a ton of legit money on it but I personally didn't really get into it so my knowledge stops here looks pretty freaking cool let's hypthetically say we wanted to scrape this resource how are we going to do it well first things first I'm going to just duplicate like for the purposes of this I'm just going to duplicate these two and I'll go down here I'm going to call this for scraping uh what is it crypto art and then I'm going to move the schedule here I'll paste oops that's not what I want to do I got to paste in this URL I'm going to get it and the reason I'm doing this is just to show you if I right click and press run module only um this is what I get so doc type HTML blah blah blah I get the title of the website which is nice I get the meta description which is nice but then I get basically absolutely nothing else and this is more or less what's going to happen if you are loading a dynamically generated website like this you're not going to get the stuff that you're looking for you're just going to get a wrapper right you're just going to get like um everything that goes around that HTML so that's unfortunate we can't actually scrape this there's no real data here like for instance um painting number four like if I look at painting number four there's no number four here there's no text that represents that number four meaning that we just haven't gotten the HTML yet U basically what happens is after you load this page there's some JavaScript on the page there's some function that calls that database or receives it and then and then the code of the website is built uh we don't really have that yet which is why we can't access it all right so how exactly do you go about doing this with apify well it's pretty easy just go to apify Doom sign in here I got a free plan account with like $5 worth of tokens or something then go to store and then type in web scraper that's the one that you want and then what you want to do here is just type in the URL the website this is just an example so I'll show you how to do it with one and then we can talk about how to do it with multiple websites and then you need a little bit of code in order for this to work like if you want to store the page HTML you do need some code um what I do is I just copied some code from another one so I'm going to do this and I'll say page HTML so what you need is you need to basically say const page HTML equals document do doent element. outerhtml what this is going to do is it's going to in JavaScript the document is just the entire object of the page the page instance document element is just a way to select a particular element on the page and outer HTML is just the selector of the whole page is outer HTML um so that's HTML including the HTML tag which is important what we want to do is we just want to return this too so I'm just going to go down here and then just go page HTML this is now the function of the page this is now going to run this code basically every time it works through what we want it to work through uh which is pretty sick so then all you do is you press start and it's actually going to go out there and it's going to run the website and then deliver all of the code to us which is pretty cool when this happens you're going to see the memory go from 0 to 4 gigabytes that's totally normal this is now just running the scraper and you only have a certain number of me or amount of memory that's allotted to you uh and so by doing this um apfi is just letting you know that hey man you got something going on right now uh we're going to wait until it finishes you then obviously have a bunch of uh console log things so it's going through this and then what we want to do is I'm going to go to Output we're going to do Json and then notice this um item right here HTML head blah blah this is the one that we want so just going to export both of these results in Json and I'm just going to actually I don't want to export the results on my do I'm just going to preview them okay and now if we type number four oops um painting number four now if we type number four notice how we actually get the painting number four that we didn't get before um this is just all the code for the whole website so since we're outputting it here we can now get it in in make.com as well like we're we're not limited to doing this in apify I just showed you how to do this manually apify just to show you how this works um all you have to do is click on this little button go to apify and then watch actor runs that's what you're looking for and then you do have to connect your hook so you have to like connect apify to um make.com it's really simple they have an API key just type in apify API key actually no I don't even think you need the API key sorry you literally just go here no no you do you go here you click connection add and then you add in your API token yeah I just realized I don't have it so let's actually do this for me it's probably the best way to show you going to want to copy this and then actually you know what I'm going to add oh weird I don't know if I have the ability to do this oh yeah I can uh this one's just going to be called like uh YouTube example just that I remember to change it or delete it later I'm going to copy this and then we're going to go back here I'm just going to paste in this is going to be called like Nick left click okay cool save wonderful wonderful then we go to actor and all I want is I want that web scraper that I had a second so apify web scraper right give that a click save now the other thing you need to do is go to apify and then you need to get data set items for whatever reason apify splits this into two different parts there's um the watch actor run which is the trigger and then this actually helps you get the specific items like the ones that we saw earlier and and in order to get this just go to search items type in data set and you want default data set ID format Json limit 100 all that stuff is cool um and then now we can actually run our flow uh I'm just going to go back into apify for now and then go to the specific actor in question which is web scraper I think I have some code here that like adds an extra example so I'm not actually going to add that in I don't think yeah I don't think I'm going to do that I think I can just delete this and it should be fine um and basically what I'm doing right now is I'm running this manually so I'm running this like myself um but you can also run this using apify and Jason what you would do is you would go over here then you'd go run an actor actually first you'd run the actor maybe you do that in a different scenario um you know so You' go web scraper and then you would go here to where were we right here and you just copy all this Json you paste it in here oops sorry should be good right here um and this is uh I know this doesn't look very well formatted we might be able to format this better if we go to Jason format let me see yeah it doesn't really format it because of all these new lines but that's all right um You you feed this input Json in and then what you want is uh where it says crypto art. this is where you'd feed in your list of URLs um and notice how this is an array in Jason so all you would do is you just feed in a big list of URLs formatted just like this then you'd be able to run it you run it in a different scenario and then you catch it in this scenario so I'm just going to catch it in this scenario for Simplicity um but what I'm going to do is I'll I'll watch the ACT runs I'll get the data set it so I'm going to click go over here go save and start and the same thing's going to happen so on the leftand side here where it says memory you're going to see that it's going to go up to 4 gabt um and then it's going to go through that whole scraping procedure I did remove a little bit of code um so just going to double check that that works um we may have like referenced the URL after or something yeah it says one succeeded I go to Output I have the output here what I'm doing now in make because I'm watching that actor run awesome looks like it finished and then in here I got the uh I got the page HTML and this page HTML object all right so that's pretty straightforward now that we have it inside a make what do we do well it's pretty simple hella zooming got a love the Zoomies we'll um put this back here and we'll feed in that page HTML item now because I don't want to wait for this to run again I'm just going to copy this whole thing oh boy this is long as a hell I'm just going to copy this whole thing just for the purposes of this demonstration it's the exact same thing as if I just left this in here and ran it one more time but I'm just going to oh actually you know what no I'm not going to do that I'm going to go back to page HTML then I'm actually just going to hardcode the data set ID instead that way I can still actually like run this module and it's not going to be total uh unlink we'll go back here okay now we're going to run this and not notice that now what we have is the exact same thing that we had before we have like a text Data essentially um so that's pretty valuable as you can imagine and now we can do whatever the heck we want with this um I don't really know what we want to do with this if I'm honest maybe we just wanted to tell us something about the website again so why don't we just copy these two here and even though this is a dynamic website um even though this is a dynamic website right should only output Jason but I guess we're going to find out and then this for9 text response uh even though it's a dynamic website we can now scrape it fully so we don't have to worry about whatever happened um you know uh I received a couple one of the reasons I'm doing this video is cu I received a couple questions on my last one like hey like for some reason when I do this I don't see the HTML on the website well this is how you do it um you can have two routes you could theoretically you wanted to scrape any website use this first route here where you feeding the URLs and then try getting the structured data using HTTP requests and then if you end up getting data that's maybe less than a certain length or maybe you feed this into Ai and say hey is there HTML here if no then return no if yes then return yes then you just use a router and you'd split it into um static website or dynamic website if it it's a dynamic website then you run an apify actor and in another scenario you catch the result and do what I just did over here um and that's how you you know go through the the dynamic arm uh you can also use a a few other services for this there's one called data for SEO data for SEO is pretty good in my community um everybody's now talking about it people really really love this stuff um and essentially the way that it works is just G to close you Nikita oops so sorry Nikita we share the same name how could I do this to you uh yeah so this is just like a powerful API stack for SEO purposes it'll basically allows you to run like what I just ran and apify I'm just doing it through one single module in make.com um it's reasonably straightforward they have a little portal which you can access here just go back here um where you have a certain balance they give you like a free dollar or something like that I apparently have like five years or four and a half years left and then all you need to do to connect this to make is just go through their API documentation which you can find inside of the data for scr module when you try and connect it and then you just need to put in a username and password literally the simplest authentication I've ever seen in my life and then the module that you want is you want pars uh parse page content right over here and this this is just another way to do the same thing U but you'll see the output is going to be a little bit different so what's this website cryptoart just going to copy this URL pasted in here and you're just going to click emulate browser rendering yes um what you're going to see is when I run this I'm not going to get the code of the page I'm actually going to get the parse text already it sort of does both of these things in one so I receive a tasks array big ID path this is just the endpoint path so V3 slon pagecontent parsing live again some API just return a ton of information to you whether or not it's particularly relevant here's a bunch of data and then you here you have the result the result is what matters so we have an items array inside of the data and we have a one and then page content Jesus Lord this is nested what you want is the secondary topic it's an array with 20 items and this is just all of the data on the page that I found so this is just the text Data it literally just goes and finds the text for you you don't actually have to do the whole text HTML thing that I talked about earlier the problem is you'll notice that this is a barried an array with 20 elements you can actually cheat um and just do all of this without having to iterate 20 times uh the simplest way to do it the one that I personally do is I go to Json and then I go transform to Json what I'll do is I'll feed in the object that I want and the object that I want is this page content element and now if I run this look at what happens it's going to go grab the website it's going to do the loading and then voila now I have a ton of HTML here instead so um this is the same thing basically sorry ton of HTML ton of Json instead so now now I have a bunch of Json and what you what can you do with the Json well now you can just feed all this stuff into AI just in one shot so instead of having to iterate over 20 different elements and trying to extract data from each now you just feed it into AI you say hey here's a bunch of Json from a website you know from a website scraper which does the parsing for us uh just like extract the following elements from it or you could try parsing it manually too if you really wanted to presumably I'm looking for something to do with painting so here's the painting one painting two it's buried within an hcore title object you could do a lot with this but I'm not going to hit the nail over the head a million one times but okay so that's how you do it with crypto art how about this other website Billy Billy uh looks like Billy Billy I actually I don't even know what language that is geez I need to work on my Asian knowledge um I think that's Chinese but I maybe like offending like a quarter billion people if it's Korean I don't know probably more than a quarter billion what's the population of Korea it's really really interesting 51 that's it 51 million why did I think there were so many more people living in Korea well I guess that's South Korea so with North Korea it's 76 but anyway um whatever it is a lot of the people on here like anime and let's say you wanted to scrape that damn I'm going to copy this URL we're going to go back here and we're just going to give this a try using uh one of these HTTP request modules also let me just um name these um scraping crypto art I think this one's also scraping crypto art we'll just do like data for SEO I I'm just going to give you all of these modules and this one here uh I'm just going to use as an example to show you what happens when you try and scrip scrape Billy Billy without a like browser on page automation so I got a uh sorry browser whatever the heck it's called Uh Dynamic like web scraping service so I just ran this um and then I got if you scroll down here I got a ton of website data but you'll notice the vast majority of this is just like svgs and stuff like that I'm not actually getting like a lot of the text on the website that I want to get unfortunately it's sort of dynamic and um looks like I do get some card information here um and in hindsight picking a service whose language I do not understand may or may not have been the best choice here neck um but that's okay I'll show you how to do the exact same thing assuming this is a dynamic website just using apify instead as you can imagine nothing different from what we've already done just going to copy all of this paste all of this here okay so scraping Billy Billy uh for the purpose of this I'm just going to assume that we ran this in like um another flow and I'm just going to run this manually myself so I'm going to go over here grab the URL paste the URL in here and then I'm going to click save and start it's now going to go through the website and find me some stuff um what I'm going to do while this is working I'm just going to click run once because I really want to get the oh sorry you know what I'm actually using the wrong data set items uh oh that's rough I'm using the hardcoded data set items from the previous run is that going to be a problem no that's not going to be a problem all right so so it's just running right now um and it just uh I think it just finished I believe not 100% sure yeah it just finished and we go to Output we get a bunch of HTML on the website as well so that's nice uh what I want just for the purpos of the demonstration is I want this default data set ID I'm just going to go back here I'm just going to feed the default data set ID and the only I just want the most recent item so I'm going to set limit to one okay so what I'm going to do in this example is I'm just going to generate a summary again and I skipped ahead a bit just cuz I had some prompt finagling to do the impr prompt is I dragged in the open AI or sorry Cloud uh module here then it says the following is a text from a website scrape and Jason return a summary of the website what they do how they do it let's just do that and the language if the website is in another language make sure to write the summary in English return in this format then I have don't output anything but the Json because I like telling robots what to do clearly and uh I just clicked run once I got the data set items and now it's going to generate the summary and it gave me an example description here along with the language which may be useful if you're doing I don't know some type of parasite campaign where you're going um to content in different languages and you're attempting to reproduce the content in another language you can imagine how that might make sense for markets that are a lot less saturated than let's say the English or the Chinese markets um you know I mean how many people on planers speak English right uh 400 million 1.5 billion to two billion and then there are 1.3 billion Chinese speakers what if you tried to I don't know convert that into Vietnamese um well now you have 86 million people which is I don't know one 15th of of English so you could easily do like a parasite campaign where you're translating content on a website from uh Chinese or English to maybe Vietnamese or something and then populating it on your own uh you could also just have it yeah you could have it just translated word for word you could have it tell you specifically about what page was posted the most recently as you can imagine there are like a million one things that are possible but that is how you scrape Dynamic websites Live Hope you guys appreciated that tutorial um really really love the ability to scrape both static and dynamic and I think that any like worthwhile web scraping solution is just going to have routes that handle both so you're going to have a route that handles um presumably static websites and then if it doesn't work out can have that second like filter that allows you to call a dynamic service like this to do it instead for you I wouldn't just scrape everything using this Dynamic Services simply because it probably be pretty cost prohibitive at higher um operation counts higher lists that's okay all right all right the last thing I want to talk about to really make this a master class I want to talk a little bit about hidden apis what are hidden apis well you know how right now we're scraping HTML if you think about HTML is the output of some process specifically in the case of dynamic websites essentially what happens is the dynamic website will go through that list of rules on the server and it will use that to generate the HTML for us well part of the list of rules will be like the data that it wants to show you so maybe its U uh directory website like Billy Billy presumably all of those fields are stored somewhere in a database anyway and what's happening when you access billyb billy.com is there's some function that goes out calls the database elements and then uses that to populate the HTML well how cool would it be if you could just grab the database elements directly instead of having to do all this BS you could literally just go out and then get everything in perfectly format at Json I'll show you how to do so using an example um what I'm going to do is I'll do this in Loom because Loom actually has a hidden API uh this was just some roast that I did to one of the members of the community as you see here I was pretty freaking Tire wow I don't know if I've ever looked like that much of a douchebag that's so funny uh anyway so I'm going to hold shift command and C I'm going to right click on the page and then press view page Source oops I lied right click on the page and press inspect what inspect is going to do is it's going to open up this menu on the right these Chrome Dev tools uh we don't need this first page what we need is we need the network tab over here what I want to do is I I basically just want to refresh this page want to see what loads what you'll notice what loads is ton of these graph ql calls so basically every line here this is a resource that is being grabbed using a network request just like our own HTTP request module earlier um our browser is making a ton of these little HTTP requests to other services so um I've gone to preview here and I'm just thumbing through all of the different and like all the different resources the files that we're we're getting and you'll see that some of these files are JavaScript some of these files are Json some of these files are HTML some of these fils are just data um right we can just scroll all the way through here and there's just a million in one what's interesting is what you'll find on a lot of websites is a lot of websites when they call a resource um they'll receive a response that looks like this and you'll notice that this is just this is Jason right this is um just Json format and a lot of the time the things that they return in Json if you call an endpoint so if you send a request to an end Point um you can actually get it to return that Json directly to like make.com instead of this and like instead of your browser I know that sounds a little bit complicated right now but let me show you a practical example um I'm just thumbing through all these graphql requests because I'm noticing that these all have the Json that I'm looking for and it looks like most of them like send me organization data that's cool uh user followed stream get workspace setting home state density that's pretty funny they actually collect a ton of data on you and looks like they use this to communicate honestly um they have some data on the video which is pretty cool with some comments they they have a list of comments uh fetch autoc comment display controls fetch video transcrip okay so what I want to do is I want to I want to call this video transcript and I want to get the whole video transcript you see what happens is when the page loads it makes an H sorry it makes an HTTP request to this URL here called caption Source URL and if I paste this in you'll see that I literally get the entire caption of my video so you can imagine that this is a pretty useful thing to do that you you could develop some app that uses the loom API or something like that Loom hidden API and then use it to call the specific resource at this specific URL but if I just copy this URL and then I go back here and I try and do like an HTTP request let's see what happens just going to pump the URL in here with a get press okay I'm going to run okay actually we got the whole thing which is pretty sweet yeah that's pretty sick okay so um that's great so literally this URL here just has the the list of all of our captions so all we have to do now is we just have to reproduce the same request that resulted in this response so in order to do that go down to payload and what you do is you basically copy um the entire structure of this request payload and then we also have to do is you go to headers here and then you have to copy all of these headers down and the Really the most important one here is this cookie that's what you want so basically what I'm going to do now is I'm going to um go to this URL and I'm going to set up a request here calling the graphql uh the request method is post let's go over here to post um if you guys don't know what I'm doing by the way feel free to check out my um my API call tutorial where I cover all of this and then I'm going to go down here to let's see request header cookies probably the main one so I'm just going to go back here I'll go to Raw Json application Json then I'll just uh actually it's probably a header right so I'm going to go to headers cookie I'll paste this in um it's usually all honestly that we really need we may need some more data actually let me just see here uh okay let's give this a try and if this doesn't work then I'm just going to add the rest of the headers in here but maybe it won then I need to send in some request content as well so we'll go to payload and I'll view this source and I'm just going to copy this as well then what I'm going to do is I'm going right click press run this module once received a um 200 status code we have the data with the actual URL the video that we want to access which is pretty sweet so I just got the transcript myself inm make.com using looms hidden API now if you think about it um what do you need in order to make this work well you need a video ID how do you get the video ID well 3 e3b that's just my URL so if you wanted to do like a cool project with this I'm just like off the top of my head here if anybody here spent more than 30 seconds thinking about this you'd probably be able to come up with something better um but you could just uh I don't know you could scrape a big list of loom URLs maybe there's just some resource out there with a big list of looms or maybe Loom has its own repository with a big list of looms or maybe you have access to some giant database just filled with looms or something like that you feed in these URLs to make.com you parse out the ID you then um have a module like I just showed you the HTP request module where you swap out this video ID with maybe a specific cookie or something like that and then you use that to get the um you essentially use that to get the uh video transcript every single time which is pretty sweet let me see if there's an example Loom that maybe I can use hopefully Loom does its uh video supports using looms um let me see if maybe I have a loom oh you know what I could probably do I'm just going to go to this URL here I'll go I'll go site loom.com share let's do that type this in grab this video here and what I want is I just want this ID so I'm just finding a random video out on the internet and I'm just double checking that I can do this with um a video that isn't just mine just to show you guys and then what I need to feed in is this video ID right here paste that in run this module only and as you see uh you can do this on on literally any video um not just my own video so now I have this person's whole video transcript which is cool uh you could then I don't know maybe you want to like feed that into some other flow which turns that into some blog post or something there's like a million billion trillion different things you can do I'm sure you can see all right so that's that I hope you guys enjoyed this video tutorial masterclass and web scraping on a lot of fun putting it together I think this thoroughly irons some of the holes in my last video which discussed specifically web scraping on a static basis this is now web scraping dynamically which means you can do basically anything and then um hidden apis are part of web scraping which is why I cover them in so much detail but if you guys want more actually have a whole video where I just like run through hidden apis how they work and more or less what I showed you here just in a little bit more detail so yeah web scripts is a really deep topic but what I just showed you was the 8020 um you can use it to determine more or less anything that you want um you can use it to get any sort of static website summarize that data turn that data into something valuable scrape certain entries scrape a you know URLs uh and then ultimately like call graphql hidden apis or other hidden apis to receive things that are valuable like video transfer scrips if you guys have any questions about what I showed feel free to drop a comment down below I know some of this was a little bit dense particularly the end otherwise if you could all do me a favor and like subscribe do all that fun YouTube stuff to help me out and uh get me to the top of the Alo and I'll catch you on the next video thanks so much bye-bye