Screaming Frog SEO Spider Overview

hi today we're going to be talking about screaming frog SEO spider now what is an SEO spider it's a WebCrawler and when you think about the worldwide web the internet itself it is called a web and when you hear a spider a spider is what goes on a web essentially Google is just a web crawler that is crawling different links in going through the web indexing it and giving you the results on a silver platter now what a web crawler does is it emulates Google all Google is is a web crawler and all screaming frog SEO spider does is crawl the web it essentially emulates Google and it gives us information that we can do to help Google bot the little spider that crawls the web all the information it needs this is powerful for diagnosing technical SEO issues and making sure your website is on a solid foundation for all of your SEO initiatives as well as a lot of cool tips and tricks to maximize your organic traffic and scale those SEO rankings so if we jump in here we can check out the screaming frog SEO spider website we can look here at what their pricing is and you can purchase this beautiful tool it's been around for almost ever uh they even have a free version that allows you to crawl 500 URLs so if you have a small website you can get this for free and you can try it out um now the paid version is $260 per year and you can purchase the license there a very fair pricing by this team you can see some of what this does finding broken links analyzing SEO data and redirects duplicate content all kinds of cool stuff but rather than just looking at the website how about we jump into the tool and I'll show you what it can do so we're opening up a new instance of screaming frog SEO spider and you can get an idea of the UI the user interface and what this tool is all about um it may look a little complicated at first I'm sure it does I've been using this tool for five six years so it's kind of home for me but when we first look at this we have a big search box up at the top this is where we're going to input our website so sample.com you're going to put in your website there you can click this big green start button and that's going to start the CW going to start going through this website and just colling all the different URLs now example.com is a one-page website so we don't have much here this is where the URLs will crawl and index now if we take our example website refill finder.com and we run this through it it's going to wait and then it's going to start crawling all the URLs you're going to see those populate here so now if this is your website.com then you can see all of your different pages this is the URL Viewpoint and this is where you can get a 30,000 foot view of your website all of its pages and see how they interact with each other how all the SEO data like SEO meta title meta description H1 H2 how those are all present on the pages you could also connect this to Google search console data Google analytics data and even Google page speed data this is very powerful as you can see but let's not get ahead of ourselves and let's get down to the very basics of how this tool works and how the U UI is organized so once we have some sort of crawl running here we see this is our URL Port but right now we have some kind of strange URLs so we have image URLs all kinds of stuff if we go on here in this overview tab you can see all the URLs encountered and we can see the crawl data here we're set to all if we just switch to HTML these are your all of your real pages so HTML pages are the real website pages that you're used to seeing we can rightclick this open in a browser and it opened on my second screen but if we see here we can see the page it's just one of the page is on our example website and so here is the basic Viewpoint you know if we pause this this website's a little large so we'll pause it for here we can see we've already crawled a thousand pages and so for large e-commerce websites as well as local websites and SAS websites you really want to see all of your URLs and you need to know them from a 30,000 Viewpoint perspective this is what this URL Viewpoint gives you and we can dig way deeper into it so if we C scroll down on this overview tab we can see different file types which aren't very interesting we really just care about our HTML Pages for the basics we can see security so you may have HTTP URLs which we see seem to have some these are not good um it does tell us the status code so it tells us like here we are redirecting from htdp to https this is critical for SEO HTTP is an old security measure you can't have those on your website and so this just give us some of the security details so looks like there's some issues we may want to resolve for this test website the next tab that's really important is the response codes um overview section here and you do have internal and external typically we're just interested in the internal and here is what those status codes represent now if you're familiar with SEO you probably know what status codes are but if you're not we can briefly review them a 404 page essentially means that it's an empty page so if we open this here with our right click we can see page not found you can't have broken pages on your website or the Google crawl is going to get very confused it's going to be landing that little web spider we talked about is going to be landing on error pages and that's going to reflect negatively on your website so within this response section we can see our 4xx errors which is all of the 404 Pages or the 430 Pages um whatever the status code is we'll investigate that and see what it meets we're also probably going to have internal links towards these pages and these are really stuff we want to clean up because you can't have Google crawling links on a website that go to a 4 or four page we can also see our 3xx Pages which are ones that are redirected currently you still normally want to clean up the internal links going towards these Pages scroll down this list and you'll see URL page titles meta descriptions H1 these are all topics that we really need to think about for SEO and so this overview tab just lets us see all of this data for all of our URLs and the important one is the page title we just spoke with Kevin indig and this is one of the top ranking signals we've known this for a long time this tab will just tell you all of your page titles by the URL what's so great is that any of these tables you can export directly to Google Sheets or to a CSV or to an Excel file you can go through those you can optimize them you can check them out what can be improved here we can see that we do have some potential improvements although overall the title tags don't look too bad and we can look down here we can see we do have titles that are too long for Google and so s screaming frog will notify you of these essential issues but what's great is you don't even have to look at that yourself you can go into the issue tabs here and it will instantly tell you as we see page titles over 60 characters it will already tell you that H1 missing you can see all the URLs with H1 missing you can export these and then you can work on fixing those issues it even prioritized these by highest priority so what is critical and what is not that being said you can't just export these issues and uh call it a day because it's very important to double check and review these so you do have you know blocked pages but our search Pages it's probably fine that we're blocking our search query PL pages so this is when someone search searches on the website through the search button um that's probably not a real issue and if we go here we can see those https issues definitely a problem we can see the 4xx issues we already noticed definitely a problem all of these will be here beautifully laid out for you with H1s missing all of the different issues you're going to export those and figure out how to fix them if you need additional details on how to fix these issues just comment below I'll tell you how to fix each issue I'll send you resources whatever you need just comment on the video below and we're going to take a step back and look at various case studies of different websites within screaming frog what you can do with it and as well as the crawl config which can gets very exciting on how much you can just do with this tool so let's jump in again and if we go up here and we actually go to our clean listing here we're going to clear this before before we ever start a crawl we can go up to configuration crawl config and within here we have a lot of options sometimes you'll want to crawl link site Maps typically I include this and I'll even specify which crawl which site Maps I want it to crawl um we can go to extraction normally I'd get this the same if you want structured data you can add that in um you can do all kinds of stuff but what really matters here is this rendering so it starts off as text only if you do have a JavaScript website you're going to want to change this to JavaScript you can do one crawl with just text one crawl with JavaScript if you don't know if you have a lot of JavaScript JavaScript websites have their own issues JavaScript SEO it's its own thing so we'll talk about that in another video we have all of these different features here we can even include URLs we can exclude URLs we can do all kinds of stuff here is one of my favorite features of the craw config is the user agent from here we can actually change from screaming frog SEO spiders to Google bot directly so this will actually create the user agent so that it looks like Google bot is visiting the URL and you can get exactly what they see if you are diagnosing some Bing traffic so perhaps you had a traffic drop on Bing you'd want to change it to bingbot or Yahoo or Duck Duck Go you have all kinds of options here um and it can be very useful to change these around I usually keep to Google bot or screamy frog spider Google bot is my normal one it's very useful to go to definitely my favorite part is we can add in apis so we can also crawl in Google search console we can pull Google search console data for every one of our URLs we can crawl Google analytics for every one of our URLs and get those data points next to we can choose the exact data points we want we can filter out branded names for the Search terms on Google search console we can do all kinds of great stuff add those into your crawl if you it's your own website and learn why we also have segments here I'll link to various features below so you can look them up but this allows us in in a Ecom website we can separate it by product categories and product pages and blog posts so this is very useful you can just add here for any segments that you might want to segment the URLs now let's switch it up and look at a local website so we have this Arnold masonry website here and this is just a local website you know it's not too big it's 180 HTML Pages we can see that here and then we have the URL Port so when I'm auditing a website with screamy frog spider i' first notice a couple things here I noticed there's a lot of redirects here we see 301 301 301 and we see a lot of 200s we don't see really many 400s or any right now but we see a lot of 300s this field here you can scroll on and it gives you a ton of data so we see the titles see that there's well probably some improvements that can be made in the titles these are very long titles and we can just scroll more we can see the meta descriptions essential we can see they have two meta descriptions we can see the H1 very important kind of get an idea of how their website is structured the basic onpage SEO with H1s H2S we can really from a 30,000 Viewpoint we can see everything about their website we can see the word count on these URLs the average words per sentence how easy it is to read there is so much here even the craw depth if you know what that means and we also have the in links this is the internal links towards that page so just off the bat we have pretty much a full idea of what this website is to a web crawler like Google I'm going to look in here and instantly see these 301's and I'm going to see something strange here I'm going to see SL AML and so a next essential feature you need to know about screaming frog spider is the filters button so we go up to the search bar here we can click this button to filter now what I'm going to filter here is/ AML because it seems like these pages are the ones having redirect issues so I'm going to say address contains AML we're going to click okay and now we have this list of URLs that are redirecting and we want to investigate directly on the website where these 301 errors are actually coming from why are they on the website because a WebCrawler crawls links you have to remember this it clicks on links and it crawls the entire web and so they had to have a link towards this page for screaming frog to pick it up and if we go down into here to the URL details we can see all kinds of details about the URL and we can go to end links and from here we can see where it crawled over to this redirect this is a broken link so right off the bat I see that okay they're linking in multiple spots towards these redirects with SL AML in them rather than the correct link which doesn't have AML in it so when we clicked on that link it actually should just go direct to the page so that's instantly a primary technical SEO issue I noticed right off the bat and now this is something that would be listed here within the um the 40 or 301 redirects so it would list out the 401 or the 301 redirects here and we would have seen this which is includes primarily the AML pages and this would be a similar way to get to that issue although you really want to investigate these urls in a kind of engaged manner like I'm doing now to get the most value out of a technical SEO audit with screaming frog SEO spider and so right off the bat I would say okay we have some 301 redirects now what can we do to export that I would go to bulk export response codes and I would go to internal which means our website not external links that are going to Wikipedia or somewhere else and I would do uh client or sorry redirect 3 XX in links now it's a complicated way to say it but what this means is that these are the broken links on our website that are going over to these pages and so now I would do uh Arnold masonry right dash and then it would be these broken links so I would just say Dash broken links this is going to export a list of broken links or sorry they actually go to redirect links so my bad redirect links and it will give us the full list of source and destination we can see okay so Gallery links to these weird Gallery URLs that we need to fix that link there and let's see how many we have here you look here should be able to it seems to not be counting it if I scroll down we have tons of these redirect links just rid I mean this is 2,300 rows of redirect links and why this is happening is because if we go to Services we actually have those AML redirect links in our navigation so this is a huge issue we don't want Google bot be clicking on this navigation and then going to a redirect think about how bad that's going to be it's going to dilute our page rank our Authority on our website all kinds of issues are going to be caused from that and so right off the bat I've diagnosed one of the biggest issues on this website as far as I can see and if we go up to the issues tab we kind of use this as a guideline it's a road map to what issues might be present these are not always the case like to in it'll give you a description of why it's an issue and how to fix it this is very useful to use uh it's not always correct and so we have to uh review these different issues read up on it Google search it do your thing and see what you can do here we do see long title tags so this is very long title tags and they don't seem optimized for SEO so that's definitely going to be something we would export as page titles over that arnold. page titles this is how you would go about doing an SEO audit with screaming frog SEO spider now there is so much cool stuff you can do with screaming frog SEO spider and I'll be going into them in future videos at least if you guys are interested so let me know some of these are included identifying duplicate content looking at your internal link structure on your website and how it can be helped you can even on visualizations visualize your entire internal link structure even on a 30,000 page e-commerce website you can use it to custom extract information from your website for example product categories with no products in them or blog posts with a certain keyword in them and you could even use it to build links so give me your ideas on what you want to see next thank you so much leave a like if you do like the videos I'm providing I appreciate you comment what you think about this video thank you guys talk soon

hi today we&#39;re going to be talking about screaming frog SEO spider now what is an SEO spider it&#39;s a WebCrawler and when you think about the worldwide web the internet itself it is called a web and when you hear a spider a spider is what goes on a web essentially Google is just a web crawler that is crawling different links in going through the web indexing it and giving you the results on a silver platter now what a web crawler does is it emulates Google all Google is is a web crawler and all screaming frog SEO spider does is crawl the web it essentially emulates Google and it gives us information that we can do to help Google bot the little spider that crawls the web all the information it needs this is powerful for diagnosing technical SEO issues and making sure your website is on a solid foundation for all of your SEO initiatives as well as a lot of cool tips and tricks to maximize your organic traffic and scale those SEO rankings so if we jump in here we can check out the screaming frog SEO spider website we can look here at what their pricing is and you can purchase this beautiful tool it&#39;s been around for almost ever uh they even have a free version that allows you to crawl 500 URLs so if you have a small website you can get this for free and you can try it out um now the paid version is $260 per year and you can purchase the license there a very fair pricing by this team you can see some of what this does finding broken links analyzing SEO data and redirects duplicate content all kinds of cool stuff but rather than just looking at the website how about we jump into the tool and I&#39;ll show you what it can do so we&#39;re opening up a new instance of screaming frog SEO spider and you can get an idea of the UI the user interface and what this tool is all about um it may look a little complicated at first I&#39;m sure it does I&#39;ve been using this tool for five six years so it&#39;s kind of home for me but when we first look at this we have a big search box up at the top this is where we&#39;re going to input our website so sample.com you&#39;re going to put in your website there you can click this big green start button and that&#39;s going to start the CW going to start going through this website and just colling all the different URLs now example.com is a one-page website so we don&#39;t have much here this is where the URLs will crawl and index now if we take our example website refill finder.com and we run this through it it&#39;s going to wait and then it&#39;s going to start crawling all the URLs you&#39;re going to see those populate here so now if this is your website.com then you can see all of your different pages this is the URL Viewpoint and this is where you can get a 30,000 foot view of your website all of its pages and see how they interact with each other how all the SEO data like SEO meta title meta description H1 H2 how those are all present on the pages you could also connect this to Google search console data Google analytics data and even Google page speed data this is very powerful as you can see but let&#39;s not get ahead of ourselves and let&#39;s get down to the very basics of how this tool works and how the U UI is organized so once we have some sort of crawl running here we see this is our URL Port but right now we have some kind of strange URLs so we have image URLs all kinds of stuff if we go on here in this overview tab you can see all the URLs encountered and we can see the crawl data here we&#39;re set to all if we just switch to HTML these are your all of your real pages so HTML pages are the real website pages that you&#39;re used to seeing we can rightclick this open in a browser and it opened on my second screen but if we see here we can see the page it&#39;s just one of the page is on our example website and so here is the basic Viewpoint you know if we pause this this website&#39;s a little large so we&#39;ll pause it for here we can see we&#39;ve already crawled a thousand pages and so for large e-commerce websites as well as local websites and SAS websites you really want to see all of your URLs and you need to know them from a 30,000 Viewpoint perspective this is what this URL Viewpoint gives you and we can dig way deeper into it so if we C scroll down on this overview tab we can see different file types which aren&#39;t very interesting we really just care about our HTML Pages for the basics we can see security so you may have HTTP URLs which we see seem to have some these are not good um it does tell us the status code so it tells us like here we are redirecting from htdp to https this is critical for SEO HTTP is an old security measure you can&#39;t have those on your website and so this just give us some of the security details so looks like there&#39;s some issues we may want to resolve for this test website the next tab that&#39;s really important is the response codes um overview section here and you do have internal and external typically we&#39;re just interested in the internal and here is what those status codes represent now if you&#39;re familiar with SEO you probably know what status codes are but if you&#39;re not we can briefly review them a 404 page essentially means that it&#39;s an empty page so if we open this here with our right click we can see page not found you can&#39;t have broken pages on your website or the Google crawl is going to get very confused it&#39;s going to be landing that little web spider we talked about is going to be landing on error pages and that&#39;s going to reflect negatively on your website so within this response section we can see our 4xx errors which is all of the 404 Pages or the 430 Pages um whatever the status code is we&#39;ll investigate that and see what it meets we&#39;re also probably going to have internal links towards these pages and these are really stuff we want to clean up because you can&#39;t have Google crawling links on a website that go to a 4 or four page we can also see our 3xx Pages which are ones that are redirected currently you still normally want to clean up the internal links going towards these Pages scroll down this list and you&#39;ll see URL page titles meta descriptions H1 these are all topics that we really need to think about for SEO and so this overview tab just lets us see all of this data for all of our URLs and the important one is the page title we just spoke with Kevin indig and this is one of the top ranking signals we&#39;ve known this for a long time this tab will just tell you all of your page titles by the URL what&#39;s so great is that any of these tables you can export directly to Google Sheets or to a CSV or to an Excel file you can go through those you can optimize them you can check them out what can be improved here we can see that we do have some potential improvements although overall the title tags don&#39;t look too bad and we can look down here we can see we do have titles that are too long for Google and so s screaming frog will notify you of these essential issues but what&#39;s great is you don&#39;t even have to look at that yourself you can go into the issue tabs here and it will instantly tell you as we see page titles over 60 characters it will already tell you that H1 missing you can see all the URLs with H1 missing you can export these and then you can work on fixing those issues it even prioritized these by highest priority so what is critical and what is not that being said you can&#39;t just export these issues and uh call it a day because it&#39;s very important to double check and review these so you do have you know blocked pages but our search Pages it&#39;s probably fine that we&#39;re blocking our search query PL pages so this is when someone search searches on the website through the search button um that&#39;s probably not a real issue and if we go here we can see those https issues definitely a problem we can see the 4xx issues we already noticed definitely a problem all of these will be here beautifully laid out for you with H1s missing all of the different issues you&#39;re going to export those and figure out how to fix them if you need additional details on how to fix these issues just comment below I&#39;ll tell you how to fix each issue I&#39;ll send you resources whatever you need just comment on the video below and we&#39;re going to take a step back and look at various case studies of different websites within screaming frog what you can do with it and as well as the crawl config which can gets very exciting on how much you can just do with this tool so let&#39;s jump in again and if we go up here and we actually go to our clean listing here we&#39;re going to clear this before before we ever start a crawl we can go up to configuration crawl config and within here we have a lot of options sometimes you&#39;ll want to crawl link site Maps typically I include this and I&#39;ll even specify which crawl which site Maps I want it to crawl um we can go to extraction normally I&#39;d get this the same if you want structured data you can add that in um you can do all kinds of stuff but what really matters here is this rendering so it starts off as text only if you do have a JavaScript website you&#39;re going to want to change this to JavaScript you can do one crawl with just text one crawl with JavaScript if you don&#39;t know if you have a lot of JavaScript JavaScript websites have their own issues JavaScript SEO it&#39;s its own thing so we&#39;ll talk about that in another video we have all of these different features here we can even include URLs we can exclude URLs we can do all kinds of stuff here is one of my favorite features of the craw config is the user agent from here we can actually change from screaming frog SEO spiders to Google bot directly so this will actually create the user agent so that it looks like Google bot is visiting the URL and you can get exactly what they see if you are diagnosing some Bing traffic so perhaps you had a traffic drop on Bing you&#39;d want to change it to bingbot or Yahoo or Duck Duck Go you have all kinds of options here um and it can be very useful to change these around I usually keep to Google bot or screamy frog spider Google bot is my normal one it&#39;s very useful to go to definitely my favorite part is we can add in apis so we can also crawl in Google search console we can pull Google search console data for every one of our URLs we can crawl Google analytics for every one of our URLs and get those data points next to we can choose the exact data points we want we can filter out branded names for the Search terms on Google search console we can do all kinds of great stuff add those into your crawl if you it&#39;s your own website and learn why we also have segments here I&#39;ll link to various features below so you can look them up but this allows us in in a Ecom website we can separate it by product categories and product pages and blog posts so this is very useful you can just add here for any segments that you might want to segment the URLs now let&#39;s switch it up and look at a local website so we have this Arnold masonry website here and this is just a local website you know it&#39;s not too big it&#39;s 180 HTML Pages we can see that here and then we have the URL Port so when I&#39;m auditing a website with screamy frog spider i&#39; first notice a couple things here I noticed there&#39;s a lot of redirects here we see 301 301 301 and we see a lot of 200s we don&#39;t see really many 400s or any right now but we see a lot of 300s this field here you can scroll on and it gives you a ton of data so we see the titles see that there&#39;s well probably some improvements that can be made in the titles these are very long titles and we can just scroll more we can see the meta descriptions essential we can see they have two meta descriptions we can see the H1 very important kind of get an idea of how their website is structured the basic onpage SEO with H1s H2S we can really from a 30,000 Viewpoint we can see everything about their website we can see the word count on these URLs the average words per sentence how easy it is to read there is so much here even the craw depth if you know what that means and we also have the in links this is the internal links towards that page so just off the bat we have pretty much a full idea of what this website is to a web crawler like Google I&#39;m going to look in here and instantly see these 301&#39;s and I&#39;m going to see something strange here I&#39;m going to see SL AML and so a next essential feature you need to know about screaming frog spider is the filters button so we go up to the search bar here we can click this button to filter now what I&#39;m going to filter here is/ AML because it seems like these pages are the ones having redirect issues so I&#39;m going to say address contains AML we&#39;re going to click okay and now we have this list of URLs that are redirecting and we want to investigate directly on the website where these 301 errors are actually coming from why are they on the website because a WebCrawler crawls links you have to remember this it clicks on links and it crawls the entire web and so they had to have a link towards this page for screaming frog to pick it up and if we go down into here to the URL details we can see all kinds of details about the URL and we can go to end links and from here we can see where it crawled over to this redirect this is a broken link so right off the bat I see that okay they&#39;re linking in multiple spots towards these redirects with SL AML in them rather than the correct link which doesn&#39;t have AML in it so when we clicked on that link it actually should just go direct to the page so that&#39;s instantly a primary technical SEO issue I noticed right off the bat and now this is something that would be listed here within the um the 40 or 301 redirects so it would list out the 401 or the 301 redirects here and we would have seen this which is includes primarily the AML pages and this would be a similar way to get to that issue although you really want to investigate these urls in a kind of engaged manner like I&#39;m doing now to get the most value out of a technical SEO audit with screaming frog SEO spider and so right off the bat I would say okay we have some 301 redirects now what can we do to export that I would go to bulk export response codes and I would go to internal which means our website not external links that are going to Wikipedia or somewhere else and I would do uh client or sorry redirect 3 XX in links now it&#39;s a complicated way to say it but what this means is that these are the broken links on our website that are going over to these pages and so now I would do uh Arnold masonry right dash and then it would be these broken links so I would just say Dash broken links this is going to export a list of broken links or sorry they actually go to redirect links so my bad redirect links and it will give us the full list of source and destination we can see okay so Gallery links to these weird Gallery URLs that we need to fix that link there and let&#39;s see how many we have here you look here should be able to it seems to not be counting it if I scroll down we have tons of these redirect links just rid I mean this is 2,300 rows of redirect links and why this is happening is because if we go to Services we actually have those AML redirect links in our navigation so this is a huge issue we don&#39;t want Google bot be clicking on this navigation and then going to a redirect think about how bad that&#39;s going to be it&#39;s going to dilute our page rank our Authority on our website all kinds of issues are going to be caused from that and so right off the bat I&#39;ve diagnosed one of the biggest issues on this website as far as I can see and if we go up to the issues tab we kind of use this as a guideline it&#39;s a road map to what issues might be present these are not always the case like to in it&#39;ll give you a description of why it&#39;s an issue and how to fix it this is very useful to use uh it&#39;s not always correct and so we have to uh review these different issues read up on it Google search it do your thing and see what you can do here we do see long title tags so this is very long title tags and they don&#39;t seem optimized for SEO so that&#39;s definitely going to be something we would export as page titles over that arnold. page titles this is how you would go about doing an SEO audit with screaming frog SEO spider now there is so much cool stuff you can do with screaming frog SEO spider and I&#39;ll be going into them in future videos at least if you guys are interested so let me know some of these are included identifying duplicate content looking at your internal link structure on your website and how it can be helped you can even on visualizations visualize your entire internal link structure even on a 30,000 page e-commerce website you can use it to custom extract information from your website for example product categories with no products in them or blog posts with a certain keyword in them and you could even use it to build links so give me your ideas on what you want to see next thank you so much leave a like if you do like the videos I&#39;m providing I appreciate you comment what you think about this video thank you guys talk soon

Transcript for:Screaming Frog SEO Spider Overview

Transcript for:
Screaming Frog SEO Spider Overview