How To Run The Ultimate Website Crawl

If you're serious about SEO, you're going to need to constantly expand the size of your site. As your site gets larger, it becomes harder and harder to manage with so many URLs to consider. You'll need to constantly evaluate your site and be able to ask and answer the following questions:

• Where are my pages linking to?
• Which keywords are my pages targetting?
• Is there proper internal interlinking or interlinking opportunities
• Have we built relevant backlinks and deep links?
• Has all the on-page SEO been done correctly?

Answering these questions by manually inspecting pages can take an insane amount of time or could be near impossible if you're running an eCommerce site with tens of thousands of pages/products. So to do all the heavy lifting we turn to web crawling tools, my personal favourite being screaming Frog (SF). SEOs prefer different crawlers for different reasons, but one thing is constant if you want a great organic website any SEO would tell you, you need to crawl before you can ball.

What is web crawling/scraping?

Before a search engine can tell you where a file or document is, it must be found. To find information on the hundreds of millions of Web pages that exist, a search engine employs special software robots, called spiders, to build lists of the words found on Web sites. When a spider is building its lists, the process is called Web crawling. In order to build and maintain a useful list of words, a search engine's spiders have to look at a lot of pages and relook at old pages for updates and re-crawl them. To replicate the behaviour of the crawler and see what they see when visiting a site we employ the help of web bots with the use of web crawling tools.

For more on crawling tools check out our post - 6 Web Crawling Tools Available

In the interest of site owners everywhere I've decided to put together the ultimate web crawl technique or rather my preferred method of scraping a site. The first thing you need to do is download screaming frog, once that's been set up you need to integrate data from some 3rd party tools.

How to connect Screaming Frog to other tools

To leverage off other search tool data you will need to create a connection between SF and 3rd party tool API's. The set up is fairly simple and straightforward.

• Open Screaming Frog
• Click on the configuration tab
• Scroll down to API access
• Select the tool you want to integrate
• Follow the connection prompts - For Google Products it will be an account sync, and then you will need to select the site profile you want to sync
• Follow the connection prompts - For non-Google Products you will need to have your API key handy during the process
• And you're done

Note: For the non-Google products you will need to set up an account first and in some cases only the paid account will provide you with access to an API key

Harness the power of API's

Now that you've connected all your accounts what will each one give you once you run your next website crawl?

Connect Google analytics - Free

You can connect to the GA API and pull in data directly during a crawl. The SEO Spider can fetch data like visits, eCommerce transactions, social activities, publisher revenue earned, goals, page load speed and more so you can view more data on your top performing pages when performing a technical or content audit.

Connect Google search console - Free

You can connect to the Google Search Console Analytics API and pull in data directly during a crawl. The SEO Spider can fetch impressions, clicks, CTR and position metrics from Google Search Analytics, so you can view your top performing pages when performing a technical or content audit.

Connect MozScape - Free and Paid

You can connect to the MOZ API and pull in data directly during a crawl. The SEO Spider can fetch data like page authority, MozRank, External equity passing links, External links and subdomain links so you can view more data on your top performing pages when performing a technical or content audit.

Connect Ahrefs - Paid

You can connect to the aHrefs API and pull in data directly during a crawl. The SEO Spider can fetch data like Ahrefs link authority metrics, backlinks, backlink value and more data on your top performing pages when performing a technical or content audit.

Connect Majestic SEO - Paid

You can connect to the Majestic API and pull in data directly during a crawl. The SEO Spider can fetch data like page external backlinks, referring domains, Trust flow, citation flow, referring IPs, link break down by TLD extension, Link breaks downs by secure and non-secure links so you can view more data on your top performing pages when performing a technical or content audit.

If you want to know how to improve your sites crawlablity check out our post - 13 Steps To Maximise Your Crawl Budget

Now that you've integrated your 3rd party tools you're ready to perform your super crawl. Enter your sites domain URL, click start and wait for the crawler to do its thing.

Learn to crawl before you can run a website

Once you've run this super crawl and exported the mega excel sheet, you'll now be equipped with a complete site overview. Which will help you identify opportunities for updating missing titles, descriptions, fixing broken links, salvaging backlinks, maximising your keyword reach density and so much more.

Using the ultimate crawl technique and putting it in the hands of the right SEO and content team can turn into a powerful tool. It can be used to identify quick wins as well as a base your next 6-month strategy.

From it, you can create that priority list of must-have tasks that can take a websites organic traffic to new heights.

Contact us

If you want to know more about web-crawlers, don’t be shy we’re happy to assist. Simply contact us here

Tags: SEO, Tools, Organic search