Crawling is a word that you may have heard many times before, but when used in the context of website indexing, few may be aware of its meaning. To understand crawling, you need to be aware that all content online can be found through search engines. These search engines, like Google, work tirelessly to locate and index web content, to produce the highest quality search results for users.
Crawling is how Google and other similar search engines find new and updated pages. In Google’s case, it uses something called Googlebot to travel to these sites and update its list of pages. Googlebot is constantly searching and adding new links to its lists of pages. It also takes note of dead links, which it also updates to steer web traffic away.
In this article we’ll help answer the question, “How often does Google crawl your site?”. But first we’ll discuss more details around Google’s crawling abilities.
It goes without being said that you want Google to crawl your site, as this is the only way it will be able to appear in their search results. And with Google being the most popular search engine worldwide, getting on their search result list will be a huge boon for you and your content. Fortunately, it’s very easy to get Google to crawl your site.
All you need to do is make use of Google’s Webmaster tool. Verify your site through there and simply click the button asking it to crawl. You have the option of requesting it to crawl the entire site, or selected pages. You will also only need to do this once, and as long as you regularly update your site, once you are on Google’s crawl list, Googlebot will continue coming back to periodically crawl your site and new pages.
This is even easier to check than asking Google to crawl in the first place. On your page, inspect your URL to see details about your Google index report. To clarify, indexing is how Googlebot organises its various lists. From this report, you can easily navigate through to see if Google has crawled your site and what its indexing status is. You can also test the live version of your site from here.
If you’ve followed the above steps and discover that Google has not crawled your site, or not crawled it for some time, it’s a sign that there is something wrong. Googlebot cannot crawl sites that contain errors, and it will therefore just move along if it encounters any resistance. Possible errors that may be impacting Googlebot’s success in crawling your page are numerous. The best way to check for these is through Google’s Webmaster Tool.
Open the tool and navigate to crawl, then crawl errors. You should be presented with a list of what the possible problems are. From here, all you need to do is work through the list and resolve the errors. It’s helpful to re-test your site after you have addressed the errors in order to make sure everything is working properly. If you continue to encounter issues, check for crawl errors again and repeat the process until Googlebot can crawl your site.
The Internet is a huge place and while Googlebot works non-stop, you need to keep in mind there is a massive amount of content for it to get through. Depending on how active your site is, you should expect Google to crawl it anywhere between every four and thirty days. Sites updated more regularly tend to be crawled more often, given Googlebot tends to hunt for new content first. If it has been longer than forty days since your last crawl, that would be your cue to check for possible crawl errors.
"Just recently, I was able to take a pretty competitive keyword from #12 (page 2), up to #5 on page 1 in 14 or 15 days."
