You will be right if you think that Google will index your WordPress website even if you stay passive and do nothing about it. After all, it is their job to provide search users with the best possible search results.
Fact is, it may take much longer than you would expect for Google to take notice and crawl your website. Plain writing content, optimizing it for SEO and waiting for the eventual indexing is not good enough, as it may cost you traffic and consequentially lost profit.
How does Google indexing work?
The list of information is in its most basic form is an index. Indexes existed in many ways long before the Internet, such as the alphabetical indexes at the end of books. Ideally, a web index is a database stored on a remote computer connected to the Internet.
This kind of database that is kept as a search engine index, stores information about billions of pages. Using a search engine you will be actually searching through the search engine index, not the whole Internet. Different search engines use a separate index, but for the purpose of this guide, we will focus on Google primarily, as it is the largest search engine.
In order to acquire information for billions of pages, Google uses crawl bots also known as “spiders”. These powerful tools are actually automated software algorithms that go from site to site and follow the links between them to find relevant information. Information gathered by crawl bots is stored in Google’s index. Because search engines strive to provide up-to-date results, crawl bots never stop doing their job and constantly add fresh information into the index database.
Why should you be worried about having your WordPress website indexed fast?
A layman may add the question: Why should you put in extra effort and make it easier and faster for the crawl bots to go through your website indexing? Fact is if your website is relatively new or does not get crawled often enough for any other reason, even if you produce fresh content regularly it will not be served to search users promptly. This delay in your website visibility with search engines can be the difference between a successful and unsuccessful website.
Arguably, yes, you can drive traffic to your website from a number of sources, but the organic traffic from search engines is much more valuable because of these people end up at your website because they wanted to be there. Search presence can help your success or the lack of it may be your downfall.
Fast and often website indexations is important for improving your website search rankings. You should want your website indexing to be done sooner so that your website will start competing for better search rankings and domain authority.
Frankly, your WordPress website takes time to rank-up and this will not happen overnight. Google statistics show that the average website with ranking on the first search page is older than two years, while the website in position number one is older than three years. All this means that you should start working on your website indexing and SEO from day one of your website publishing.
All of this implies to your website if you have changed or switched the website domain name. Migrating absolutely all of the website content will not change this, Google is gonna view your new domain name as a completely new website. While on this subject, we can note that Google does help to a certain degree with this process by providing a “Change of Address” form.
Improving WordPress website crawl speed with a plugin
The greatest benefit of having a WordPress website is the sheer amount and consequently, the quality of plugins offered. WordPress users first choice for this purpose can be the Yoast SEO plugin. This plugin is made with overall website SEO in mind and it is a very good choice for beginners and experts alike.
Many Yoast SEO users value the most how it simplifies the process of optimizing your website. It will analyze your on-page SEO for every single post you add and point-out issues with tag, meta description, and readability. The free version of Yoast SEO offers plenty of features, including those we need to improve website crawling.
Now, we will focus on the XML Sitemaps feature of the plugin, as it is meant for improving indexation. Actually, the process is relatively straight-forward and easy. After you have installed the plugin, go to its dashboard and select XML Sitemaps. Within this setting click Enable under “XML sitemap functionality”, and then click Save Changes on the bottom. Whoila, your website now has a sitemap! This new sitemap will cut indexing time to minutes, while without it the process may take a whole day.
Another important piece of information is that the sitemap conveys to Google how often to look for changes and new pages, and if you add a blog post, product page or other content on a regular basis or your content publishing is time-sensitive, then you will want Google to know and crawl your website more often.
One free alternative would be Screaming Frog SEO spider, which is free for websites with under 500 pages. Again, the process is relatively straightforward and you do not need any coding knowledge. After you have installed the plugin, enter your website URL into the appropriate field and click Start. Once the plugin has finished and ready, click Sitemaps and then click Create XML Sitemap. While in the Sitemap Export Configuration, you will be able to include a tell to Google for how frequent you want your website to be crawled. The option “Calculate from Last Modified Header” will save you time and effort in setting this for a larger number of web pages, by automating.
Consequently, if you have added content in the 24-hour interval, then crawl frequency will be 24h. Other options include opting to have images indexed too. One good practice is to have your sitemap with less than 50.000 URLs or no larger than 50MB as this will keep the sitemap in one file while breaking any of these two rules will lead to sitemap be contained in two files.
Submit website sitemap to Google Search Console for improved indexing
Next step is not totally necessary but it will, again, increase indexing and search crawling speed for your website, and that is submitting your sitemap to Search Console.
Once on your Search Console home page, on the left sidebar click Site Configuration, Sitemaps, then Add/Test Sitemap red button on the top-right. Most commonly, your sitemap URL is http://your-website-address.com/sitemap.xml. You can check for the sitemap location in the WordPress plugin you used to create it. A submitted sitemap will be Pending for a while, and then Approved.
Bear in mind that once submitted sitemap is only true for the current state of your website, and as you add content and pages, your site will grow and the sitemap will become obsolete. Make sure that you repeat this process following your website expansion and Google will always have a fresh version of what your website looks like.
How to disable indexing?
On a side note, some websites may want to avoid indexing some pages or the whole website for various reasons. The most straightforward way to go about this is to create a robot.txt file on your website that will give instruction to crawler which pages should index and which should not. In a specific case where your robot.txt file says “don’t index me” the crawl bot will not do any indexing and just skip the website entirely.
If you ever need a more complex robot.txt file or do not want to do it manually, then a smart choice would be to use a generator such as SEObook or similar. One of the more common reasons why you would want crawl bot to skip some of your website pages is to be more efficient. Disabling crawl for specific complex pages helps it focus on the more important content doing it fast and often.
Important to note is that crawl bot or “spider” will follow links from pages and this increases the significance of links between individual pages of your website. The spider follows the path formed by links. Make sure you have created links between all your important website pages.
Well, thought-out website architecture enables hierarchy among pages, and it helps crawler acknowledge important pages and page relations. Every page on your website should have at least one link pointing to it.
Are you using Tables of Content?
Internet users often come in a hurry for an answer and if they can’t find it fast they would leave the website or page. Having a table of contents on a page will help them look for a specific piece of information. WordPress users can utilize the Easy Table of Contents plugin. Just like the name implies, the process is simple and involves a few clicks in the plugin settings, and yes, you can set the plugin to do a table of contents automatically.
Conclusion
WordPress is a versatile platform with so many plugins which can turn any menial chore into a trivial task resolved with a few clicks. Improving your website indexing is that easy, though it is one of the most important tasks for a successful website. We hope that you will adopt these techniques as your favorite, and check with us regularly for more informational content.