Why Your WordPress Blogs Are Not Crawled?

You might wonder why your blogs that uses WordPress platform aren’t crawled by search engine spiders or included on the search results of the leading search engines (as I have mentioned on my posts on oes tsetnoc crawlability). Specially if you are using Fantastico on the CPanel of your hosting account. There are several possible reasons why newly created site that uses WordPress is not crawled. But the basic problem lies on how the Robots.txt and the meta tags are set on your site.

Let start with the Robots.txt. First check if you do have a robots.txt created on your site. If you don’t have it, just create it on notepad and save the following text on the file:

User-Agent: *
Allow: /
Sitemap: http://www.yourdomain.com/yoursitemap.xml

This code tells all search engines to be allowed to visit all the files on your site and declare the default sitemap file. Upload this file on your site.

The next thing that you need to check is if you set the Privacy Settings of your site properly (which I failed to do on my oes tsetnoc blog). I have seen today that by default, Fantastico, the script installer of CPanel, install WordPress with a private settings. Which mean, it adds a NOFOLLOW, NOINDEX meta tag on your site. You can set it on the Dashboard -> Settings page of your WP-Admin. Here’s a video to help you find the place where to set the said settings.