These are the Dangerous Robot.txt Settings Aplied To Blogs

Optimizing SEO on blogs by setting Robot.txt is fine. But please note, that not all Robot.txt settings that you find on google search are good for the continuity of your blog.

These are the Dangerous Robot.txt Settings Aplied To Blogs

Consider the blog that you manage, both from the form of all the contents of the blog, the goals and the audience you want to archieve.

If your blog aims to share tutorials, present unique informations, review articles and other informations that you think is useful, of course you want to get visitors from the various devices used, both from mobile phones and computer.

What does this have to do with the Robot.txt settings

We know that Robot.txt aims to filter urls to be crawled or no crawled on search engines. But that doesn't mean you deliberately restrict visitors from the devices they use.

So, if you are one of those people who want to bring visitors from the various devices that I mentioned earlier, then you should avoid the following things in the robot.txt settings.

Below is the code for robot.txt that you shouldn't apply to the robot.txt settings. Avoid using robot.txt Disallow Settings M=1 & M=0

User-agent: Mediapartners-Google
Disallow:
User-agent: Googlebots
Disallow: /Search
Disallow: /?m=1
Disallow: /?m=0
Disallow: /*m=1
Disallow: /*m=0
User-agent: *
Disallow: /search
Sitemap: https://your-domain.com/feeds/posts/default?orderby=UPDATED

Description:

Code that Disallows M=1 & M=0, the code aims to overcomes duplicate blog pages. But the consequences are fatal and should not be implemented.

The result of applying the Robot.txt setting is as follows:

As a result of applying the Robot.txt Settings

1. Blog posts Are Definitely Not Mobile Friendly

We know that the robot.txt code blocks blogs on mobile displays. You are not sure? Please visit the link. Then take the blog url that you applied with the robot.txt setting above, and watch what happens.

Then, if you check the webmasters, you will see so many error crawling blog on mobile devices. Of course this is really not good.

2. Missing post descriptions on search engines

Obviously the descriptions of a post is very useful in search engines. In addition to making blog posts look SEO friendly, this descriptions contains small snippets or sentences or a short explanation of a post.

Instead it shows a description like this "The Descriptions for this result is not available because of the site's robot.txt".

3. Make potential Visitors Run

So when they see something like "The Descriptions for this result is not available because of the site's robot.txt" they will likely think it's a virus which will be harmful to their phone in the future, then move on to another link.

It's a little excessive, but the smallest things that need to be considered for the continuity of your blog.

4. Over time, your posts will not be indexed by search engines

If the blog is banned for mobile devices and the loss of description for each post on the mobile devices search engine, then your blog posts will not longer be indexed by the google search engine in other words, your blog has become bad in Google's serp.

There are many other ways of optimize blogs that are so easy to apply and also have so many benefits for your website.