How to Install Robot.txt in WordPress

Table of Contents

Is a simple text file that has the function of controlling the Google search engine working on the website. Robot.txt is responsible for separating pages that you don't want to display on the Google search engine.

What's the point? Is to increase the site's ranking on the Search Engine Result Pages (SERP).

By providing robot.txt on a blog or website, it will give a sign to Google where access must be made and which pages must be indexed. Apart from that, robot.txt is also useful for SEO. Where every article on your blog/website will be easier to index on the Google search engine.

How to set robot.txt in WordPress

In general, robot.txt already exists in the root folder of the site file storage server. So that you can modify robot.txt rules, you have to create a robot.txt file which is placed in the public_html folder and enter the configuration script manually. Here are the robot.txt settings in WordPress:

User-agent:*
Disallow:/cgi-bin
Disallow:/wp-admin
Disallow:/wp-includes
Disallow:/wp-content
User-agent:Bingbot
Disallow:/

Apart from the options above, you can also apply robot.txt settings to your website to allow some search sites to crawl. You just need to add some scripts below:

User-agent:Mediapartners-Google
Allow:/
User-agent:Adsbot-Google
Allow:/
User-agent:Googlebot-Mobile
Allow:/
Sitemap:https://domainsobat.com/sitemap.xml

There are several ways you can activate robot.txt in WordPress. The easiest and simplest way is to upload the robot.txt file manually.

The following is how to set robot.txt in WordPress:

1. Using the all in one SEO pack plugin

• Instal plugin
• Enter feature manager
• Then go to the all in one SEO menu
• Click robots.txt
• In the menu add user agent, rule and directory-path
• Finally add rules

2. Using the Yoast SEO plugin

• Instal plugin
• Enter the tool page > SEO Tool
• There will be several options
• Click File Editors
• On the page Click A New File robot.txt
• Enter the rule that you want to apply on the web

3. Manual file robot.txt

• Create a robots.txt file
• Enter the rules into the file
• Click save
• Upload the txt file to the hosting server by using FTP
• Upload via the hosting service panel

4. Submit Google search console

Make sure the robot.txt file that you have installed is correct, so there are no errors.

Thus this article about how to install robot.txt in WordPress. We will update more and less in the next article.