How to access robots.txt in WordPress?
If you have FTP access to your host, to access the robots.txt file in WordPress, you will need to connect to your web-space using an FTP client. Once connected, you will need to navigate to the WordPress installation folder, typically inside the /public_html/ directory.
Within this directory, you should see a file named robots.txt. You can then right-click on this file and select the “View/Edit” option to view and edit the file.
If you’re unable to find it, you can create a new file and name it “robots.txt”.
The easiest way to plugin a robots.txt file in WordPress if you don’t have or want to connect to web-space via the above method is to use a plugin like Yoast SEO. Once you install and activate the plugin, simply go to SEO -> Tools and scroll down to the Files tab. From there, you can upload your robots.txt file.
What is WordPress robots.txt?
The WordPress robots.txt file is a simple text file located in the root directory of your WordPress site. It tells search engine crawlers which pages or files they should access and which they should ignore.
A typical WordPress robots.txt file might look something like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
This file instructs all search engine bots to stay away from the WordPress admin area, but allows them to access the admin-ajax.php file.