Store Manager for Magento news and announcements

Set Magento 2 Noindex and Nofollow Tags

What Is the Purpose of Noindex and Nofollow Tags?

When you're starting a new project on dev server or make some transactional pages, you may need to hide some of your pages or whole website from search engines not to be indexed beforehand. The best way to tell robots visiting your site not to index or follow your content is to use robots.txt file and nofollow and noindex meta tags in html head of the page(s). Magento website allows you to configure such kind of settings from the admin panel with no code used.

What Is robots.txt File And Why Do You Need It?

Let's say, some web robots want to visit your site https://store.emagicone.com and index or track its content. Before starting the indexation the robots will first enter your https://store.emagicone.com/robots.txt file and check it. This file is located in the root of your Magento site and contains the instructions for those robots on how to process or not to process your site.

Setting Noindex and Nofollow in Magento 2.1.x

  1. Find admin panel on the left side of your Magento installation and enter 'Stores' section;
  2. Under 'Settings' press 'Configurations' and go to 'General settings' - 'Design'.
    On the right you will find 'Search Engines Robots' settings.
  3. Uncheck 'Use system value' box to activate 'Default Robots' drop-down menu.
  4. There are four 'Default Robots' options available:
    • INDEX, FOLLOW - allows web robots to index your site and track it for changes;
    • NOINDEX, FOLLOW - instructs robots not to index your site, but allows to track it for changes;
    • INDEX, NOFOLLOW - instructs robots to index your site, but not to check it for changes furtherly.
    • NOINDEX, NOFOLLOW - forbids the robots neither to index nor to track you site.

    You can select the option basing on your purposes or use 'Edit custom instruction of robots.txt File' field, in case you need to block access to some specific folders or URLs.

  5. Input the following directives for robots into this field depending on your needs:
  6. To block all robots from certain folders:

    User-agent: *
    Disallow: /folder name

    To block specific robots from certain folders:

    User-agent: robot name
    Disallow: /folder name

    To block some bots to access one folder, but allow to access another one:

    User-agent: robot name
    Disallow: /folder name Allow: /folder name

Method 2

There is another way to block robots from processing your web pages from admin panel:

  1. From admin panel on the left select 'Content' section. Under 'Elements' select 'Pages'. You will find a table containing all pages of your store.
  2. Select 'Edit' option from the 'Select' drop-down menu on the right.
  3. Expand 'Design' section and find 'Layout Update XML' field. In order to prevent this page from indexing and following, insert the following code into this field:
Method 3: Setting Noindex/Nofollow For Specific Magento 2 Categories And Products

The above indicated code can also be used to configure permissions to some of your category or product pages whether in 2.1.x or in 2.2. These settings can be reached this way:

    Blocking product page:
  1. Select 'Products' from 'Catalog' section on the admin panel. Press 'Edit' on the right side of the product field. Scroll down and find 'Design' section. Expand it and enter this piece of code into 'Layout Update XML' field:
Blocking category page:

Select 'Categories' from 'Catalog' section and select necessary category from the list on the right. Find 'Design' section and enter the above indicated code into 'Layout Update XML' field.

How to Set Up Noindex/ Nofollow in 2.2.0, 2.2.1 and 2.2.2?

  1. Find 'Content' section on Magento admin panel and select 'Configuration' under 'Design' subsection. You will find there a table containing all your storeviews.
  2. Select the storeview you need to modify Noindex/Nofollow Settings for and Press 'Edit' in the 'Action' column.
  3. Scroll down and expand 'Search Engine Robots' settings. Select necessary 'Default Robots' option or use 'Edit custom instruction of robots.txt File' field to change access settings for some specific folders.
  4. Uncheck 'Use system value' box to enable 'Default Robots' drop-down menu.
  5. Select necessary option or use 'Edit custom instruction of robots.txt File' field to block access to specific folders or URLs.

  6. Input the following instructions for robots into this field basing on your needs:
  7. To block all robots from certain folders:

    User-agent: *
    Disallow: /folder name

    To block specific robots from certain folders:

    User-agent: robot name
    Disallow: /folder name

    To block some robots to access one folder, but allow to access another one:

    User-agent: robot name
    Disallow: /folder name Allow: /folder name

Conclusions: Do I Need to Block Access to Some of My Links?

Definitely, yes. It is essential for an online store owner to be able to manage access to some of his/her site’s folders or URLs by forbidding or allowing web robots to index or follow them. Beside limiting access for unknown crawlers, you can considerably improve your SEO by applying these restrictions.

The thing is that, when filling your site with content, you probably strive to optimize it for Google as much as possible in order to show up in the first positions of search results. Although, some of your site links might have some negative influence on your Google rate, so it would be appropriate to disallow search engine robots to index or track them.

Magento provides its users with a majority of ways to manage site access in few clicks directly from admin panel. Take advantage of any of them and keep your web store protected from any undesirable indexing or tracking.

No comments :

Post a Comment