Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Filters and example code for Yoast SEO robots or WP robots.txt
<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/
/*
* Replace Disallow with Allow Generated Robots.txt
* Credit: Unknown
* Last Tested: June 09 2020 using WordPress 5.4.1
*/
add_filter('robots_txt','custom_robots');
function custom_robots($output) {
$public = get_option( 'blog_public' );
if ( '0' != $public )
return str_replace('Disallow','Allow',$output);
}
<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/
/*
* Remove robots meta tags from Yoast SEO
* Credit: Yoast development team
* Last Tested: May 09 2020 using Yoast SEO 14.0 on WordPress 5.4.1
*/
add_filter( 'wpseo_robots', '__return_false' );
add_filter( 'wpseo_googlebot', '__return_false' ); // Yoast SEO 14.x or newer
add_filter( 'wpseo_bingbot', '__return_false' ); // Yoast SEO 14.x or newer
<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/
/*
* Change meta robots using Yoast SEO
* Credit: Yoast development team
* Last Tested: Nov 1 2021 using Yoast SEO 17.4 on WordPress 5.8.1
*/
add_filter( 'wpseo_robots', 'yoast_seo_robots_remove_search' );
function yoast_seo_robots_remove_search( $robots ) {
if ( is_search() ) {
return false;
} else {
return $robots;
}
}
<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/
/*
* Change meta robots using Yoast SEO
* Credit: Yoast development team
* Last Tested: Dec 12 2017 using Yoast SEO 9.2.1 on WordPress 5.0
*********
* DIFFERENT POST TYPES
* Post: Change 123456 to the post ID
* Page: Change is_single to is_page and 123456 to the page ID
* Custom Post Type: Change is_single to is_singular and 123456 to the 'post_type_slug'
Example: is_singular( 'cpt_slug' )
*********
* MULTIPLE ITEMS
* Multiple of the same type can use an array.
Example: is_single( array( 123456, 234567, 345678 ) )
* Multiple of different types can repeat the if statement
*********
* The return false removes the robots tag on the page
* Or you can return index/noindex follow/nofollow like
* return 'noindex, follow';
* Or
* return 'noindex, nofollow';
*/
add_filter( 'wpseo_robots', 'yoast_seo_robots_remove_single' );
function yoast_seo_robots_remove_single( $robots ) {
if ( is_single ( 123456 ) ) {
return false;
} else {
return $robots;
}
}
@gopalkumar315
Copy link

gopalkumar315 commented Aug 3, 2017

Thanks a lot, it's really useful for me....

@codename2
Copy link

codename2 commented Jan 10, 2018

where can i upload this php file?
BTW, thank you

@amboutwe
Copy link
Author

amboutwe commented Jan 10, 2018

Typically code snippets are added to your theme's functions.php file.

If you're not familiar with using code snippets, these articles have wonderful advice:
http://www.wpbeginner.com/plugins/how-to-easily-add-custom-code-in-wordpress-without-breaking-your-site/
http://www.wpbeginner.com/beginners-guide/beginners-guide-to-pasting-snippets-from-the-web-into-wordpress/

@gholm
Copy link

gholm commented Sep 18, 2018

If I understood correctly, the first snippet removes noindex and nofollow from Yoast SEO entirely. But how can we change the noindex for “Special Pages” only, please?

I'd like to take out the noindex directive from the 404 pages, but retain the possibility to add noindex for taxonomies, for example.

Edit: I think the following snippet just did it (hopefully I didn't break anything):

add_filter( 'wpseo_robots', 'yoast_seo_robots_remove_404' );

function yoast_seo_robots_remove_404( $robots ) {
  if ( is_404() ) {
    return false;
  } else {
    return $robots;
  }
}

@SeryozhaK
Copy link

SeryozhaK commented Jul 11, 2019

Hi! How to noindex tag page where only 1 post?

@amboutwe
Copy link
Author

amboutwe commented Jul 11, 2019

@SeryozhaK You or your developer can write custom code to retrieve all tag pages, count the number of posts for the tag and using the two pieces create a list of tag pages with 1 post. Once you have that list you can use the code above to write a noindex tag.

However, if you have a lot of tag pages with only 1 post and you're not a really new site, your time is probably better spent removing, combining and reorganizing tags that are used infrequently.

@gabeshaughnessy
Copy link

gabeshaughnessy commented Jun 3, 2020

Can you confirm that setting $robots to false is still the correct way to filter robots? I am seeing that doing so results in no tag being output. Updating my filter to set robots to a string 'noindex, nofollow' seems to be working.

@amboutwe
Copy link
Author

amboutwe commented Jun 5, 2020

This is not the proper place to request support. Please check out our extensive help section or visit the free support forum. If you require further support, upgrading to our premium version provides you with access to our support team.

@catalincic
Copy link

catalincic commented Aug 31, 2021

Hey, I want to mention one thing in regards to the demo which may or may not be of interest for you and your team.

When returning 'noindex,follow' from the filter function, as specified in your examples, the meta tag looks like this:
<meta name='robots' content='max-image-preview:large, noindex,follow'>

If returning 'noindex, follow' (notice the space), the meta tag looks like:
<meta name='robots' content='noindex, follow'>

It is not important, but may be worth updating the demos and also maybe support commas without trailing spaces as it definitely looks like the missing space breaks some logic.

LE: As a matter of fact, the trailing space does seem to break important functionality in the Yoast SEO plugin - in the first case, when returning 'noindex,follow', the canonical link tag is not automatically removed from the page. When returning 'noindex, follow' the canonical tag is correctly removed.

@amboutwe
Copy link
Author

amboutwe commented Sep 3, 2021

@catalincic Thank you for bringing this to my attention and I have fixed the code snippets above.

Please open a new issue to request the support of commas without trailing spaces here: https://github.com/Yoast/wordpress-seo/

@vojcha
Copy link

vojcha commented Sep 21, 2021

Has something changed with the latest versions of YOAST and WP?
This function doesn't work anymore on my websites:
yoast_seo_robots_remove_search.php

@amboutwe
Copy link
Author

amboutwe commented Sep 24, 2021

@vojcha This is not the proper place to request support. Please check out our extensive help section or visit the free support forum. If you require further support, upgrading to our premium version provides you with access to our support team.

@stock-reaper
Copy link

stock-reaper commented Oct 30, 2021

For people looking to index search pages, unfortunately the filter (yoast_seo_robots_remove_search) provided here no longer works on new yoast versions.

Since the yoast team is not providing support on using these filters anymore, if anyone successfully removed the noindex from search result pages, please be kind enough to share with the community.

@stock-reaper
Copy link

stock-reaper commented Oct 30, 2021

Screenshot_20211030-141110

@amboutwe
Copy link
Author

amboutwe commented Nov 1, 2021

@stock-reaper I tested the is_search() version of the snippet and it works in a default environment. As it's not working for your site, you likely need to modify the snippet to target the correct search variable. As the issue is specific to your setup and not the code snippet, a developer can help you customize the snippet for your needs.

We recommend checking out Code Poet, WordPress Jobs, or WordPress Professional Network for development & consultancy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment