Filters and example code for Yoast SEO robots or WP robots.txt

wordpress_robots_custom.php

<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/

/*
 * Replace Disallow with Allow Generated Robots.txt
 * Credit: Unknown
 * Last Tested: June 09 2020 using WordPress 5.4.1
 */

add_filter('robots_txt','custom_robots');

function custom_robots($output) {
     $public = get_option( 'blog_public' );
     if ( '0' != $public )
     return str_replace('Disallow','Allow',$output);
}

yoast_seo_robots_callback_hooks.php

<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/

/* Check which functions have been used to hook into `wp_robots`
 * Credit: Yoast team
 * Last Tested: Unknown 
 */

\add_filter( 'wp_robots', 'list_hooks' , \PHP_INT_MAX);

function list_hooks( $robots ) {
	global $wp_filter;

	echo "<!-- This is a list of callback functions hooked into the 'wp_robots' filter:";
	echo json_encode($wp_filter['wp_robots'], JSON_PRETTY_PRINT);
	echo "-->";

	return $robots;
}

yoast_seo_robots_change_author_archive.php

/*
 * Change meta robots to index all author archive pages
 * Credit: Yoast development team
 * Last Tested: Jul 20 2023 using Yoast SEO 20.11 on WordPress 6.2.2
 */

add_filter( 'wpseo_robots', 'yoast_seo_robots_change_author_archive' );

function yoast_seo_robots_change_author_archive( $robots ) {
  if ( is_author() ) {
    return 'index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1';
  } else {
    return $robots;
  }
}

yoast_seo_robots_remove.php

<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/

/*
 * Remove robots meta tags from Yoast SEO
 * Credit: Yoast development team
 * Last Tested: May 09 2020 using Yoast SEO 14.0 on WordPress 5.4.1
 */

add_filter( 'wpseo_robots', '__return_false' );
add_filter( 'wpseo_googlebot', '__return_false' ); // Yoast SEO 14.x or newer
add_filter( 'wpseo_bingbot', '__return_false' ); // Yoast SEO 14.x or newer

yoast_seo_robots_remove_search.php

<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/

/*
 * Change meta robots using Yoast SEO
 * Credit: Yoast development team
 * Last Tested: Nov 1 2021 using Yoast SEO 17.4 on WordPress 5.8.1
 */

add_filter( 'wpseo_robots', 'yoast_seo_robots_remove_search' );

function yoast_seo_robots_remove_search( $robots ) {
  if ( is_search() ) {
    return false;
  } else {
    return $robots;
  }
}

yoast_seo_robots_remove_single.php

<?php
/********* DO NOT COPY THE PARTS ABOVE THIS LINE *********/

/*
 * Change meta robots using Yoast SEO
 * Credit: Yoast development team
 * Last Tested: Dec 12 2017 using Yoast SEO 9.2.1 on WordPress 5.0
 *********
 * DIFFERENT POST TYPES
 * Post: Change 123456 to the post ID
 * Page: Change is_single to is_page and 123456 to the page ID
 * Custom Post Type: Change is_single to is_singular and 123456 to the 'post_type_slug'
    Example: is_singular( 'cpt_slug' )
 *********
 * MULTIPLE ITEMS
 * Multiple of the same type can use an array.
    Example: is_single( array( 123456, 234567, 345678 ) )
 * Multiple of different types can repeat the if statement
 *********
 * The return false removes the robots tag on the page
 * Or you can return index/noindex follow/nofollow like
 * return 'noindex, follow';
 * Or
 * return 'noindex, nofollow';
 */

add_filter( 'wpseo_robots', 'yoast_seo_robots_remove_single' );

function yoast_seo_robots_remove_single( $robots ) {
  if ( is_single ( 123456 ) ) {
    return false;
  } else {
    return $robots;
  }
}

@catalincic Thank you for bringing this to my attention and I have fixed the code snippets above.

Please open a new issue to request the support of commas without trailing spaces here: https://github.com/Yoast/wordpress-seo/

Has something changed with the latest versions of YOAST and WP?
This function doesn't work anymore on my websites:
yoast_seo_robots_remove_search.php

@vojcha This is not the proper place to request support. Please check out our extensive help section or visit the free support forum. If you require further support, upgrading to our premium version provides you with access to our support team.

For people looking to index search pages, unfortunately the filter (yoast_seo_robots_remove_search) provided here no longer works on new yoast versions.

Since the yoast team is not providing support on using these filters anymore, if anyone successfully removed the noindex from search result pages, please be kind enough to share with the community.

@stock-reaper I tested the is_search() version of the snippet and it works in a default environment. As it's not working for your site, you likely need to modify the snippet to target the correct search variable. As the issue is specific to your setup and not the code snippet, a developer can help you customize the snippet for your needs.

We recommend checking out Code Poet, WordPress Jobs, or WordPress Professional Network for development & consultancy.

@amboutwe add_filter('wpseo_robots', '__return_false'); is now returning <meta name='robots' content='noindex, nofollow' />

Is there any other filter to completely remove the robots meta tag from HTML?

@amarilindra
Did you this one:
remove_filter('wp_robots', 'wp_robots_max_image_preview_large');
?

@vojcha

Thanks for the hint.

remove_filter('wp_robots', 'wp_robots_max_image_preview_large'); alone didn't work.

However, the following combination did the trick.

function ikva_remove_robots_meta() {
	return null;
}

add_filter('wpseo_robots', 'ikva_remove_robots_meta');
add_filter('wpseo_googlebot', 'ikva_remove_robots_meta');
add_filter('wpseo_bingbot', 'ikva_remove_robots_meta');
add_filter( 'wpseo_canonical', '__return_false' ); // Optional
remove_filter('wp_robots', 'wp_robots_max_image_preview_large');

return null outputs <meta name='robots' content='max-image-preview:large' />

and remove_filter('wp_robots', 'wp_robots_max_image_preview_large'); stops outputting the robot meta in HTML.

@amarilindra
Thanks for the solution! The Yoast SEO plugin is very dangerous because with every update there is a risk that something will break.
Sincerely

@amarilindra Thank you!

Hi @amboutwe
I have that kind of filter issue found in my search console, how I can solve the following filter issue?
https://fatimafurniture.ae/product/austin-study-table-with-bookshelf-in-azure-blue/?utm_source=rss&utm_medium...
https://fatimafurniture.ae/product-category/metal-furniture/?orderby=date
https://fatimafurniture.ae/product-category/new-arrival/?orderby=price-desc
https://fatimafurniture.ae/product-category/tv-unit/?orderby=price
https://fatimafurniture.ae/product-category/l-shape-sofa/?orderby=rating
https://fatimafurniture.ae/shop/page/23/
https://fatimafurniture.ae/product-category/sales/page/2/?orderby=popularity
Due to these issues, I have many errors in my google search console.
kindly reply with a code here where I can set all filters "Noindex and no follow"
Screenshot

@FatimaFurnitureSEO This is not the proper place to request support. Please check out our extensive help section or visit the free support forum. If you require further support, upgrading to our premium version provides you with access to our support team.

Hi all.
I also want to share my results.
You need to understand that WordPress itself changes the robot meta tag name='robots'.
First, disable all affected plugins and check.

To influence WordPress you need to use a filter wp_robots
Below is an example for a search results page:

<?php
add_filter('wp_robots', 'my_wp_robots_directives');
function my_wp_robots_directives( $robots ) {
  if ( is_search() ) {
	unset( $robots['max-image-preview'] );
	unset( $robots['noindex'] );
	$robots['index'] = true;
	return $robots;
	}
}
?>

But if you turn on Yoast SEO, it will fix this tag as it sees fit, according to Google's recommendations. Filter wpseo_robots

<?php
add_filter( 'wpseo_robots', 'yoast_seo_robots_remove_search' );
function yoast_seo_robots_remove_search( $robots ) {
  if ( is_search() ) {
    return null;
  }
}
?>

return false; does not work
return null; works
so the examples above might not work for you.

Thank you a lot @amarilindra and @ws256.

To bad this function is not built in in Yoast. I totally agree @tomybyte, it's always very dangerous to update Yoast. I, for that reason, don't like the plugin, but it's still standard for SEO.

Hi there, I need some help. We have Yoast installed on our website and I need to add a noarchive to the WP Fusion restricted posts. The condition is definately working . I tested the condition with an echo statement and it does add the ", noarchive" to the $robots string, but it does not update the robot. We can add the noarchive manually to the posts, but we need to automate it to ensure that it is present when there is a restriction on the post. What would the reason be for it not updating the robot?

add_filter('wpseo_robots', 'add_noarchive_directive');
function add_noarchive_directive($robots) {
 	global $post;
	$post_id = $post->ID;
    // Check if the current post is restricted for the user
     if (wp_fusion()->access->get_post_access_meta( $post_id )['lock_content'] ) {
			// Add 'noarchive' directive if the condition is met
        	$robots .= ', noarchive';
    }
    return $robots;
}