Google’s Mobile-Friendly Test Doesn’t Follow Robots.txt Rules via @MattGSouthern
Google’s John Mueller recently advised the company’s mobile-friendly test doesn’t follow the rules written in robots.txt files.
This topic came up in a Google Webmaster Central hangout where a question was asked about the mobile-friendly test and its relation to crawling and indexing.
Here is the question that was submitted:
“If I can see my text in Google’s mobile-friendly test, can I ensure that Google doesn’t have a problem with reading and indexing my content and my pages?”
In response, Mueller said it’s correct that if the mobile-friendly test can render a page then that’s a sign it can be crawled and indexed just fine.
So, in addition to the mobile-friendly test, site owners should be using the inspect URL tool in Search Console to make sure everything is working as intended.
Hear Mueller’s full response in the video below, starting at the 28:36 mark:
One of the things to watch out for here is the mobile-friendly test – as far as I know – doesn’t follow the robots.txt rules.