.Google.com's John Mueller answered an inquiry about why Google marks web pages that are actually refused from crawling by robots.txt as well as why the it is actually secure to overlook the related Look Console reports concerning those crawls.Crawler Visitor Traffic To Query Specification URLs.The person asking the inquiry recorded that robots were actually producing hyperlinks to non-existent query specification Links (? q= xyz) to web pages along with noindex meta tags that are actually likewise obstructed in robots.txt. What cued the question is that Google is creeping the hyperlinks to those webpages, getting obstructed by robots.txt (without seeing a noindex robotics meta tag) at that point acquiring shown up in Google Look Console as "Indexed, though shut out through robots.txt.".The person talked to the adhering to question:." However listed below's the major inquiry: why will Google.com mark pages when they can't even view the content? What is actually the conveniences in that?".Google's John Mueller affirmed that if they can't crawl the web page they can not see the noindex meta tag. He also produces an intriguing mention of the site: hunt operator, urging to neglect the results given that the "typical" users will not observe those end results.He created:." Yes, you're correct: if our experts can't creep the webpage, our experts can't view the noindex. That stated, if our team can't crawl the web pages, then there's certainly not a whole lot for us to mark. Therefore while you may observe several of those pages with a targeted site:- concern, the typical customer won't view all of them, so I wouldn't bother it. Noindex is additionally alright (without robots.txt disallow), it just implies the Links will definitely wind up being crept (as well as find yourself in the Look Console record for crawled/not catalogued-- neither of these statuses induce problems to the rest of the web site). The important part is that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the constraints being used the Web site: hunt advanced search operator for analysis reasons. Among those factors is given that it is actually certainly not linked to the normal search index, it's a separate point altogether.Google's John Mueller talked about the site hunt operator in 2021:." The quick solution is actually that a web site: question is certainly not implied to become full, nor utilized for diagnostics reasons.A website query is actually a specific sort of search that confines the end results to a specific internet site. It's essentially only words site, a colon, and after that the internet site's domain name.This query restricts the outcomes to a details site. It's not indicated to be an extensive collection of all the webpages coming from that web site.".2. Noindex tag without using a robots.txt is actually alright for these kinds of conditions where a robot is actually connecting to non-existent pages that are receiving found by Googlebot.3. Links with the noindex tag will create a "crawled/not recorded" entry in Browse Console and also those won't possess a bad result on the remainder of the website.Check out the concern as well as answer on LinkedIn:.Why would certainly Google index web pages when they can not also find the content?Included Graphic by Shutterstock/Krakenimages. com.