.Google.com's John Mueller responded to a question about why Google.com indexes webpages that are prohibited from crawling through robots.txt and also why the it is actually risk-free to dismiss the associated Search Console records regarding those crawls.Bot Visitor Traffic To Inquiry Specification URLs.The person asking the inquiry recorded that robots were generating hyperlinks to non-existent query parameter URLs (? q= xyz) to webpages with noindex meta tags that are also shut out in robots.txt. What prompted the question is actually that Google.com is creeping the links to those pages, getting blocked out through robots.txt (without noticing a noindex robots meta tag) at that point getting turned up in Google Look Console as "Indexed, though obstructed through robots.txt.".The individual talked to the complying with concern:." But below is actually the large question: why will Google mark webpages when they can't even find the web content? What is actually the conveniences in that?".Google.com's John Mueller verified that if they can not crawl the web page they can not view the noindex meta tag. He also produces an intriguing mention of the site: hunt operator, suggesting to ignore the results because the "average" customers won't view those outcomes.He wrote:." Yes, you are actually right: if our experts can't crawl the webpage, our team can not view the noindex. That pointed out, if our team can not creep the webpages, after that there's certainly not a lot for our team to index. Therefore while you may observe a few of those web pages along with a targeted website:- query, the average user will not view all of them, so I would not bother it. Noindex is actually additionally great (without robots.txt disallow), it simply indicates the Links will wind up being crept (as well as end up in the Browse Console record for crawled/not catalogued-- neither of these conditions induce problems to the remainder of the site). The integral part is actually that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the limitations in using the Web site: hunt advanced hunt driver for diagnostic reasons. One of those explanations is actually due to the fact that it's certainly not connected to the routine hunt index, it is actually a different thing entirely.Google.com's John Mueller discussed the website search driver in 2021:." The quick response is that an internet site: concern is certainly not meant to be full, nor utilized for diagnostics purposes.A web site inquiry is a certain type of search that confines the outcomes to a certain site. It's basically just the word internet site, a digestive tract, and afterwards the internet site's domain name.This query confines the results to a particular site. It is actually not indicated to be a detailed assortment of all the web pages from that site.".2. Noindex tag without utilizing a robots.txt is great for these sort of conditions where a crawler is actually linking to non-existent webpages that are acquiring discovered by Googlebot.3. Links with the noindex tag are going to generate a "crawled/not indexed" entry in Search Console and that those will not have a damaging impact on the remainder of the internet site.Check out the concern and also respond to on LinkedIn:.Why would certainly Google.com index web pages when they can't also view the web content?Included Photo through Shutterstock/Krakenimages. com.