How to read the Indexing Report

[music] >> Bonjour, buon giorno, greetings, and hello. My name is Martin Splitt and I welcome you all to a new episode of Search Off the Record, the podcast where we, the Google Search Relations team, are taking you behind the scenes of Google Search and hopefully have some fun along the way. With me today is John Mueller, who's the mastermind of our team. Hi, John. >> Hi, Martin. So good to see you, or hear you. >> Yes, I have something I want to talk to you about and double-check that I got it right. >> Okay. >> I don't know if you remember, but recently at a Search Central Live event, there was a question about the indexing report, the coverage report in Search Console, where someone's like, "Well, there's this page that I have and it's indexed, but the Search Console report shows it's not indexed, and I why is it wrong?" Was basically the question. >> But they were very friendly. You make them You make them sound very angry, but I think they were friendly. >> That's because I'm German, I think. Like, "Was ist nein, nein?" Anyway, so I thought about this a little bit and I think I know what happened here, and I recently saw another example of that with someone on Reddit who thought they screwed up because they have a bunch of non-indexed pages. >> Yeah, I think it's something that a lot of people, especially in the beginning when they start using Search Console, they run into this kind of thing where they look at the page indexing report and they see, "Oh, there's like hundreds of pages that are not indexed. This is an error, clearly on my side, something that I have to fix." >> Yeah. >> And then they go off and try to do things to understand it or fix it. >> I mean, you even get emails about it, and that kind of urges you to go into action, and I think the mechanism we have for people to acknowledge that they understand what's happening is this like fixed confirmed fixed or Marcus fixed or something. And I think while generally a great thing, I think in this case it's a little it's triggering the wrong response or the wrong expectation, I think. Would you say that's maybe the case or >> I think maybe. Wow, maybe it it depends. I have to do it depends, otherwise this can't be a search [laughter] podcast. But it's something where we we try to use a similar UI across search console for a variety of different features, and that includes this marked as fixed button in a number of places. And in some cases, there are things that you can fix. Like if you have a bunch of 404 pages and you realize your server was set up wrong, you can tell us you fixed this issue, and we'll try to fix it. >> Yeah. >> On the other hand, if something else went wrong in indexing and it's more like our systems decided we weren't as curious as you might be, then it's not something that you necessarily fix yourself. It's not something purely technical. >> Yeah. >> Right? >> generally agree, and I think I had a talk with someone from search console, with Hillel from the search console team, brilliant guy, lovely person, very helpful. And I think that opened my eyes to how we should probably talk about this and how people should probably look at this feature, and it's slightly different from what I think people are looking or how they are looking at it now. >> Okay, tell me more. >> So, all right. So, he said, "Oh, I wish people would not look at it as a list of things to fix. That's what we kind of have told people, and also not a static inventory that you need to monitor. Like, oh, is is this page still in index? Is this page out of the index? What's happening here? >> Mhm. >> Instead, he says it's great to look for patterns and to look for things that are unexpected and then dig deeper into that. >> Okay. >> It's an interesting way of looking at it, I think, because it's not like, is this index? Is this not index? It's a different thought. It's like, is the site in the index doing what I expect to do? And I think it can help you to spot things that haven't happened the way you intend them to happen. So, for instance, if you're migrating >> Okay. >> and you're setting up redirects, then indexing report shows you, "Ta-da! A steep rise in in redirects. Page with a redirect. It's not index because it's redirecting somewhere else." And if you look at it as an inventory, as a monitoring tool only, then you're like, "Ah, what's happening? Why is there pages that are not index because they're now redirecting?" When in reality, that's exactly what you want them to do. >> True. >> Because you just set it up. And on the other hand, a problem would be if that wasn't showing up. Well, a problem. I say a problem, but like, it takes a while for search to see this change and actually process this change. And you know that once it's reflected in the data, aha, search has understood this. As long as that hasn't happened in the index report, then you know, ah, okay. Uh, interesting. We have to wait a little longer because the processing hasn't completed yet. >> Yeah. >> And I think that's a really powerful tool. And it also then doesn't matter as much that the data is not always real time. That's the other thing. Like, people publish a new section or a new category on the on the website and then they're like, "Ah, they are not indexed." And in reality, it might just be that we have a data delay or it might take a few hours longer. We sometimes do have longer delays, but we normally communicate these in Search Console, right? We have these like little >> Yeah. >> notifications that we send out. But yeah, so I think that's really, really handy. The other thing is that sometimes you see changes that you haven't triggered. I mean, with a redirect or like if you remove a whole category of stuff on your website, then you expect to see a steep rise in 404s because you removed stuff from your website. And again, if that hasn't happened, then that tells you something has gone wrong maybe or something hasn't happened yet or hasn't been processed yet. >> Yeah. I also think, especially with 404s, that's something that sometimes throws people off. >> Yeah, yeah, yeah. >> Because they're they're labeled as an error and it's almost like your website is returning an error and then a lot of people see that and say, "Well, I have to fix this error." >> Yeah. >> And I have to make sure that Google doesn't see a 404 because if Google thinks my website has errors, surely Google will think my website is not as useful. >> Yeah, yeah. It's not good. We need to fix this. It's an error. You want to fix an error, right? But it's an error that is expected. It's actually a good thing. The problem is that and then I get the question like, "So, but why does Search Console show it as an error?" Because it is an error. It's just an expected one. >> Yeah, I I think that part is always tricky for for people in the beginning and especially tricky if you have, I think, management that doesn't understand it so much and management tells you it's like, "You should make sure Search Console is always green. Everything's is okay." And then people might be worried that's like, "Oh, I have this error report and this number is not going down." "What What do I need to do to make this error number go down so that my boss thinks I'm I'm doing a good job. And actually, if you're removing things from the web or if these pages never existed, >> Yeah. >> they're supposed to return an error. That's actually the right thing. >> Yeah, absolutely. And And the other thing is so it can't judge for you if this is a problem or not. You have to do the judging yourself. And the other thing is sometimes it's changes that you haven't caused. Like page with a different canonical. That's something that makes sense to look into, but it's not something that should alarm you because that just means that the report as is is now reporting for different URLs. And I know that the numbers then sometimes look like ah, it's going down, but then that usually means it's going up somewhere else. And I find that useful because it tells you how Google thinks about your site and your URLs and if if you have like I don't know, dub dub dub example.com home/blah, it might be that external people are all linking to a different version of that, the non dub dub dub, just example.com/blah. And and maybe Google thinks, "Hey, you know what? This is shorter and everyone uses this, so maybe we use that as the canonical." >> Yeah. >> And that can help you change your canonicals to something that everyone else would prefer anyway. And I think that's a cleaner way of doing things, but it's not a problem that you need to fix, right? >> Yeah. I I think fundamentally, the question of whether you should change your canonical to what Google thinks your canonical is or not, >> That's up to you. >> That's a very different question. >> Ah, fair. >> But uh the aspect of Google picking another URL within your website as a canonical, that happens all the time. And that's normal. >> And it's not something that you need to worry about because it's just it's still in the index. People can still find it. >> Yeah. >> Normally. >> Exactly. >> Yeah. >> That's something, by the way, that is a little bit easier if you have a domain property set up in Search Console. >> Yeah, that's true. >> Because then you don't have to worry about dub dub dub, non-dub dub dub, and whether you should say W or dub or whatever. >> Mhm. >> All of these things are a little bit easier, and basically all of that is shown in the performance report. >> That's true. I do think it is useful to see the site from a kind of like bird's-eye view, and I really, really enjoy the report for that. So, I wouldn't use it to to look specifically after like, "Oh, this page needs to be in the index, and this page isn't in the index, and ah, what do we do now?" But more like, "Are we seeing some pattern moves, and why are we seeing them?" >> Mhm. >> And for that it's it's great, because you can see like, "Oh, picked a different canonical for a lot of pages." You can click on that, you get a bunch of examples, and then you can analyze one or two of these examples, and then basically get a feeling for, "Ah, okay, so that's happening on the page." >> Yeah. >> And sometimes you can basically just conclude, "Well, everything is fine. There's an a shift of something, but the shift was to be expected, or at least the shift isn't malicious." Right? >> Yeah. >> So, I think the way that Hillel presented this makes a lot of sense to me, because it's more like a don't think of it as inventory. And we have a similar problem with the site query as well. >> Yeah, Hillel. >> Where people are like, "It's not showing up for the site query, so it must not be in the index, but Google Search Console says it's in the index. So, what who's right?" >> Yeah. I think site query is an artificial type of query, so >> Yeah. >> that's something where partially you can use it to double-check if a URL is indexed, but Search Console is really kind of the the source of truth there, because sometimes we we don't show things in site queries, and sometimes we do show things inside queries that are not actually indexed like that. For example, if you do a move from one domain to another, and you do the move properly with redirects and all of that, then basically, if you ask Google about the old URL, the old domain with a site query, Google will tell you. It's like, "Oh, yeah, I know about this." And it's like, "Here's a bunch of URLs." And this will happen maybe a year or two or even three afterwards, after you do the move. And if you look at purely the site query for that, you might conclude that, "Oh my, the site move actually didn't work. Like, Google is still indexing all of these old URLs." And sometimes, when you look at those results, you'll see in the title maybe the new domain name or something like that. So, you can tell Google saw the move, but it still knows about those old URLs. And if you explicitly ask it about something, then Google will be like, "Sure, I can give you what you're asking for." It's like, "I'm I'm not saying that this is the current source of truth, but it's like, you're asking for this site, and here it is." And it might be, like, if you're a normal user, you might might have remembered the old domain. You're like, "What happened to this old website that I really liked?" And if you look it up on Google, Google will tell you. It's like, "Here's the content that was there." And if you click on it, you'll of course be redirected to the new site. >> Does that sometimes also happen if you have hreflang set up and have different language versions? >> Probably, because the way hreflang works is the individual versions have to be indexed, but then, when they're served in the search results, they're swapped out against the local version. And I don't know, for example, what happens with the site query if you do, I don't know, site: something.ch, and uh you're in Germany, maybe you see the German version, you just see the Swiss version. I don't know. Like >> Interesting. >> Someone should try it out. >> Please try it out and and write it in the comments. We're curious to see how that happens. Uh that would be interesting, yeah. >> I I think may maybe taking a step back to some of the errors that you mentioned. >> Mhm. >> One of the things that I think is worth watching out for is whether or not there any systemic issues with regards to your hosting setup. And this is something that you do see in the page indexing report as well. And it's a kind of thing where you look at and you're like, well, this is unexpected. Why are so many URLs suddenly considered 404 or 403 or I don't know, dropped for other reasons or even canonicalized to some other page. And sometimes we've seen CDNs, for example, or hosting providers, they have some kind of bot protection, and maybe they turn it on when there's a lot of crawling happening, and depending on how this bot protection is set up, it can happen that suddenly your CDN or your your website hoster serves 404 or 410 or 403 or something instead of something reasonable, at least from what I would consider reasonable, something like a 503 that basically says, oh, I can't deal with you right now, come back later. >> Yeah. >> Maybe if they think Googlebot is a malicious bot as well, maybe a 404 or something makes sense. But that's something I think worth watching out for. Like every now and then taking a look at that report to see has there been a significant change in the number of crawl errors or in the number of pages that are dropped from indexing? And if so, what are some examples that I can follow up on? The other one which I think is a bit harder for sites to diagnose is when the hosting provider shows something like an interstitial. Something like, "Are you a bot or not?" And it serves that with a 200 response code. And you can guess what happens. It's like we'll try to index that on the one hand, which will result in your page's content kind of disappearing and being replaced with this am I are you a bot page. But what also happens is because this page is shared across a lot of other pages on your site and maybe other pages across other websites, it can happen that we pick a canonical of someone else who has the same error page. And that's really challenging to kind of debug because you basically have to look at those pages and look at the canonical or look at the page that Google chose as a canonical when you look that up. And sometimes you have to almost like guess a little bit of what would happen when Google accesses this page because as a user you might not see that interstitial at all. But if you see a lot of pages on your site being duplicate eliminated in favor of some random other page, oftentimes that's a sign that you have some kind of a soft error on your page, uh some kind of soft block that tries to block crawlers and bots and basically just ruins your SEO. So, that's something where of course that's the kind of thing where I would say you should go off and try to get that fixed with your provider or your providers if you have a CDN and hosting, figure out where it's coming from. And then using something like marked as fixed in that report makes a lot of sense because then you're really saying like there was a significant issue on my website like you found all these pages and you canonicalize them somewhere else or you called them 404, but actually they exist. How dare you take away my pages? And then we will try a sample of those. So, the way the marked as fixed works is we try a sample of the pages that you're basically telling us are fixed and if we see that they're actually fixed, then in most cases, we will trigger a faster recrawl of the other pages. So, it's not so much that we wait and like, oh, well, we'll see if this is actually working better, but we'll try to recrawl that a little bit faster. >> That makes sense. And again, this is about patterns. So, if you see like a bunch of errors, then you can figure out, ah, so this is all the products because XYZ or all of this is all the site, so this is not just like a specific template that is misbehaving. And then it helps you troubleshoot a little bit faster because if it's just one specific template where pages are experiencing some specific problem, then it's probably something within your hands, so to speak. But if it's the entire site or an entire part of the site that is behind a CDN, then it's probably something to do with a CDN. And again, these patterns are worthwhile. Same with the trend lines. I think we haven't [clears throat] really talked about the trend lines in the report yet. So, not only do you get the reason for a problem, but you also get this line because sometimes in a server has a hiccup. There's some sort of problem. We are seeing some 500 something error, which can or cannot be a problem. That depends. If you're seeing a little blip, then maybe look into the server logs, like what happened here and then someone tells you, oh yeah, we upgraded the host, we moved between different computers in the data center, whatever. And then there's not much you can do about it and it's kind of fine because it's going to work itself out. But if it's a steep line going up, uh, maybe look into it. >> Definitely. Yeah, yeah. >> On the other hand, you might also see that and it's like a horizontal line and it's like a few blips, and it doesn't Yeah. >> Yeah. I think it's important that people realize that computers don't always work, and for the most part, the internet is built in a way that is supposed to be a bit resilient against that, and just retry. >> Yeah. >> So, it's like when when we look at the crawl stats of a website, every now and then you'll see things like, oh, I don't know, handful of requests failed, or handful of DNS lookups failed. And of course we want to report that, because maybe you care about those handful, but for most sites, it's something where it's like, oh, this didn't work this one time when Google tried it, but the next time it worked, and everything kind of worked itself out, and it's not going to be the case that if Google runs across an error, and it just exists for a couple minutes or whatever, that it's going to cause any issues across the site. >> Yeah. >> Even for the most part, it's when we run across errors, and they exist maybe for a day, which is like a really long outage in internet time, that's something that we try to kind of look over and say like, oh, maybe it's something temporary, we'll just kind of like keep things stable, and come back and check again. >> Mhm. >> But if it's if it's longer than a day, then of course our systems are going to react to that. >> Definitely, and you want to have a look at it. And also, if you add or change your site, or if your site is very new, then you can actually also use this report to see a little bit how your site goes through the different stages, because at some point you're going to see pages in discovered, currently not indexed, which tells you, we know they exist, but we haven't actually visited them, and if we haven't visited them, we can't put them in the index, versus crawled, currently not indexed, which means we visited them, and we didn't put them in the index, and that can have all sorts of different reasons. Would you say that that's is often or only sometimes a sign of a quality issue? >> Sometimes. So, it's definitely the case if our systems are seriously worried about the quality of the website that they will reduce the number of pages that they index. Because if we if we have strong concerns about the overall quality, then it doesn't make much sense for our systems to spend a lot of time on the website. So, we'll probably crawl a lot less, we'll index a lot less, and then you'll see things like crawl not indexed or discovered not indexed, which from from our point of view is basically our system saying, "We know about this, and once we're happy >> We looked at it. >> once we're happy, we will take another look and see if we can index it." It's not so much that I would say you should take these situations and try to fix them. >> Yeah. >> Like from a technical point of view, it's not that you need to fix this technical issue that Google is not indexing this page at the moment, but rather you almost need to when you recognize a bigger pattern like this like Google is not indexing a lot of your pages and there's no technical reason, you almost need to take a step back and think about the quality overall. And thinking about quality is really challenging because a lot of times it's your website and it's your baby and of course it's the best baby ever. But taking a step back and trying to look at it with the eyes of someone who who's not directly involved with your website, sometimes that opens up some ideas for areas where you can improve. Where maybe if most of your website is AI generated and it worked for a while, it might be that people look at this AI generated site and they're like, "Well, I can tell this is AI generated. There's nothing unique or valuable that is available here for me. That's not to say that all AI-generated content is bad, but sometimes you just run across websites where you're like anyone could have written this. This tells me nothing. >> Yeah, yeah. That's true. And I think what makes this difficult is not only the fact that obviously the way you wrote it is the way you thought is best and that's why you think it's high quality, of course. So, that's really really hard to kind of step out of of your own perspective. But sometimes it's also there's so much other stuff that is just as good. So, why would we add it to the index? And then that can tell you like, "Maybe >> Yeah. >> this content isn't as valuable as I thought it is because other people are covering the same thing and then what's the value of this version of it being in the index?" >> Yeah, that's true. >> I feel we we could have a whole podcast about quality. I think maybe one other thing that is worth mentioning with regards to quality is it's not just the text. So, a lot of times people will say like, "Well, my text is unique or I my articles are good and they're packaged in a page that is terrible to access where anyone who when they try to load it like their computer fan spins up and they're like, "Oh my gosh, I have to run away to make sure my computer doesn't explode." So, maybe that's an extreme case, but you you've all seen these pages where basically the text is there, but it's almost hidden away, hidden behind ads, hidden behind interstitials, hidden behind other things that are moving and coming and going, maybe hidden below a bunch of filler content which we sometimes see for example with recipes where there's this really long story on top that maybe most people don't really care about and then the recipe comes. These are all the kind of things where the overall quality is much more than just that piece of text that you say like this is my main content. This is what Google should be counting for my site. And from our point of view, we we almost have to take into account the full experience on a page because that's that's what users see. It's not that users go to a web page and turn on some magic mode that just pulls out the text, but rather they have the full experience of this website with all of the 3D 4D animations and everything. >> I agree. I very much agree. Oh my god. The other thing is there's no such thing as a ratio between index and non-index pages. I mean, there is, but like it doesn't matter. I've seen so many healthy websites that have like a million non-index pages and like half a million that are in the index and they're doing perfectly fine. >> Yeah. >> So, I think that's the other thing like a lot of people are seeing this and they're like, "Ah, Google probably thinks my website is low quality because only I don't know 20% of my pages are indexed." No. That's not something that you need to worry about. That's not something that tells you like this is good or this is bad. >> Lots of it depends. >> Lots of it depends. I think that's something that people don't normally know. >> Yeah, when I take a look at the search console for our developer documentation, for example, I think it's something like 5% of the pages are indexed. >> Wow. >> Which when you look at that report, you're like, "Oh my gosh, something must be seriously wrong here." But when you look at the details, you're you see it's like, "Oh, a lot of them are no index. There's a bunch of 404s, like canonical things." Like the important content is all indexed. Uh we see that important content also in the performance report, so all of that is okay, but if you just look at the page indexing report, you're like, "Whoa, this looks like something is seriously broken." >> [laughter] >> I wonder how this looked when we did the big migration from the old structure to the new structure that we did a few years ago. That must have looked really chaotic. >> Yeah, I imagine some of that, but also probably a lot of code samples and probably I don't know from the other different product areas that are hosted on the same site. Maybe they have different setups where they say like a lot of this content should be no index. Maybe I don't know if you have APIs with older versions, maybe you say all the older versions are no index, which is a decision that you can make and is fine from our point of view. It's just in the report, you'll see that reflected or you usually will see that because we look at these pages and initially go and say like, well, maybe we should index this and then well, maybe not. >> [laughter] >> Maybe interesting, but I think we're good. Yeah. >> Yeah. >> Anyway, I I think that's summing it up roughly what I wanted to bring out there. So don't think of it as an inventory that you need to fix. Don't think that everything on the website needs to be indexed. Don't think that the ratio between index and non-indexed is some sort of like quality measure or some sort of score that you need to look at. I think as long as everything you care for is indexed with some URL and again, domain properties make this easier because you can see if if canonicals just change and don't have to worry like, oh, this dropped out because of canonical and then you have to like figure out, oh okay, so the new canonical, that's fine. Yeah, so look at it as a detection tool for changes or patterns in the site that you expected or didn't expect, right? So if I make a change, it should reflect in this report in the pages there as well as if I didn't make a change and something happened, what happened and is that a tendency, is that a trend, is that something new, is that something that always has been like I've just seen for our dev site actually, we have like a lot of 500 errors, but I mean a lot. We have some 500 errors, but it's stable. So, if this hasn't been a problem yet, then it hasn't been a problem like a month ago and it will not be a problem right now because it's it's the same amount. Some requests will probably fail for whatever reason, maybe a solar wind, who knows. >> Yeah, I think I looked into those for the developer site at some point and uh some of those are things like API requests. >> Oh. >> Where the documentation links to some API requests and then the developer site server says it's like, "No." It's like, "That is a bad URL and I'm not giving you a 404 because I kind of understand what you're trying to do, but you're giving me a bad request." >> Oh, I see. Interesting. So, that's where that comes from. Huh. >> And that I think especially for things like technical documentation and website technical documentation, I don't know how you would call that, where basically you're describing URLs on your website for technical reasons and you're giving examples, then you see a lot of different kind of errors. Whereas, if you have a blog about recipes, then you don't see a lot of like bad API requests because you're not writing about any APIs. >> [laughter] >> I see. Interesting. Yeah, I wouldn't know how I would call these requests either. Anyway, so I think we looked at the indexing report quite thoroughly and I think it makes a lot of sense to treat it differently than some people are treating it today and I hope that we took away some scare out of this report and made it more useful for more people. So, um thank you all very, very much for listening and thank you John for joining me and discussing indexing report with me. >> Of course, happy to be here. Thank you, Martin. >> Thank you. So, we hope you had fun because we know we did and please leave us a like and subscribe and comment if you are seeing different language versions in the side query and if you want to hear more from us, definitely check out our future episodes, maybe check out our earlier episodes if you haven't listened to them yet and cheers and bye-bye, everybody. >> Bye. >> We've been having fun with these podcast episodes. I hope you, the listener, have found them both entertaining and insightful, too. Feel free to drop us a note on LinkedIn or chat with us at one of our next events we go to. If you have any thoughts, let us know and of course, do not forget to like and subscribe. Thank you so much for listening and goodbye. >> [music]