Should you block your Search result pages?

[music] Well, hello, hello, hello everybody and welcome to a new episode of Search of the Record, the podcast where we from the Google Search Relations team take you behind the scenes of Google Search and hopefully have some fun along the way. My name is Martin Split. I am from said Google search relations team and with me is John Miller. Hello. >> Hi Martin. So good to see you or hear you. Hear you. Yes. Yes. >> Hear me. Hear me. Hear ye. Hear ye. >> Ah. So what did you bring us today John? You have a topic in mind. >> Yes. I thought we could talk about search results pages. >> Uh what do you mean >> our search results? Well, >> search results search >> also our search results but not not the normal Oh gosh, how do I position this? Uh, so basically [laughter] if you have a website, do you have a website, Martin? >> Yes, I do. >> Yes. Fantastic. >> More than one. >> Oh my gosh. Okay. Wow. >> Big domain farm. >> It also has search results because you can search on the website and find things there. >> Exactly. That's what I wanted. Yes. >> Ah, those search results. My search results, not Google, >> your search results, my search results. Okay. >> Or anyone else's search results pages. So, if you have a website and you have a little search feature maybe on the top or on the bottom where people can say, "Oh, let me search for Swiss cheese on this website because I know this is from a Swiss person. It must have Swiss cheese. >> Where is the article or the photo or whatever about Swiss cheese?" This is something that a lot of sites do. I think it makes sense for a lot of sites, especially if you have a non-trivial number of pages. >> If you have maybe five pages on your website, of course, people can just click through to those five pages. But if you have a bunch more pages or if you have a lot of different kinds of content, then having something like a search feature on a website generally makes sense. And pretty much all content management systems provide some kind of functionality around that line where it depends a bit on how you're set up of course where maybe they have something like a live index where they search through your database and show the results to the user and bring that up directly or they have some kind of other setup maybe even a third party that they're using to perform this kind of search service across their website. And the reason that I bring that up is not so much that like your search results and Google's web search results should be the same, but rather because Google's systems like Googlebot goes off and crawls your website and sometimes it will find your search functionality or sometimes it will find some links somewhere that go into your search functionality. So it might be for example that I don't know someone links to your search page for Swiss cheese and then Google doesn't know that this is any kind of a special page but rather it looks at that URL and says oh martinsplit.comarch whatever swiss cheese and then it goes off and tries to crawl your pages. >> Okay, >> which can be fine. Do you have Swiss cheese on your website? >> I do not have Swiss cheese on my website, John. But >> Oh my gosh. >> No, I'm sorry. >> How can you have a website in Switzerland and not have Swiss cheese? >> This is forbidden. >> Oh my gosh. >> Google will count that against you, I'm sure. >> Yeah. It's a quality strike or something, right? Like we get Yeah. Yeah. Sure. Sure. No, but is is there something on a technical side that I need to do if I have search result pages on my website? So there are two things. One is more of a technical thing which is if Google finds these search results pages within your website. It can happen that it crawls those pages. Maybe you have related search results linked on the bottom. Maybe someone else links to a bunch of search results and that can result in Google basically finding an infinite number of URLs on your website. M >> so if you have this search feature someone could link to every word in the universe and say like check Martin's website for this word which is unlikely but it could happen and from a practical point of view we consider this a so-called infinite space because it's an infinite number of URLs that potentially exist here >> depending on how your website kind of deals with these search results it can be that you also have maybe some filtering mechanism for different categories or different types of pages or it's like alphabetical order or date order or time order or I don't know like any other kind of ordering and all of that basically means we find or we could potentially find an infinite number of pages on your site and I know some people are like wow if I had an infinite number of pages in Google then I would be king but Having an infinite number of pages known to Google is not a good thing because Google will try to crawl all of those pages. And you can imagine what what happens uh when we see I don't know 100 million pages that are new from Martin Split will go off and try to crawl those. >> Interesting. Okay. So that's something where where then suddenly crawl budget becomes a question and your your server load and your server is like oh my gosh this Google bot is crawling infinitely I am going to block it because who knows what they have planned could happen I mean it's it's not unreasonable if you suddenly see millions of requests coming in and especially search pages tend to be something that are not cached so that means that every time someone accesses one of these search URLs, the system has to go off to the database and do a lookup and figure out like what is the best matching thing and then it has to run a ranking algorithm over your pages and figure out like which ones should be shown on top, what is a snippet and all of these things. So that's something which I think from a technical point of view can easily cause problems. Of course, because the internet is so big and so many websites do search pages in different ways, we try to recognize this. So when when our crawler systems come across a website, see it's like, oh, this looks like a search results pattern. This looks like an infinite space. like we could potentially find tons of URLs here then we will try to dep prioritize that. >> So it's like we try to do some things but it would be easier if we didn't have to do these kind of things right. >> Ah yeah true. Plus, it might actually like depending on how efficient the search algorithm is on your page, it might actually cause your server quite some work and actually might make the site even slower for other people, right? So, you want to avoid that. And >> yeah, okay, >> that's kind of annoying. I mean, it can be really annoying depending on what actually happens and how your site is configured. >> I've seen sites that link from one search results page to maybe 10, 20 other pages with like, oh, have you tried this query? What about like if you add an attribute to your query and then we're like huh okay wow 10 more pages wow let me let me get those too. >> Oh no. >> So that's something which >> we try to encourage people to avoid and there basically two ways that we recommend dealing with that. Uh one is with robots text where you can say it's like oh by the way this search page you don't have to crawl it. It's like my site is completely crawable. It has clean categories. It has maybe a sitemap file, maybe an HTML sitemap, maybe an LLM.ext. You don't have to. [laughter] Sorry, I had to >> Okay. >> Uh, back to serious face. So, it's like maybe your site has all of these features to make make it crawable in a normal way and you can just use robots.ext text and say everything in search is blocked. Like you don't have to go in and look at these at all. I think that's probably the easiest approach. That's also something that a lot of people can add even if they're using some kind of content management system. If you're using WordPress or Drupal or Joomla or whatever else is out there nowadays, you can easily usually edit your robots.ext file and say like by the way search disallow. M >> so that's one approach. The other is to serve the search results pages with no index in either the HTTP header or with a robots meta tag which is slightly different in that Google can still look at those search results pages but then it sees like oh there's nothing for me to do here. I will stop at one. It's like I might find a bunch of these, but I'm not going to crawl infinitely through this pile of pages. And that's also something that our systems when they do scheduling, they try to learn and say like, oh, this batch of pages is all no index. Like, you don't have to deal with that. >> Okay, that makes sense. But if I don't have full access to the source code and I don't have robots txt, then I just have to hope for the best. Or is Google like detecting this kind of stuff or is it even like treating my site differently if it finds one of these infinite crawl spaces? >> Um, it's surprisingly common to run across these kind of infinite spaces. So we do try to recognize it but I don't think we have any information in search console where we tell you it's like by the way we found this and we fixed it for you and it's always if our systems have to automatically recognize it maybe it takes a while for them to recognize that and depending on your hosting setup maybe that it takes a while is unproatic. Mhm. >> Maybe it's something where you see like, "Oh my gosh, my server is going down. I need to do something immediately." One thing that I've seen people do or try to do to fix that is to use the removal tool in search console. >> Okay? >> Because like it feels or it could feel like no index and removal tool is kind of the same thing. But the difference is the removal tool basically filters it out in the search results pages. It doesn't prevent crawling. >> Uh so >> okay. >> Yeah. So if Google has gone off and crawled all of your search results pages and keeps crawling them, using the removal tool will hide them in search, which is maybe okay, but it won't prevent crawling. >> Yeah. It doesn't solve the actual issue, right? It just hides the symptom. Oh, >> exactly. Which I could imagine if you're working for some site and someone's paying attention and seeing this, maybe they think it's fixed if you hide it like this. But, uh, definitely not not the way that you should handle it. >> I mean, other things that you could do is maybe serve a 404 result code for these search pages. I don't think it's very clean >> because then also some users might have issues with caching and things like that. True. What you definitely shouldn't do is serve something like a 500 uh result code which like theoretically you could also it's like oh by the way this page is not working for you but what happens there is that our our systems will recognize this and say like oh we tried to crawl and the server basically broke which means maybe we're crawling too much and then we will reduce crawling overall for the whole website including all of the pages that are not in your search results pages. So definitely don't do that. >> Okay, that makes sense. Okay, so now I guess robots txt is the best way to go about this. >> I think it's the easiest way. No index I I think is almost cleaner because you avoid kind of this potential state of a roboted URL being shown in search. I think it's very unlikely that if someone searches for something where you actually have content that we would show a roboted URL because we actually have better content for those queries, but theoretically that could show up. And with a no index that's prevented with a no index though, I was like we would still crawl the page. >> So it's pros and cons. Okay. >> Oh, may maybe another thing worth mentioning. I've seen some CMS's in particular Blogger does this. I think I've seen some others which basically take search results pages and use them as tag landing pages. So if you have different categories of blog posts for example and if you click on that category name then what Blogger does is basically a search for that category. Oh, >> and usually you would want a search engine to be able to crawl and index category pages. You wouldn't want them to be treated the same as a normal search results page. So that's something where I'm now that I think about it, not 100% sure like how you would manage this on Blogger. Maybe there is a URL pattern that you can differentiate between this kind of tag category page link and a normal search result. Then you could just block those search results pages and leave the category pages because I think category pages in general are very valuable. They make it a lot easier for search engines to crawl a website to understand the context within a website because then you really have this cleaner understanding of here's this category of things and here are the individual things that belong within this category. >> You almost have that hierarchy within a website a little bit clearer. >> Okay. Interesting. Now, I can't just have robots rules for every possible search term because I don't know what people will be typing in. And we know that we keep seeing new things people type into search boxes. But I I'm always nervous whenever there's patterns or wild cards being used in robots rules because you have to be a little careful. How would you go about that? Would you try to have one rule as broad as possible to filter out search results or >> Yeah. Do you split them up or how how do you do that? >> I would try to have one rule that covers everything. >> Okay, >> from from my point of view, if you have one clean rule that covers the whole pattern, then it's easier to maintain. It's easier to recognize which rule applies. >> I'm sure you've looked at robots.ext text files and someone has come to you and like why is this one page blocked by robots and you're like >> let me look at your labyrinth of text >> which >> most websites have very short robots at text files but some really try to be nuanced and that makes it a lot harder. So for example, if you have something like slash search and then question mark q equals whatever for the different queries, I would just block slash search and then question mark. And that way you're not blocking things like search for cheese.php, but rather like anything that is just in this slash search and then that follows with some kind of URL parameter. Okay. But you would try to get one rule that is trying to cover as much ground as possible. >> Definitely. Yeah. Maybe another thing worth mentioning around search results pages is way back in the early days in our web master guidelines when they existed as web master guidelines. I think we also listed search results pages as something that you have to block from Google. And we listed it I think either as automatically generated content or something like these infinite spaces type things because theoretically it's an automatically generated content. It's also infinite space >> and we recommended blocking those pages in the web master guidelines. Nowadays, we don't have that listed in the search policies, but it's still something that I think just purely for technical reasons makes sense. So, it's not necessarily something where we would say we think your site is spam if you allow your search results pages to be indexed. It's just like >> you're you're being very inefficient. Yeah. And that's up to you. It's like go for it if you want, but uh it's it's very inefficient. >> Interesting. Okay. So, we are not seeing this as a quality indication or a quality issue. It's more like a well, why are you doing this? What's the value behind this? Basically. >> Yeah. It's almost like, well, if you have this under control, it's like that's up to you. Just like we don't have any quality guidelines for calendars. It's like, oh, you should not make your calendar accessible to the year 2099. It's like if you do it's like whatever but it's like up to you what you serve there. >> So, but that would mean that if I have a few searches that I know are super important for me to show up in search results. >> I don't have to block them for reasons like so I can All right. Okay. So you could theoretically do something like a nuanced robots.ext where you say I'm blocking all of search and I'm allowing this more nuance pattern within this search results page. >> If you link to that that URL from within your website, we'll probably pick that up and show that. And that could be kind of a replacement for category pages for a handful of things. Um, but it's like that's almost something where I would say probably you could do a better job if you just created a real category page. Yeah. >> Rather than creating like this, I don't know, like weird mix of search results page, category page that is >> kind of like when someone lands there, they think it's more like a search results page. page and if you made a category page of your own, you could provide more context like why is this term important and which of these pages are actually the best on your website for this term? >> Yeah, I was wondering like maybe there are some people who have good reasons. Actually, I'm interested if you have a good reason for having your search result pages in the index. Let us know in the comments because I besides the scenario, it's interesting. I thought of the exact same scenarios like what if my system doesn't have category pages, but I can use or abuse search results to do that. But then I mean like some some websites probably have some legitimate reason for indexing their search results, but I'm not so sure what that would look like. >> Yeah, >> it's good to know that it's not like a penalty thing. >> There is one place where you could run into quality issues though with search results pages. Oh, tell me more. >> Namely, if you let people search for things that are totally irrelevant to your website and your search results page >> includes those terms on those search results page and is indexable. So, for example, like you said, you didn't have any Swiss cheese on your website, but maybe you have photos. So, if I go to your search page and search for Swiss cheese photos, then perhaps your search page will be like, "Oh, I have photos. Here are some photos." And you search for Swiss cheese photos. And puts it in a big heading on top. Because suddenly now if that page is indexable then it can happen that your page shows up for the term Swiss cheese in the search results because suddenly you have this page on your website where it mentions Swiss cheese and >> it's like Swiss cheese is pretty I think innocent. It's not so much that you'd be like oh my gosh someone hacked my website to show up for Swiss cheese. That's terrible. I'm lactose intolerant. How dare they? I don't know if you are, but >> I'm not. >> Okay. [laughter] But you could imagine someone doing that and saying, "For Swiss cheese, call this number and then photos." >> Yeah. >> And then your search page is like, "Oh, photos. I have some photos." And it has a page with a big heading on top that says, "For Swiss cheese, call John." and with this number [laughter] and that could be problematic. So of course with Swiss cheese probably like nobody will care like if someone basically abuses your website to show Swiss cheese but you could imagine casinos, pharmaceuticals, any kind of restricted content, any adult related content. It could be suddenly like all of these long adult terms and then a phone number and then photos in the end and your website basically has a page for these terms. Now with that phone number then suddenly your pages are showing up in search for these problematic queries. And it's not so much that someone has hacked your website to do this because your website is doing that freely and basically saying like, oh, you search for photos, here's a photo, but because it's accessible for any search term that comes up, it's suddenly a liability. It's more like a vector for other people to spam. And we've seen that happen that people do that at scale. They will try to recognize common CMS's that don't have their search results pages blocked and go off and link to thousands of sites with millions of pages all with maybe some adult terms and a phone number or pharmaceuticals and a phone number or something else and a telegram address or some other kind of contact mechanism where the goal is not so much that people go to your site and kind of see your photos but rather that in the search results they'll see for these pharmaceuticals call this number on martinsplit.com and sometimes that does show up for these kind of queries and when we see that happen we might flag that as hacked. So in search console you might see that as something that is flagged as hacked. >> It can also be that algorithmically we recognize that and say like this is clearly content that doesn't belong on this website. we will try to block it which works a bit in your favor where it's like oh well Google took care of the problem but maybe Google only took care of part of the problem of part of these kind of problematic queries. >> So it's a lot better to take care of that at the source and say everything in search I'm going to disallow by robots.ext text or all my search results pages have a no index tag on them so that like it can be seen but like clearly not indexed like that. So that kind of plays into the same pattern. >> It's interesting. I I didn't anticipate people would do that, but it makes perfect sense like your services show up in search riding the wave of another website basically. That's an interesting strategy. And uh yeah, I don't think relying on automatic systems in general is a good idea in terms of like, oh, Google will fix it for me. Yeah, but we do the best we can, but that doesn't mean that it's like reliable and always 100% working in your favor. >> I think that's definitely a case. It's also not the guarantee that we recognize it immediately. Maybe it takes a month >> and maybe during that month some normal customer of your website looks at your your website or does a site query or something and sees all of these pages where it's like I'm searching for martinsplit and then it's like four pharmaceuticals call this number martinsplit.com. >> It's like >> it's that's not great. And it's just so easy to take care of this from the start. And if you're working on multiple websites for other people, just putting that in robots text by default makes it so much easier. >> True. Okay. All right. So, we learned a bunch of stuff about search results on websites. So they can be infinite crawl spaces because we can like basically generate pages upon pages of these and maybe we even link to like did you mean and then we create even more that the crawler sinks into and it sounds like they're relatively easy to get rid of with robots txt andor no index. Huh. >> Yeah. >> Okay. All right. So well I think that sums up that. I didn't know where you were going with search results. I'm like, "Oh, yeah. Search result pages are looking different every day. Maybe he wants to talk about a new feature that I haven't even seen." But okay, now somehow my website needs to take care of this. >> Yeah. >> Interesting. >> I fixed it differently on my website in a really lazy way. Basically, I have the search query box on top and if you click [snorts] the search button, it redirects to Google and does a site query. [laughter] >> That is brilliant. And that's one way to do it. I might just remove the search page because my pages are relatively easy to find. I don't I don't know why people would search. Anyway, so cool. Thank you so much. That was really really useful and insightful. And uh I thank you all out there for listening into this episode. And uh of course John for joining me. Thanks for being here. >> Yeah, thanks for having me. >> So we hope that you all out there had fun and learned something today. And um if you're interested in more content like this, then definitely subscribe to the podcast and leave us a like. And if you have to tell us something, then please do use the comments. We're looking forward to hear from you all. Also, what kind of topics you're interested in in future episodes, leave that in the comments. We are looking forward to hear that from you. And with that, I'd like to say thank you all for listening and goodbye. Bye. We've been having fun with these podcast episodes. I hope you, the listener, have found them both entertaining and insightful, too. Feel free to drop us a note on LinkedIn or chat with us at one of our next events we go to. If you have any thoughts, let us know. And of course, do not forget to like and subscribe. Thank you so much for listening and goodbye. [music]