Flock 2026 Scrapers Gotta Scrape Scrape Scrape

Okay, my name is uh Kevin Finsey and uh I'm going to talk to you today about a lovely scourge that we have uh going on uh with scrapers. Um so when you think of the word scraper uh it it has a particular connotation, but um yeah. So so we're going to go kind of over Oops. Come on. Now it's not letting me. Oh, there it goes. Okay, so uh we're gonna I'm gonna kind of go over a few topics here. Um kind of talk a little bit about the history of pulling things off the internet and scraping content and how it was in the past and how kind of things have changed. There's uh a lot of different types uh of scrapers. I'm going to kind of go over uh what they are. uh how they look from my perspective. Um then I'm going to talk a little bit about impacts that we've seen in Fedora infrastructure from this stuff along with uh some impacts that have appeared in the media or other open source projects uh that have also seen the same sort of stuff. Uh then I'm going to talk a little bit about about defense. you know, what you can do to protect yourself or weather the storm uh from these these uh scraper guys. Um, and there's a variety of things we'll talk about there. And then I'm going to talk a little bit about what happens if this gets worse or there's no solutions to it or things like that. And I'm I'm not going to sugarcoat it. It doesn't look very good. Um, however, in the end, I'm going to talk a little about some more optimistic possibilities. And there's going to be a lot of speculation in that section of things. I don't actually know what the intent of the people who are doing this is. I just just am able to speculate as to why they might do it or how they're doing it. And I have have some good ideas about how, but the motivations are are just speculation. We don't actually really know uh in the end. Uh if if anyone has questions, go ahead and shout them out or I'm hoping to have some time at the end also. So So a short little history of uh web crawling. Um web crawling has been around a really really long time and there's a lot of very uh well-known tools to do this. uh you know there's crawler bots written in Python and Pearl and all sorts of things. Um the uh one of the first standards that came out of that uh is the robots.ext standard uh which is a RFC and this came out of the fact that some of these people who were scraping content uh caused problems for the sites and of course in those days at least everyone wanted to be polite and not cause problems for other people. And so they got together and they thought, well, how can we do this in a way that doesn't bother the actual site? So, uh, robots.ext is is just a little text file that that tells these crawler bots that are well behaved, uh, you know, how often can you pull content uh, how often you can refresh it? Are there certain paths on a website that they should not use? Uh, things like that. And it's completely voluntary. there is uh no requirement to pay attention to it and indeed uh lots of people do not but in in the early days that was uh that was something that people did actually pay attention to. Um let me see where is that? I had a little note here. Oops. Yeah. And I'm trying to find it. Uh, okay. Nope. Anyway, um, so there was a ton of norms that that spiders had at this time, like, you know, how often you can crawl, how often you can refresh, that that kind of thing. But again, these were all norms. They were not any kind of regulation, any kind of requirement, anything built into the protocol. These were all just like politeness uh, type things. Um, tons of well-written building blocks. Uh, like I said, there's a lot of scripts that uh that handle this stuff for you. You can instantiate a crawler bot fairly easily with with some of these uh uh building blocks. Uh, and finally, I wanted to mention archive.org. Uh, how many of you are familiar with archive.org and and have used it? Yeah, almost everybody. Um, so archive.org org has been around a very long time and uh they were established back in this era when everybody paid attention to robots.ext and so in their crawling they're actually very polite. It's very hard to actually notice that archive.org is crawling your site because it's uh at a slow rate and done very politely. Uh, of course, search engines uh in this era were starting up Google and the like and they also had kind of a a good incentive to uh be polite and not cause problems for sites. Sometimes they did, sometimes there were issues, but overall like the major search engines were honoring the the norms of the time, if you will. Uh so um there's a number of different types. I've kind of divided this stuff into types. And the first type, which I'm calling type zero, is the one that I've just been talking about. So oh, it did not [sighs] advance. There we go. Uh the first thing I'm the first type is this type zero. And this is the one that I was just talking about. They're generally well behaved. Um occasionally there's a runaway script. somebody creates a a new web spider or something like that and they have a bug in their script and it just goes in a loop or it does something horrible. Um, these are usually from a single network or uh IP address. They're pretty easy to deal with. We've we've been dealing with these for 20 years. Most of the time they're fine. Occasionally there's a bug or issue and then you see, oh, you know, I'm getting a lot of requests from this IP address. I'll block just block them or uh oh the script I I can see is causing problems but I know who runs that script. I'll just tell them you know hey your script is causing problems. Uh we we have run into this in Fedora infrastructure. I remember a number of years ago there was some bot that was scraping something and I could see that it was coming from the uh from the Red Hat check office and so you know I contacted people there and I said can you ask around and see who is doing this and they actually found who was doing it and the guy was like oh sorry I didn't mean that to happen. Um so you can handle these pretty easily. They're uh fairly uh trivial. Oops. That did that advance. It advanced too much. Now it's not going back. Uh so many AV problems. [sighs] There we go. Okay. So the next type uh which I've been calling type one um these are something that just appeared in the last maybe 10 years uh these are all bit almost always cloud-based so like a particular cloud AWS or Alibaba or something like that um they come from a known block of cloud IPs and these usually don't pay much attention to robots.ext text. Uh, but they're usually relatively well behaved. And when they don't, they actually pass a user agent. Um, when when a a web uh connection comes through, you can pass a user agent, say if you're Firefox or if you're whatever. And these guys usually will pass, you know, Google bot or uh, you know, some crawler bot. So you can see you know what traffic is is from that particular thing. Uh these are usually real companies that are for whatever reason doing trying to get this content uh search engines or uh particular companies that maybe some company wants uh like all the through our sources for doing something. Um, so these guys again usually don't cause a whole lot of problems, but if they do, they are fairly easy to block also because they're coming from a known set of IP addresses and they're passing things like user agent that you could block on if you need to block on. Um, so they're not generally uh too bad a problem. Now we're starting to get to the uh the fun ones. uh what I call the type two uh are still mostly cloud-based and they're still coming from specific cloud blocks. So, a particular cloud provider, but these guys use random a user agents. They do not say they're XYZ crawler bot. They actually come up with random ones. And it's kind of funny because these random user agents are nonsensical a lot of the times. that are like, you know, uh, Mac OS 10, uh, Power PC, blah, blah, blah, you know, some things that do not actually make sense when put together. Uh, so it's just something is randomizing that completely. Uh, these are usually similar to the last type. They're they're well-known comp or they're companies, uh, but they're shadier companies. They're trying to get this data for some probably less legitimate purpose and they are trying to conceal themselves. Essentially, these guys do not pay any attention to robots.ext. Um, they sometimes are a problem and we'll get to some u the mitigation stuff uh for these guys. And here's the fun extra fun ones. So, uh, this is the the last type, the type three folks. They seem to come from everywhere. They're any IP address. They're residential IP addresses, cloud IP addresses, uh, different countries, just all over the world. Um, they also use a completely random user agent or none, uh, depending on the particular one. And these guys are almost certainly botnetss or reseller uh type things. Uh we'll get into at least a speculation of how these guys operate here a little later in the talk, but uh these are the worst type because it's very difficult to block them. There's nothing you can really pinpoint uh to block. There's not a specific IP address or network. There's not uh user agent. there's not a particular pattern to to any of this. So, these guys are the the really tough ones uh to block. So, uh let's talk a little bit about impacts uh of this stuff. Uh obviously they're some of the worst types like that last uh type type three guys uh do denial service and um you could certainly recall or notice if you do a lot of work at Fedora that we've had problems with these guys. So occasionally there's a service that goes down and it's like well the the bot scrapers are are after things again. Um, this can al also cause bandwidth costs. Um, especially if you're a small company or an individual user and you just happen to have a site up. Uh, this can really like drain your bandwidth and uh cause you some high bills. I I've seen posts from people that are like, "Oh, I had my source code up on my, you know, personal computer at home and it had a little web server and, you know, so I could serve it to my friends and the bots found it and I had to turn it off because it made my connection completely unusable and cost me money and bandwidth. So, uh, we'll get into that also a little later, the uh further effects of that. Obviously, there's there's CPU, memory costs, that kind of stuff. Um, and I I I kind of put licensing issues here because well, we'll we'll get into that a little bit more with the uh speculation type of thing. There we go. So um I want to talk about a few specific things that we have seen and I think a lot of people have seen um just as kind of examples of the various types and and what is going on. So um Fedora provides a hotspot.ext text file. And this is just a tiny little file that says okay in it. It's asy. It has two characters and that's it. And the use of this file is so you can detect if you're behind a hotspot. So if you fetch that file and you get a file that says okay, you know you're not you're you're directly on the internet. If you get HTML, some kind of portal page, you know that you're in a captive portal. You're behind a hot spot of some kind. So, we we set this up for network manager. I'm not sure if it's using it by default anymore. I think it might be. I'm not sure. Uh but we get a lot of hits on it. A lot of people do use it. And um when was it? Uh month or so ago, we got a lot of traffic. I noticed a lot of bandwidth being used. And it was this two character tiny little text file being fetched millions and millions and millions and millions of times. Um the bot just decided for whatever reason that this was great and you needed to to fetch it 100 million times in case it was different from different regions or something like that. Um >> it may it may be [panting] >> yeah it's this also was a problem for us because we don't do any caching or strange things to that file because we want it to always just serve that file nothing else no matter what. And if there was a caching layer and the cache messed up, then it would return a cache page and that people would think they were all behind a hotspot. Uh, in the distant past, we have messed up that file a couple of times for various reasons and then the reports flood in. Everything says it's a hot spot. What's going on? It's so it is a an important file. But yeah, that's that's not fun. Um, CSS fonts and uh legal pages. This is another thing. If you've noticed, any of the Fedora site pages have a little footer that's like, you know, Fedora project blah blah blah, privacy policy, blah blah blah, and they're all using kind of a common theme uh and CSS files, and those things get hit uh just a metric ton. Um, so we did do a bunch of caching on that, which I'll talk about in a in a little bit. But if you have things that are included in other pages, um that is is something that you're going to see a lot of uh scraper activity on costly endpoints. Um there's actually a really good example um both on source.forro.org and pagar.io. Um you may or may not have noticed, I don't think very many people have noticed. uh you can't pair has a a feature where if you're looking at source code or in a git repo you can actually uh click a get blame and see when each one of the lines changed in a commit. Well, that's disabled now. [laughter] And for this particular reason, because the bots would come, they'd hit a a get page, hit blame, and get links to every single one of those commits, and then follow each one of those commits, and then hit get blame again, and like just basically poorly uh translating the git repo via scraping HTML pages. So, uh we we had to disable that. We've had to disable a couple things on Cooji at times. Um there is a Cooji page that gives you RPM info. So you can click on an RPM, it'll show you all the packages in them in it and all the files and modes and all that stuff. And for a while they were really hitting that and that was causing huge database loads. So it would make a request, give me the RPM info for this RPM, and then Cooji would have to like pull it out of its database and format it and send it back and then it would request a 100,000 more of them. Um, so we did actually manage to mitigate that. That's that should be all enabled again. But that's an example of a expensive endpoint. So if you have something that hits a database backend or does a lot of complex queries or things like that, it's it's definitely going to be something that gets affected by this issue. Ah, and the last but most fun one. So OS tree, we ship OS tree repos uh for our atomic folks. And the way those are constructed, it's a tree and it's sort of like a git repository. It's got objects in it and refs and so forth. Uh but for its efficiency in the objects directories, it makes every directory have 32,000 objects because that's the the kind of most efficient way to to represent this. And OS tree itself does not look at those directories or list them or anything like that. It's just pulling objects out of those based on the other information that it has. However, we had unfortunately left indexes on in on that tree. And so the scrapers found out that they could just do a directory listing on everything. And of course they did. So the objects directories, it would do directory listings. And that would require the backend server that's serving that to stat every single file in there so that it could print the little pretty uh date stamp and size on everything. And some of those requests were taking like two and a half, three minutes and it would fire off a thousand of them. And so of course the server dragged to its knees. This has a simple solution. Obviously we turned off directory indexes but uh again that's a example of something that's expensive uh maybe that you don't even know uh is out there and the scrapers will find it. Um ah yes this uh a little segue here. Um so the scraper as I mentioned in the very beginning there's software to do spidering and crawling of websites. It's it's very well known. It it has been you know created over many years and bugs have been worked out and so forth. These guys, especially the the type two and three, are not using that at all. There is no efficiency whatsoever. Individual files are fetched hundreds of thousands of times by different IP addresses. Uh, and I don't know if that's just because they don't care. They have enough resources to not worry about reconciling or uh, you know, doing things in a more efficient manner. or perhaps it's that they vibe coded this thing and they have no idea what it does. Um so it we we don't know. We can we can only really speculate. No delaying. Um it does things over and over and over again. It gets into loops sometimes. If you have a file A that that points to file B that points to file C that points back to A, it'll just like keep going around that loop over and over and over again fetching all those files. um like the hotspot.ext was another example where you know it had that content. It's two characters. It is nothing rocket science there but it yet it still had to fetch it hundred of hundreds of millions of times. Um there's no error handling. So sometimes it'll hit an endpoint, get an error, but it'll just keep hitting that endpoint in the hopes of perhaps catching it when it isn't tossing an error. Um, so I I can only really speculate on what they're doing or or whatnot, but it is not efficient in any way. So, let's talk a little bit about mitigations. Uh, one of the first things that we did that's easy to do and is actually a win for other reasons, uh, is to set up caching, right? So if you have something that's getting hit a bunch of times, you can just cach it. And uh there was a lot of low-lying fruit in that arena for us. Uh we we had a lot of them hitting source. Project.org, but like every one of those requests was doing the CSS and font files. We cached those. Um there was just a lot of things that were very easy to cache and improve uh performance on and that actually benefits everybody, not just the scrapers. It makes things faster for everyone. Um [clears throat] of course you could throw more resources. We've had this issue as well. uh our wiki started getting crawled uh a few months ago and it was having trouble like just processing the pages page requests uh there. So we ended up just giving it more resources. We gave it more CPUs and then it was able to keep up relatively well. Um that may be an option. It it just depends on on what your infrastructure is like and and how it is set up. um stop serving things you don't want. This is a a good pointer back to that OS3 thing. We don't need directory indexes there. We can turn them off. Um there are certain endpoints that we don't need. Uh there have been services or things that we found that start to get crawled and we're like, "Oh, nobody uses that. That should have been removed in the first place." So, we just take it down and drop it. Um so yeah, stop stop serving things you don't need or want uh is another possible option. Um let's see. So uh just talking about the particular types, the the zero and one guys, you know, it's it's fairly easy to mitigate. You can block them by IP, you can block them by user agent, uh you can contact users, you can ignore them if they're relatively well behaved. that it's pretty easy to deal with. However, the type two and three uh guys are again more interesting. Um it's impossible to kind of block them via IP. We actually started doing that when this wave of of scrapers hit uh about a year ago, I think it was. uh we thought, "Oh, well, it's just some kind of uh group or or something like that and we'll just block IPs when we know them they're abusive and hit something." And we [snorts] ended up blocking like millions of IPs and it just became completely untenable, you know, even using IP sets. H >> what was the country we had to block? >> Yes, I was just about to get to that. So, there was a point where we were getting saturated and we we just wanted to kind of get things back online long enough to try and fix things. And it turned out that a lot of the traffic was coming from the country of Brazil. Um, and I have a theory for for why that is uh which I'll talk about in a little bit. But yeah, it's blocking by IP or net blocks just doesn't work. Additionally, a lot of people have blocked. You know, say a request comes in and it's scraping. It's hit 100 things. It's obviously not a human. You want to block it. Uh, a lot of people are like, "Oh, well, to make it easier, you just block its net block. You block its class C network of that IP address." And the problem with that is that many of these guys are coming from residential IP addresses. So, sure, you can block that, but then a whole bunch of random people that maybe want to use Fedora can't actually reach things anymore. Um, block, we did block Brazil briefly and then we removed it because we actually had some contributors who said, "Hey, I can't get to anything. What's going on?" And we're like, "Well, sorry about that." And we were able to put other mitigations in place and then get rid of that block. uh but there's definitely regions that are are uh where this stuff is coming from more than others. uh there are okay so there is a proof of work which I'll I'll talk about briefly uh which we're using Anubis for there are other things that do this um in fact uh you could do a very simple one implement a very simple one yourself easily all you have to do is you get a connection you make the connecting party do something and you give them a cookie and then you just allow everybody who has that cookie passed. Uh these are very easy to implement on that level, a very simple level. Anubis does a whole bunch of other things also, which is kind of why we're using it. Um it does the proof of work. It does it blocks a whole bunch of these botn nets that are identifiable via IP address. It already has those lists available in it. Uh and it has a policy framework. So you can say this kind of request is is fine. let it pass through this kind of request is very suspicious. You should give it a very difficult proofof work problem. Um you know you can adjust the weights and so forth. Uh the way Anubis works is is that same s simple principle. A request comes into it. It evaluates this little policy to see what's going on, if it should reject it outright or allow it or whatever. And then it it pops up the little, you know, uh, anime uh, thing and your browser is given a proofof work challenge. It does that for a certain amount of time. It gets a a signed cookie back to the browser. So in theory, after that, that cookie is still good. It doesn't have to do any more proof of works. You you've proved Yeah, there's there's some issues around that. Um, unfortunately, so one of the things that people see as a problem here is that the the type three scrapers, the the botnet army, uh, doesn't care so much about proof of work, right? It's they're they using captured machines or something like that. They don't care if that request is a little slower or whatnot. However, it does actually provide one other good feature, which is it slows them, right? So, if they could normally make a thousand requests, having to stop and do the proof of work slows them down to where you have much fewer requests that are actually being processed uh on the other end. So, uh Anubis has been great for us. It's it's helped uh a lot with a lot of sites. Uh Anubis is now also adding that other feature there. Uh you can tarpit or maze these things. So if you find something that is a suspected bot, you can like feed it a bunch of bogus generated data to keep it busy. Uh we don't have any of that stuff enabled. I don't think that that's particularly productive um because you're you're having to generate stuff on your end. you're still wasting the bandwidth of talking to them and it just doesn't seem particularly worthwhile unless you were a like a a bot investigator type who needed or wanted uh that level of things. Um, another thing that I've seen a number of sites do now is authentication. Um, I know RPM Fusion had to move their Koji behind a login password and there's a lot of other sites on the net that, you know, start getting scraped, start having all these problems and then they're like, uh, let's just put a username and password over it and oh well. Um I think that's sad because a lot of these things are open source and they would have been open before but now they get put behind this login wall uh just because of this particular problem but we'll talk about that in a moment too. Um so here we have some industry responses to this particular problem. Um so uh Cloudflare uh created a thing called a uh crawl control. I don't know how many of you have heard of this thing. Yeah, it's not very widely known. So their idea is basically that you register with them and then when crawlers crawl your site then they pay Cloudflare and Cloudflare pays you and so there's like a monetary renumeration for the crawling activity. Um, this, as far as I can tell, has gotten almost no traction. Nobody cares about it. And in order for it to work, they would have to have relationships with those particular scrapers. And I think universally, the difficult type of them are not legitimate companies that would have a business relationship with pretty much anyone. So, uh, that that hasn't really taken off in my experience. Uh, there's another one that also has been proposed as a standard this time. um RSL standard. Anybody heard of it? No. So, it's actually written by the same guy who created RSS. Um and it's a very similar idea that you uh have a relationship with the scrapers and there's money that takes place transmitting around. And also, this has gotten no traction as far as I can tell. Nothing supports it. nobody is actually using it and no particular bots are uh paying any attention to it. Um, another option that a couple of companies have taken is just to try and sell your data out cheaply enough to where they don't want to scrape it because they could just buy it from you. Um, and there have been uh a number of the companies that have done this. Uh, I know with like Wikipedia, you can buy Wikipedia, all the content of it in, you know, a pretty easy to digest format. Um, instead of having to scrape their sites. Um, this I don't think has really taken off that much because again, it's very difficult to have a relationship with some un unknown crawler person. uh but also well we'll get get into that with the uh speculation part of things as to why they are doing this. So let's uh talk about the grim uh part of things. Um so caching adding resources things like that uh which we're okay now we're we're serving everything but we're still being scraped pretty much all the time. So what happens if uh they put on a burst of speed and our caching no longer is enough and our you know adding resources is not enough and I don't know what the answer is to that but it's not good and we have a lot more resources than you know a lot of other people and so we're already seeing this with like home you know I have a small little Forgio instance where I put my open source code oh no it's getting you know scraped into the ground. I'll just take it down. You know, it's not and that's sad. I think it's closing the internet a bit more to the world. Um, there's now a bunch of sites, I don't know if you guys have seen this in the news, that are blocking archive.org because uh not because archive.org is scraping them and causing them problems, but because companies do not want the scrapers to scrape archive.org. or and get their data. [laughter] So, uh, a lot of news companies, uh, are doing this now, like they're just basically forbidding archive.org to to archive their news sources, things like that. Um, this is again a consequence of uh just the the reason that that all this is happening. Um where's uh it's the next slide. Okay. So, um many sites are moving behind off. Um so, one of the one of the things we should kind of mention is why why are they doing this? What is what is the point of all this? Um and we don't know. We we can't be sure, but it seems pretty likely that it is being pulled for training data for AI companies. um and they need or want a continuous influx of uh new data to be used for training. So you know they need need to keep uh feeding that training data. Um let's go ahead and go on to the next one. So there there is some hope uh I think and we'll do some speculation here. Um so there have been some very large botnet takedowns. So far in 2026, there have been at least three uh botn nets taken down that have been over like 10 10 million nodes, something like that. Um and usually these are taken down by concerted efforts of large players like uh you know FBI and uh Microsoft and those sort of folks and they get control of the the botnet's command and control method and tell it to shut itself down and then like arrest the people who were running it. Um, I think that case of Brazil where we were talking about earlier, um, I looked a bit into that to see if I could figure out what was going on. And this is again speculation, but there's a lot of VPN providers out there. Have you guys seen ads for, you know, run a VPN in your browser, you know, $19 for the rest of your life? Um, if you look at the fine print on some of those, they say, you know, you can use this VPN and you can browse other places, but we reserve the right to use your internet connection as we want in the background. So, I have a feeling that there was a very popular VPN network in Brazil that a bunch of people bought that was being used for the purposes of scraping other sites. Um, and so that's why it was coming from Brazil. Uh, I think I even found the VPN provider that was doing it. So um yeah uh a lot of these things are IoT machines. A lot of these things are like users desktops and they have no clue and it's yeah uh but anyway they have brought they have taken down some of these guys. So that's good. Uh what if the scrapers become smarter? What if they started paying attention to robots text? What if they you know they're scraping all this data over and over and over again. there's no reason for them to scrape the data 10 million times. Um, they could also be smarter about, oh, I'm scraping this. Oh, it's a git repo. Why don't I just get clone it? And, you know, um, so I think there may be some of this, it'll depend on what the appetite for new data is. Um, but you know, they've scraped us completely, right? They've scraped everything that we have open. So, why would they scrape us again? You know, again, it's just poor poor coding and there's no incentive for them to to do this. But if their botnetss get constrained and perhaps they want more efficiency out of what they're doing, they'll they'll start doing things like this. A lot of sites also provide, you know, downloads that have all the content like here's all of our spec files in this tarball. So, you know, things like that could be easy for them to actually automate. um speculation. This is a long shot especially uh for if any of you are from the US but um there could be legislation for this. There could be things where uh if you scrape sites uh you have to do it in a certain way or it's illegal to scrape sites in a certain other way or it's illegal to use data for training that is from an unscrupulous site that doesn't have providence as to how they fetched it or things like that. This is a long shot, but I thought I'd add it up there. Uh, and then of course the last fun one there. What happens if the bubble bursts and there stops being a market for the training data? Well, that could happen also. Um, or it could be that, you know, there's less uh less use for that after a while because everyone's moving to smaller models that are training on like specific data instead of like the internet. Um, so it's possible. And that is it. And it took longer than I thought, [laughter] but questions. Anyone here? Well, over here. Um I I would love a question like if you've considered um the influence of like user agents I mean the AI agents people deploy these days on their laptops and whatnot on this because >> it seems to me like this is like a new source of these requests which is coming from all over the place because yes they are all over the place and they are disorganized and they don't respect the robots txt or anything else because their users haven't asked them to. They just asked them like, "Hey, um, get me information about like Fedora packages >> and the helpful agent, what does it do?" Well, I mean, it looks up Fedora. It reads first like um 10 lines of whatever because it doesn't fill its context. Then asks for more and progressively goes through, you know, repos and finally after like 20 minutes and 10,000 tokens responds, yes, there is a package. [laughter] >> Yeah. No, we have thought about that. It seems like the the patterns of access aren't like that. It's not like fetching this and then fetching kind of related things. It's more like you know a toz going down a list type of thing. So I think most of those requests from users are kind of lost in the noise of regular users who go to the site. But it's again it's speculation. It's hard to know what what they're doing. Uh so you said we could use legislation but there is already the computer food abuse act and since your computer is down due to set scraper uh we work for the same company so I likely have the answer. Did you try to discuss that with your legal department already or uh >> I haven't um that's a good suggestion that's certainly worth an avenue. Yeah. >> So, assuming that the scrapers are going to remain stupid for a while, can we get a magic header that we all agree on? Like I met Kevin at vlog header and when I add it to my browser, I will bypass Anubis all the time. >> Um, that's a good suggestion. We could look at that. um Anubis if you are if you pass it a content uh type and a user agent of any kind it actually almost always will let you through but I see what you mean like as using your browser you want to be able to to bypass yeah I mean Anubis can block or can it has a pretty good policy language so you can say if this user agent is this or you know whatever so We could we could potentially do that. I don't think they would ever notice that it existed cuz they don't >> put it on the internet. >> Yeah. Yeah. >> Or or perhaps related to that um like have some sort of cookie uh check for a cookie that you've logged into for Fedora account system or something. >> Yeah. or that you have like a package or a role just so like you can't just create an account that just scrapes everything anyway. >> But >> they don't go to the trouble. >> Well, they might. >> Yeah. >> Um but yeah, have you considered that maybe some people quote just want to see the world burn and [clears throat] enjoy wasting resources? Okay. >> Um, yeah. I mean, uh, yeah. I I suppose we we don't know. I mean, it could be. It could be. >> Isn't there a way to kind of rate limit anyone with some kind of exponential, you know? >> Yeah. So that's kind of uh one of the things we tried early on and other people probably have tried too. But again, you run into the IP problem with those those later types. So uh they have say 10 million IP addresses and it's scraping your site at 5,000 requests a second, but almost none of those are coming from the same IP. So like one IP will only hit you you know once an hour or something like that whereas >> yeah so so that because of the distributed denial of service type of thing that that becomes difficult um >> uh okay so we at AWS are also one of your scrapers so we basically scrape whatever in Fedora in our own infrastructure uh to build packages. So previously we were hitting the Anobis system while uh downloading tarbles from Fedora but somehow it was fixed by adding some magic headers and we don't know why it works but we shamelessly copied it from >> yeah uh I think we mentioned this earlier if you're if you're getting stuff and you pass an accept header like what things you want to accept and a user agent of any kind that you know not null >> um it will almost certainly let you through. So it might be those headers that got changed somehow. >> Yeah. >> And if you do not have those headers, Anubis thinks surely this is a bot because that you should have those headers for like a browser. >> So how can downstream distro scrap you the right way? >> That that also is a really good question. Um, I think we try if if it's coming from a single IP address and it's being well behaved, we we don't have any problem with that. I think if if it needed to be more distributed or it was a higher rate, contact us and just, you know, get approval or at least acknowledgement that that's what you're going to do. But I the like legitimate scraping stuff is just so much less problem than the distributed stuff. So we don't even notice it usually. >> Okay. Yeah. I have a very related question. Uh like we are doing some AI assisted automation and we've uh bumped into this issue. So I guess like the answer is similar like try to work with you like we are uh hitting similar issues with other like upstreams as well but uh so you just suggest to using the headers and uh if that doesn't help probably contacting you directly and trying to work this out. Yeah. And we can we have worked with I don't know half a dozen folks that are doing particular things in our infrastructure and and they pass a particular user agent that we allow uh or some other characteristic of their traffic. Um so yeah. >> Oh hi Kevin. Oh we well you know that we have had our fair share of troubles with how anubis misbehaves let alone letting scapers misbehaves. That's a different part of the problem. Um, is there a way with which Anubis can know that the request is coming from inside the network? You know, instead of us having to do this monkey business of changing headers every now and then because right now it's the application/JSON, tomorrow it might be something else, right? And especially for an application that we maintain for your migrator, you know, the one that takes things from pure, right? Um I think that there should be a better way of doing this because we upstreamed it. They are not happy but hey it works. >> Yeah there there may be a better way to do that. Uh you can Anubis has a really rich language. You can do IPs and networks and so forth. Um sometimes it's easier to do user agent because just because >> it gets things done. >> Right. Right. But yeah, I mean I think they're they're probably the Anubis upstream folks I think haven't written like a guide to how to do this because they don't want to write a guide to how to do this so that >> you know people will misuse it. Um >> no worries. We can scrape your uh documentation. >> Yeah, I'm sure. I mean Anubis is really popular. It's being used all over the place. So I I have a feeling they're watching it. Any else? If nobody has a serious one, I can do I wanted to tell the story of the AI scraper system that wasn't. Um, so there was a time when Kevin was on PTO and the rest of us were all terrified. >> I do take PTO from time to time. >> Yeah. And I was, you know, monitoring the network quite late at night one night and I thought, oh god, there's this huge flood of requests coming into Koji and we have Koji behind Anubis. has somebody figured out how to scrape through Anubis. This is a huge problem and Kevin isn't here. We're doomed. So, I started looking at it and it was weird and I found all the requests were coming from specific internal IP addresses. And then for people who don't know, I'm the open QA guy. And then I realized the IP addresses were the Open QA servers. And then I realized I had DOS myself with stupid Koji thing [laughter] producing hundreds of thousands of requests. So yeah, I did it to myself. I'm sorry. [laughter] >> Yeah, I I want to share one fun story as well. uh that's uh uh we are maintaining copper and it was when uh botn nets fall in love with the copper uh discovered it and I said okay so uh our copper front end uh runs on four CPUs 8 gig RAMs and it's fine it serves all your community pretty fine with these resources and say let's not fine with B let's feed them in the throat as fast as we can and double the resources. It lasts few minutes. So double the resources. We are sponsored by AWS thankfully for that and as I said like let's see who is bigger with the botnet or AWS. So we were double the resources every often. uh and I think we end up with 128 CPUs, 1 TBTE of uh RAM and it was not enough. It last like few hours before the botn nets destroy uh the computer again. So so then we deploy the Anubis and we roll back to four four CPUs and a GB RAM. So that's [laughter] Um do you remember a couple of years back I was working on this thing called uh Fedora contributor activity um service you know and I was hammering on data grea like there was no tomorrow and I was using GitHub actions for it because why not um and you know they they got a faster speed so lesser latencies but at some point in time um you came to me and you were like Um well this this is definitely not him because he's in India and this this IP address is of us right and well that happened and well at that point in time I I think you know quering data grapper is something that is going to happen pretty soon given just how our focus has become in finding community health metrics. So we better be careful from here on out right? >> Yep. Absolutely. No more questions. Yes. [laughter] >> All right. Thank you guys. Thank you.