Vaultaire: a data vault for system metrics, backed onto Ceph

so most of us have been living the dream staring and graphs looking like this for the better part of the last 15 years and one of the problems with systems metrics is presented by our Rd is that they're kind of it's taking up an awful lot of screen real estate to not tell you very much this is you know a scale or number eight servers in this cluster but the you know there's there's all this you know logic to work out averages and transitions over time one of the things that goes into the core of the way the argue was designed was that once upon a time disk space was expensive or released a very limited resource and so we couldn't just record all this system data ever and keep around forever because we would exhaust our disks so what was built into the system was something that would compact over time that you would set a policy it would retain data at a high resolution for an hour and then it would average it down and compact it so that it we then maintained a five-minute type average for a week and then compacted again at a month and year and so on with the result that you had a constant size database you could at the moment you initialize the database know exactly how big that file on disk would ever be and thus managers were saying this was a good thing for system is once upon time the trouble is that it's a little malossi side we had an event where apparently we had 7.8 servers in the cluster and and that's obviously a result of averaging it's moving right and it's a side effect of of this dis kind of storage mechanism the so the question is can we do something about that I would like to build a system that doesn't that keeps the data because the challenge that we have is if we want to do analytics of any form on data in the present using proper mathematical mathematical models you need to give those models all the points you can't pre smooth them you can't average them you need to give the raw data to the people doing the math because it's an analytics decision whether or not to throw away data and it shouldn't be won in the storage system so we need to store all the data at raw at its source resolution we want to give that data in its original form not derivatives this is something that it's kind of clued into this ISM in community a few years back Jimmy Wilkinson speaks very eloquently about this but we don't want to calculate rates it's very tempting to store first derivatives in your systems database but you end up doing it wrong basically whose idea of time are you using what you know how many points went into a system if you're talking about events and you're sampling that to once per second or something like that you're throwing a huge amounts of data potentially especially when we're looking for anomalies and it's the little transitions there actually end up being really significant so we want to store a raw data points not first derivatives and the biggest thing of all is that I don't want to worry about this space on the one hand no it's not our deal and anymore and all hail to Tobey but you know we can expand accept it at you we now deal with problems like customers with hundred gigabyte MySQL databases in a single table and wondering why things don't work or 27 terabyte manifest systems we won't talk with that the the thing is is that I want a system where I don't have to worry about whether the file system is getting too big or how am I going to grow this MySQL database or Postgres database I'd like something that would just scale horizontally without me needing to worry about it and you know it would be nice if it was fault tolerant you know I mean maybe I'm asking for too much here but so I turned to Seth at anchor we had an SF cluster we've been hearing about stuff for some year from monitors from sage here at the conference and just even before I joined the firm they had already set up as an experimental stuff cluster I'm like hey this is terrific I'm pretty enthusiastic what's up it's good to see that you guys are - let's use it the those of you know many of you be familiar with stuff but some some aren't if you have worked with or you can just learn to its f chances are you've worked with one of these three things ratos GW which is s3 or swift compatible front-end because you HTTP access to objects the we're the place where cefalu attraction was RBD a block device that's backed on to on to SEF and this power is virtual machines in quite a number of organizations and it's it's fantastic because it means that the virtual that the water the walk image is behind a VM don't have to be co-located as local storage with machine so it means that the whole problem of local storage and worrying about how to get that local storage to another machine when trying to do migration is just evaporates because it's all effectively networked back storage which gives pretty good management efficiencies that's actually all secondary to the real point that stuff was created which was that the original goal was to create a better clustered file system and it's called sefa fess and really all the work that has been done in the last decade since they first started was trying to put in place a really solid reliable object store to be the backing store behind this file system and that'll come up again a bit later what most people don't tend to see is that all of this is built on top of a library called liberate O's and this is a client code that talks across the network to the storage cluster to get things to happen and for a while now we've been wondering you know can you actually program against liberate O's directly it was sort of suggested by people talking on behalf of the Ceph project that yes you can use the API directly but what does that actually involve was that practical or was that just sort of a nice idea that wouldn't really ever come ever come to say anything ray dos is this it's a reliable autonomic distributed object store which is one of those names that at first glance looks really ugly turned it to mean something but in the end you don't care because an object is its own best description so here's the picture that has been seen ad nauseam and in Ceph talks but again is for anybody doesn't know what store is you've got lots of machines a few machines that are designated as monitors to enforce quorum and otherwise just a large number of machines each of which have spinning disks one of the things it's a little different about an object store in this model is that every single disk spinning disk has a process in front of it called an object storage daemon and that they work independently so the fact that to overseas might be in the same chassis is kind of irrelevant they don't have anything to do with each other their individual processes so you managed you don't build raid arrays on machines you just manage individual disks which was a bit unusual after all those years we spent building raid arrays the this isn't meant to be a talked about but self but I just want to describe one more thing about it which is that it has a very elegant placement algorithm which works out that you specify as a policy where things go the point is to articulate the failure domains so if you have data that needs to be stored you want it to be let's say you have a data center where you've got rows and the rows are quite expensive to do in your connectome a very limited bandwidth between the rows but between inside the rows inter inter rack you've got good connectivity and you want to have a setup where let's say the power domains are that you expected maybe you'd lose power to one rack but not another I mean that's you know not necessarily what your data center would be but it's a possible failure to set up and so you would articulate a question which says that I want to pick from any one row to pick a row and then find me three racks with one server per rack and one disk each of those servers and that way you've got if you want three copies of the data and that way you've got that data spread across different failure domains and this is the kind of thing that crush is really good at now that kind of tree layout isn't terribly exciting but the really novel part I think the part that made Ceph something that we all set up and take took note of was that this calculation is done by clients the bottleneck that was emerging circa 2005 wasn't the ability to have object demons and wasn't the ability to put lots and lots of things on them it was to locate where an object lives what machine do I talk to what disk do I talk to to get to a piece of piece of data and the original solution that was what you have index servers that would know where all the objects are became a huge bald neck it's somewhere between five hundred terabytes and petabytes and everybody was just crumbling as they tried to scale past that point SEF solution to this is to it's a pseudo-random number generator and you feed this the policy map into it and it comes out with 742 and 129 and the thing is is that you the client can make that calculation as can all the servers so when 42 dies they just crank the calculation and discover that 791 is the fourth one in that set and they now know where the third copy should be and the thing is it's not just the clients that know that all the individual demons know this too so the moment that this cluster state changes every participate in the cluster can think about oh wait a minute I've got objects I miss the tertiary for this for this particular object I now need to make sure that that one over there knows that it's now responsible for the being a tertiary copy because I've now effectively the secondary and the former second and the secondary was gone in the foreign primary yeah so that they work as if in a peer-to-peer fashion to work out to restore the level of redundancy and balanced over a large cluster this has you know some pretty nice properties so that's that's all ratos cluster so the purpose is pretty basic we gather metrics we put him in self and then later on people pull him out they do analytics to do work and then we get pretty graphs right good theory the real question is actually that I would turn to it was the final point that I'd mentioned which was how do you build a fault-tolerant system well well fault-tolerant distributed they're pretty much synonyms at this point how do you build a distributed system that's fault tolerant that they can handle something like this and it turns out that there are two properties of our data that make it a little different than traditional database because I know there's some there are some database discovers in the room and some of them are rolling their eyes already what are you talking about building another database like ours and there's not even a shortage of time-series databases I think we have a couple insights to offer one is that if you choose you can treat your data points as immutable now we're talking about system primarily the data we're gathering here systems metrics you gather the data point and that's it it's never going to change it has identity you're like but where it came from it has a time stamp associated you and a value and that's it and there will never ever be an update on that data point which was a pretty big insight because almost every database structure out there is concerned with updates and dealing with updating indexes and all the side effects that happen when you have to make changes like that we found ourselves the data set that is immutable that has the correlates very nicely with another property which is that if you want to build a system that is tolerant against failure it really helps if your operations can be idempotent because that gives you the ability to not have your credit card be charged three times right I mean this is the the core problem of the web is do not click this button twice and you know it's just it's kind of like the only thing you need to worry about because we get that right pretty much everything else falls out of it in in our case the the question was well I would like a system where I can have multiple you know you want a system where you at least one copy is written can you articulate a design where it doesn't matter if more than one copy is written the way that we did that was and oh if the final thing is do not carry any state in the system because after all there is no state except that which we make and that leads me to Voltaire which is the system we've been building the properties that they the the weights design is that we have data points being collected that the I'm not trying to enforce the universal idea of time the time is whatever time we get sent in it's just a timestamp the identity is also assigned by wherever the day is coming from Voltaire yourself doesn't care it just namespaces it and hashes it and sends it on its way on today show you a bit more about that in a moment the idempotency property this was actually I thought you know very clever and as all courts always you know I've learned enough of mountain to be you know weary whenever I have a clever is probably gonna have consequences but nevertheless it was a good idea the time and remains to be seen whether it still is in the long haul but the clever idea was rather than trying to work out that I have a point written a duplicate on write work it out on read when you're reading a whole bunch of points are gonna pull them into some kind of an or a hashmap or in order you know ordered ordered list if you've got an ordering property and timestamps are a pretty good one then it doesn't matter if I read the same point five times as I'm scanning down a block of data I can just sort that in to deduplicate by putting it into a into a set not for this case and the and while you're at it if it's a sorted set backing a sort of you know backing a map well then you get your ordering property because sooner or later this is if you need to order the data anyway but on the way and I don't need to have it ordered so suddenly I've got two things here one is it doesn't matter what order points show up in and it doesn't matter what I write them twice so that was the original design you know and I apologize this blurry but you know I sort of like fun to look back I'll be simple we'll just have points and if you think saying there'll be some kind of a bus and we'll have multiple ingestion demons absorbing stuff and then writing down to stuff which will end up in buckets and then things will come out at the other side you know that took an afternoon right and again very pleased with ourselves and that's what it looks like in production it's a lot of pieces every single one of those boxes is a different server and what sort of bring me to the point that this isn't a small system that you can go install your laptop tomorrow and I actually feel kind of bad about that but it's a large system it has the properties we're after the point that I'm talking about in terms of item potency and immutability immutability this is a characteristic of the clients generating the data data and potency property comes in because the writer demons and label the ingestion here they just take stuff that comes in whenever it comes in if there's a connection loss if a writer deem it fails if there's a network dropout if the brokers collapse the clients retransmit I mean this is your vendetta bleed everything industry systems comes down to retransmit time and you can be a long time about 10 minutes later you can get returns and it doesn't matter and patterns like well what happens if a point was written down but the acknowledgment was lost right the two journals problem well in our system we can get around that because we don't care if it's written twice and or eight times or mmm any more than that one I had real bugs so because I mean you can imagine that things go really wrong here like LAN I'll just sort of put a little teaser out there let's say that your timeout somewhere in the middle of the system is maybe 60 seconds well surely points to get written down a couple seconds if you acknowledge back right and in testing that's exactly what it looked like think about that for a minute the other thing we did is early on and I thought this was clever too and it was utterly wrong about was that we use proto bust both to encode on the transport and is the serialization format down to the wire Pro buffs are a nice way to articulate data they give you a slight scheme on top of which although in a binary encoding they if you've read if you work at a place like Google where everything is encoded in protobufs that's lovely it's I anecdotally it's used as the interface format for virtually everything there it is sometimes uses a storage format I was hoping that this would be a good way to make have different clients and different consumers agnostic about you know I was working in language that she'll be not named yet but I didn't want any of other people have to worry about that and so protobuf seemed like a good way to create that abstraction the trouble comes that a whole bunch of these fields are variable lengths and it's surprising how expensive that becomes and all the and there's two things going on here in the original design if anybody has used collecti for example you'd know that every single data point gets sent along with host slash CPU slash CPU zero and then a value you get labels like that and this can be generalized to field value pairs it could be the data center came out of and maybe something about a customer ID anything like that but you end up with these key value pairs which identify a source and although this was all very tightly what are you worrying about you know how many bytes and should be using four numerix verses or two more general types and and the the general encoding and proto bus for integers is variable length and it uses like this many bits to say whether or not it's longer and then this many bytes bits of storage and it rolls long so a small integer can be encoded three bytes whereas a larger 64-bit integer one I'm taking seven right and so the sweet spot is that you know the point is that if it's new each since you can you ship around small numbers there's no reason to pay for bytes every single bloody time you want to or eight bytes for word size to every time you want to encode to the number 12 right and so on average that work that pays off pretty well in terms of over the wire efficiency the trouble is is that frankly at this day at this point the size over the wire isn't the problem so I spend all sorts of time worrying about getting the variable encoding right and implementing all this to optimize for something that really isn't it certainly currently I maybe this will change again in a while but in a datacenter on a LAN between machines you know over the wire bandwidth is not the bottleneck that's the other thing you see here is that these field in order to articulate what is just a map of key value pairs you know it's pretty easily constructed you can see that but you're of course going to have to serialize this in and out so what we were doing was and this is the part that I thought was get would allow me to get away with creating a database without having to pay the ten a decade or two of building a full database management system was that the whole idea was to punt everything to SEF I said that that there was no state that all the pieces in the system other than let's go back to this slide everything in that system other than Insaf carries no state at all there's no statement brokers there's the only state in the clients is whether or not the pipe point has been acknowledged not so they maintain points that are in flight but everything else it gets down to SEF and that's it and so we encoded the location of points at terminus tically there's a generation tag and then the FM to xr-7 part when the name state in origin is just a digit hash of what turn emails I don't like data center an IP address the collector or something it's just a just a namespace and then a hash of the source and so what I did there was serialize out that that map so all the key value pairs which uniquely identify this host knit you know hostname 71 CPU /c p u0 and you just put that down looked into a hash function and come out with that and is the deterministic address and again and that's sort of the final driver of this was determinism the whole thing can be calculated you don't I don't need to maintain a whole index of what the address addresses were because every participant system can figure it out so and then finally a timestamp to the nearest metric day I made that up ten thousand hundred thousand seconds you know why not and and that just seems a reasonable system I mean trying to do you like work out what day it is was silly just rounding it to the nares hundred thousand was easy and that gave me names to then pass to seth and the crush hash function and and the placement group algorithms would then work out where it goes and into the system it is it goes and it's reliable it's duplicated and mirrored and that's it so that gave us a system whereby we could write arbitrary amounts of data to to SEF and pull it up again the writers didn't need to coordinate with the readers and I mentioned before the business about all that business about deduplication and not having to worry about it that with the magic that made that work is append yet idempotency if you if you're a pen safe and there's an append operation and liberate us and so once we work out the name of the bucket where it goes we just take the data point serialize it two bytes and append and if you append the same thing six times Edward doesn't matter because we sort out ordering and deduplication on read that's a good design it had some problems the burned a few things here on the first thing we burned to the ground was this after the second thing we burned to the ground were the server's writing to it South mates a few promises or not promises it just you asks you know sage would would get up in front of us and tell us that you can have an unlimited number of objects and that is more or less the case I mean unlimited is a big number but you can have the number of the knit and the namespace of objects isn't the limit so the scheme we had where we would have you know all these different objects labeled by name by namespace and hashed address and then over time partly that up wasn't a problem the size of objects they also tell us that that's unlimited and well yeah that's true they really probably should roll that back because there's this the trouble is is that are both registry WN RBD stripe their data over four megabyte chunks so when you write an object array dos GW it actually breaks it off or whatever hash scheme and then and then writes them down in order same thing with a block device it's there for mag blocks and that makes perfect sense I might well be some size and pick one the trouble is is that down in the guts of Ceph object demons they eventually end up being single threaded the concurrent system comes from its sheer mass it's it's it's lazy parallelism there are a lot of places within the system that is nothing of the sort and which is fair enough the trouble is is that one lesson is G is responding to a request it's kind of busy doing that and only that and so there are there are cases and it's it's not so much server Wes as is doing replication that there's certain operations and to be honest every time we thought we'd pin this one down we kind of figured that we were half right and found in it so getting an authoritative answer on any of this is actually a little bit tough but our impression is is there are operations that happen one at a time and so if you go and dump a 200 gigabyte object into a sect to Hungary by filing to any staffing a single object it'll take it because you know it's it's you know that it doesn't it will absorb that rights as long as there's this space out there but when it comes time to replicate that object or read it back things will kind of grind to a halt while serving that request and only that request and so it turns out to be a really good I like that we the experiments we did and this was up to the point where we became Network beyond but we found that the sweet spot was between about 400 K and 20 mix the the bottom of the curve was indeed at four Meg's so it's not a hard-coded limitation but it certainly is in the DNA of the Ceph code that form eggs is a good size for objects so if you're writing something use that size and then of course the final thing is and nobody actually promised this but I sort of have made an assumption which is pretty stupid and that was that I could just sort of blast out asynchronous there's asynchronous operations and you fire it out and they're in flight and eventually you get an acknowledgment back via a completion mechanism it's a primitive little your futures thing and that seemed all very coherent and we read through the code and all seem pretty sound and it's not that complicated it's just a straight it's actually the synchronous mechanism uses the async mechanism under the hood it's it's fine and well no and it turns out this makes sense because when you've got a block device let's say on top of SEF and you you write - I object over here I'll block over here you dirty a block and it gets flushed and so this one lights up and it's two friends get you don't have to synchronize what and then you know I'm standard operations the patterns you're over here and then you write do another block and it gets it that OSD takes the right it talks to its two peers if your application level three and they get back and they get back to you that's fine this is happening in random sarcastically over the cluster no problem the trouble in if you got a large read suddenly across a logical the continuous array then you know you might light up 12 of them at once but again it crossed a large cluster with hundreds of OS Jesus isn't a problem the trouble was is that we were finally the number objects about quarter million when we started out that represents a quarter million data points we were taking in per / / check cycle and there are sizes were fine as like I said sometimes we overdid it but in general all most of our points weren't even triggering me I'm getting past a couple hundred K in data we had allowed for some some larger higher rate ones and that would have had it's around 4 Meg's and so the trouble was is that we found that it was taking you know upwards of you know well this is large error bars on this for three and a half to five minutes per right cycle to get those quarter million points out that translates to about 2,500 operations per second it's it's ops per second that is the limitation in a theft Koster not object size not a number of objects but the rate at which you can manipulate them and it would appear the scales linearly with the number of voice T's we don't have hard numbers on that we were kind of busy you know desperately trying to sort this out it would be interesting experiment to do but you did one of those things it doesn't show up in development it's all working great but when we were launching this many points at it it would take three and a half minutes to complete now there were two problems I mentioned before about time outs well guess what happens when you've got 60 second time edits on the clients that are you know waiting for that indulging packet to come back they're like oh you must have lost my right I'll send it to you again so we got up into the tens and even maybe close to hundreds of millions right spending and it drained once we figured that out and stopped you know stop pre sending everything we were able to drain it and in a better course of a day Seth absorbed it we end up with some pretty large buckets but we were able to do it the thing was this was absolutely maxing out the cluster because I was sort of you know waving my hand resting a couple year couple year we were writing 200 s DS simultaneously now the reality is is that no matter how a distributed system is once we want to achieve concurrent consistency sooner or later you have to linearize it's a basic systems basic property of distributed systems and so underneath the hood sooner or later one ôs DS can need to talk to another OSD to establish it it's got its copy of the object it's trying to do a write on and get acknowledgment off and that other irty is busy and so you wait and so the whole thing just kind of ground halt and averaged out at the pace of around about 2,500 second right and it was not a small cluster by the way it was on the order of seven machines with on the order of 60 or 80 discs in it at the time so you know not huge but not small but the point was not to maximize the cluster I mean this was this is an internal system this is overhead I can't have this system Domino wrecking an entire production object store I need that storage for other things we had a couple of VMs back onto it as well with our BDA needs to say I was doing very bad things to the performance of those those VMs so that had to stop the other thing was happening oh dear I'm so that's v1 right copy and paste here so yes I was saying we had 2500 operations a second the other problem was the Machine writing we had to get a really big one and we were we had a 16 core machine with 32 gigs of ram in it and we were flattening it I mean I've never seen that many bars in H top over to the side cranked up I've never seen 1600 per CPU usage either and it was regularly running out of memory like a this thing was ooming and like then filling swap and moving it and gone and which of course was another reason I was losing all these rights in flight there's a right cycle I would lose a whole cycle because it was just yeah and what was going on it was it turned out that so we were working in high school we are working in high school and we were using the the standard libraries map anybody who's worked in a you know one if the languages that was where our roots are immutable language like like see we're used to tree structures being mutable and and trees tree base map structures are fantastically easy you know easy to implement or clean the implement when you can manipulate and change pointers Haskell is a language which uses immutable data when you create when you make a change to an object it creates a new one it's copy-on-write thing and that has certain properties and and you can create this by the way in Java you know I've built persistent it's called a persistent structure when you have a previous version and the next version of a strut of a data structure yeah created maybe the change you've got both now that you'd think that would be wasteful of a memory it's not because you can amortize space over time is you're only changing the nodes up to the route that are the structure that's changed which means it's actually quite cheap to store both trees and the reason that's worth doing is that when you've got a concurrent system and lots lots of threads it means that any given thread can always have a consistent correct not in progress version of that tree to work with or data structure in general which is fantastic which means that the all the problems you have about having to lock data structures in as traditional mutable program just go away I have a copy of tree you have a copy of the tree and like I can we can both iterate over them and no problem which is amazing and gives you you know if you build a system that you only point you have synchronizes the handover of the tree group between between threads so it's a phenomenal thread safety advantage and it's just sort of given in a functional programming world the problem is is that remember those field value pairs that were a map that identified a source well I each value was on the order of a hundred 50 bytes with 100 100 of which 120 of which was the this map and so I was bringing in those key value pairs building a map of that in order to which was by weight sterilized on disk in the protobuf bring it up into memory into a map in order to order it serializing and that would again in alpha or just cuz I had to make sure that through then hash encode it to then trim it down to become that address it was label all very deterministic sure but I was building up these maps and then throwing them away in fact I was you know taking something serialize building an app and then serializing EV again which was just you know I kind of silly in hindsight but it wasn't the effort of doing a building building on structure such as the problems the garbage collection we were generating tons and tons of garbage and it was took it we had to do quite a bit of performance tuning in hooray for perf top but I I was worried because it was it was seemingly correlated with the right cyclone what's going on and I was really worried the crush map calculation was was what was burning me to the ground crush wasn't even taking 1% of the CPU as advertised it has all the GC so it there were a couple ways we could have worked around that it turns out there are mutable data structures in haskell ask allows you to easily drop in a mutable state you just need to isolate it so that you can maintain the properties of the language elsewhere and we couldn't fix it that way we could've tuned the garbage collector but that really wasn't the way ahead so yes so it's but the two things so one was it like this the the portable structure itself is fine but you're gonna have to bring it into your language to do something with it and virtually any language and the modern age is going to require you to do you know to convert it out of its native format that the guys who have made something called captain proto which is floating around it's actually very clever they've created a serialization format which doesn't have the variable like thing going on and is laid out any memory exactly like a structure now how you actually get structure alignment to line up with a machine you have I think that's a pretty big assumption but nevertheless the basic idea is good if you can if you're a C programmer and you've got a structure off the wire you can just drop into memory and then access as a structure that's pretty clever there's no manipulation required and a higher-level language Python and Perl on up you're going to need to turned into that languages terms which means taking the original source serialize structure and then reformatting it in internal terms and that will work that will create all this this this this fluff so there's really no avoiding that and that's what makes what was a cool idea not such a good idea ok so what we're going to do about it the changes we made were first of all deterministic addresses are like predetermined addresses excuse me you can do a deterministic calculation to get an address but I was just like you know we're 64 is pretty big we will give you an address and either you can ask the system for the next one we had an auto increment in there so if you want to guarantee next clean address or which is a an unused address we had a mechanism to go get to do a random number generator check and see if it was there which across to 64 is you know pretty good or you can calculate yourself maintaining yourself do that calculation yourself it at least was outside of the system that was happening so for example our legacy checks didn't have the way to reach back to the core system and so you know they they did that calculation and we exposed unab ility to make that you know that deterministic calculation in the library the change we made was to hash those resultant sources over 128 buckets pretty simple really it's like okay well it turns out that just writing to SEF was fine it was not a bottleneck in fact one of our sort of workarounds temporarily was just to take everything off the wire imma write it to disk right to SEF and that was glorious none of the CPUs twitched SEF didn't even notice we were dumping everything off the wire just into a cash bucket basically into a temporary bucket and it was great the machine that was picking that up and then you're falling it off according to region algorithm was burning but it sort of really Illustrated that we could taste off that wire speed and and get into SEF and and stuff we wouldn't even notice so the data volume wasn't the issue so we sort of took that idea and mutated a bit and say well any for any object source coming in figure out which bucket it goes into and and fire it in the number of buckets is a if a factor in the equation and the other is this size the time that book so it was no longer fixed to a hundred thousand seconds because one of the problems there is what happens if you got a high rate source it'll end up with you know a huge bucket whereas you know low rate sources would be the very sparse so the system adapts so that's why the timestamps here now I think as seconds are really long in hindsight or like this was done by by my colleague Christian we came up with a design on a Friday and I came on Monday you'd already done it on the weekend so I certainly wasn't gonna complain one thing I would change in hindsight is that when you do a radio snot that you want to be doing radio LS very much but if you were to if you're doing Diagnostics and you do a radio cell s on the pool that has these buckets at these objects in it this doesn't order very well it sent orders by bucket by hash slot not by timestamp and you really want to see how many you know where you know see them all together so that would be one thing I'll change in the v3 generation is just put it back so that it does list out in order nicely the other thing we changed was that there are two types of points there are simple points which are simply just words you see for values and extended ones which are arrays of bytes so we saw the ability to take in arbitrary length data because one thing we'd like to store is is web server logs having the ability to correlate systems data with what's going on a web server will be very nice and so we did one the ability of taking arbitrary strings and so that's that's what the objects look like and stuff so still a deterministic lookup like it once you conform this object you go get it and Steph gets it back to you and that's no problem the again not maintaining state the the evolution of the number of buckets in and given epoch and the side so the length of that book is Riyaz maintained in two to seven objects one for simple one for extended and that looks kind of like this and again just very simple serialization format and and the data points themselves the file change was is that we kind of work this out exactly the same time that Yaron Minsky who's the head of engineering at Jane Street capital in New York came down and gave some lectures in Sydney and he was talking about it's not directly relevant here but it's a fascinating insight it's like if you want fast code market high frequency trading fast code you can't allocate you have to use fixed size buffers you fill a buffer you process it you empty it and you refill it no allocation and that's the way to make fast code that's not for me but that's for my one of people that makes the markets work so that was a nice reinforcement of the idea we had which was it seemed to us that the way to get around it seems for a long time we avoided creating file format so we've kind of drummed into us I walked for a while there that we shouldn't create our own file formats wait a minute if you can articulate what the rules for format are and what the offsets are you just use it and so we can't with something very simple it's three words and and we have one for the address one for the timestamp fits in very nicely and the payload that's it so for every 24 bytes that's a data point that's it that's you know could we make that smaller sure but it does mean that we have the ability to we're not even doing much like a pre allocation tricks but it does mean that it's fairly compact and that comes off a wire and we just take it off the wire and write it just once we figure out where it's going and that did reading is then finding the bucket that holds that that source that it's been hashed to and then we just read the whole bucket in and scan down it it's a very fast filter operation and find out the three bytes that have the three words that have there or if you can want to real just running your friend your finger down it pull up the ones to have your source and there you go and then do the deduplication all the times so that's what goes on for the simple data points the extended data points we it comes in mush together but then when it goes out to disk we right out of sync a simple point thing includes the offset in what is just a long a pending file for where the that it is so we say okay it offset 65 300 there's then there's a length prefix and I'm in the data you know your web server long pulling or whatever and you know one of the guys they work for stresses hey can I put kind of what sound do it in there like no and hey it's happening and in fact you know I finally designed something that works I just pulled these numbers up an hour ago this is the current size of our of our production vault there are not even a million objects yet and we're running about one and a half terabytes this represents about ten months of every data metric that anchor systems has collected and for system metrics and it also now includes the last three months all the data we've collected on metering and measuring whatever the second subtree so salaah meters data is feeding in here as well and it's rolling at under 100 operations per second oscillates there are some some factors there that it's galo in but we're running about 77 operations a second and the machines aren't twitching that's 50-minute leverage on the writer and I'm pretty pleased about that it means it's it's a viable system one quick digression but back to Steph because I did say this talk was about writing staff in production there's two classes of low we're putting all that load is going into the object store but of course we're running are the nether nodes the system on on OpenStack on RBD and that in turn is running on a different staff cluster the one we just brought up for our production cloud and if you've never worked with Steph this will be a bit gibberish if you're in old stuff hand you'll recognize this but this is what Seth looks like when it's healthy there's some monitors there are it tells you something about how much disk space there is available and it says that the placement groups the modules except hashes over before sending it up to its individual machines is all happy and clean when a when single OSD is down you get something that looked might look like this it says that well I'm missing one there's only 167 168 are up and most of them are happy but the other ones are happy but they are currently being served by their peers and then it goes in other phases like it evolves from there to suddenly decides after a period of time that no it looks like that machine's not coming back so I'm going to need to worry about cascading the data on that's very clever and that if it's a reboot happening there's no reason to rebalance the whole cluster for that gif for every given object if the disk is coming back and just machine is unavailable for a little while so that's different machine being down and I machine me out and that's just by default is ten minutes but you know you really kind of you can configure that and you can also flag it to not do that if you're doing heavy clothes for maintenance but you see things like this in a fairly ordinary case it comes up with health worn as the indication but really that's probably something that we should change a bit because this is entirely nominal behavior but unfortunately is the same thing when it looks like this and I had this on the wall at work and this is this is like you know so does it Timmy's like I take a picture this yeah this is real this happened we were having some network troubles and what was happening was InfiniBand and XFS were fighting each other for the highest order of allocations and memory manager looks like we were one of the first people to hit this one but since we've heard other people squealing in pain so at least it's not just us and we've worked out some workarounds and and but the problem was ultimately that a InfiniBand would start dropping packets because XS past was trying to store to make extended attributes because anyway and so packets were getting lost and the other machine would assume that since it wasn't getting reports of health checks or a package loss it was like oh that machine must be down and so we start telling all of its friends that that machine was down even though it wasn't copes with that but this was happening on mass but then it started happening to the monitors too and so the monitors would go away which was triggering and monitored elections so you shouldn't really get an unmonitored election more than like once a month and that's just cuz it's bored we were getting a monitor election every couple minutes and needless to say the whole thing came to a screaming halt I've you know never want to see recovery Rio looking like that ever again but you know what I'll tell you something every single VM was wedged like you're sitting there pressing enter on your shells and why is nothing happening you know everything was stopped which permit available the stand point represents loss yeah we mean the other thing was down we were down in terms of any external consumer but all of the m's were safe they were just blocked SEF is a consistency store non availability one right cap theorem see AP most systems nowadays are built as a peace tourists you have lots of copies you write to it you write an update and eventually one will overwrite the other if it's last writer wins or maybe if you're really smart you build to see rdt it's the topic for another day I mean if you're not really smart you'll have a shopping cart like Amazon's that just things magically appear and disappear the but dynamo or react Sandra these are all availability stores and indeed s3s as a whole is present Esther he's great if you write to it once and look knowledge that right done just don't update anything in it well and expect to have the you know what consist an answer anytime this week which is why you don't want use s3 for backups the orders to Billa file system on anyway the I told you the beginning that SEF ray toasts the object store the entire system of all these peers talking to each other was built to be the mechanism underneath the file system a distributed file system and the semantics of every file system we've ever built has been the assumption that the the foss system code owns the blocks this is why you don't hook one drive up to two different machines and in order to build this is the hard part about building attributes file system is that you've got multiple machines all trying to hit the same blocks right so the Ceph abstract that the rate of abstraction presents a consistency model using paxos to guard the master map of what's available and then the nodes in themselves or create consistency Seth will block if it can't complete a right Steph will block if it you know it would just sit there and stop the whole thing grinds will halt terrible hundreds of terabytes as you saw there just suddenly unavailable and that's the point and you kind of really have to sort of clench your hands hard go yes this is what I wanted because we're kind of used to languages that yes it compiles and runs it doesn't work and you know it monkey patched it engine sensibility but but it runs dammit you know and you know why are we working in a statically compiled strongly typed language well the you know things things don't compile when when something downstream is broken and then we have a build a swell that's the point I don't want to be able build the software if it's broken and I prefer the compiler telling me that then then find that our users finding out the runtime CP versus AP is the same thing so when we were able to figure out the problems that I was alluding to the previous slide and we hit it you know execute one of our workarounds suddenly what whole by Jenner you know shell prompts go by because nope it's it's happy now it's it's got its read back you know and carries on so the client games that we're running the system never fell over and and likewise similar behavior in the object store we haven't lost we have you know our systems absorbing data have occasionally forgotten to write or fallen over but once Volterra's accepted a point we haven't lost a single one in ten months of running so that's really not down to me so much as it is to stuff be an amazing piece of engineering and if you can tolerate the consistency property um it's somebody's really consider looking at we've been in production for a while as I said the second version improved the write performance problem there is a corner case that we've hit which of course is that you've got you know a bucket with it's very fast for a single read you know you just scan down it and pull that data point out and gather them together and give them back the reason I built this monster was to do analytics on and so the first thing that our data scientist page DS do would want request all the data which way no one squared on me needless to say so a pretty obvious optimization is gathering a bulk request figure out what reads are going into a single bucket and do all those at the same time that's gonna happen now and then we got some further look to you oh the final thing is my buddy could come off with an easier really quick alright the final thing is caching I talked about immutability data being a big insight the other insight into the system that I want to offer is that for us to improve performance is obvious that we can put caching in front of the thing the biggest insight we've had is that there are different types of data so that we don't just want to arbitrary cache there's three signs of data we care about first and foremost I want those bulk reads to not mess up the system but there are there's a live set of data there's basically everything has happened in the last couple hours eight hours or so I need that all in memory because I can build models are evolving over time and looking for patterns and doing alerting and calculations and turns out is only gonna be like a hundred 80 gigabytes I can keep that in RAM no problem the last last eight hours a day does 180 gigs even if it's really slow to that I was with a 10x multiplier for memory layout and so we can keep that a memory no problem so we're gonna create a fast path across the top that doesn't go down a step just any point that comes in and get broadcast across the live set cache and so it makes those points available the second set of data that we care about is a very very limited set of but the the few data points that we do care about for billing right if a customer wants to look at their bill they want to look at their current complete bill which is last month they probably wanna compare it to the month before that fine and we need to be accumulating the current set so that we can tell people what their usage is in real time and that's all I need to carry care about I just need to make sure that those are up I can't have a slow query being blocking on the data Bowl so that's the second specific type of data set will have cache and end and then third one is to take those the remainder of the traffic is bulk reads cache it because sometimes people ask for the same query multiple times so there's no reason we can't accelerate that but just basically pass that through and keep it other way no you know I'm not gonna give follow leanest rule on this one but it's a lot of pieces if I actually in hindsight I probably should have ruthlessly suppressed the number of rot repositories that this is proliferated out to it basically matches mostly haskell modules but also other languages in there the stuff talking about today is the three at the top they were both faced a lot of people have gone into making this what it is it's I can't say it's an open-source project to the sense that it's a large community yet but down the bottom there Aaron Zimmerman is the first person outside of anchor who's called instead of trying to install it can you help me out so we've got you've got users outside the company which is terrific and a big shout out to Katie whose talk on Machiavelli you might have seen yesterday who's built a visualization system which is a big consumer of this and the entik people outside the company as well who have offered us help on design and the indexes will build on the next generation so the last thing I mentioned that I wanted to build analytics on this and this is just a very quick example of the one of the early outputs of the system in terms of being able to use the knowledge that we've gathered and this is correlating what kind of failure modes happen in HTTP servers with meta system metrics and a decision tree models so that's what I had to say thank you for time the half of LCA please take this skipped Andrew and please thank Andrew good