Micro-Service Architecture, by Fred George

[Music] [Applause] well thank you in fact uh it's always humbling to come to our conference and you listen to the constantine and all these children who are speaking about these things that they're so much smarter than you are and it's too late for me so i'm going to talk to about microservice architectures and to some degree this is a bit of a personal journey but let me be clear do not take this journey it is too painful so i want to sort of share some of my pain and kind of where my head is at now pretty much hoping you will not try to do the same thing i'm an independent consultant originally from the u.s i've worked in a lot of countries i've worked in india i worked in china i've lived in london now for the last six years uh so i kind of go where the fun stuff is so this story kind of starts at the beginning um in the beginning we basically were building systems that i would sort of say resemble stonehenge if you notice we have some nice towers of functionality we have some cross-functional services and please do not tweet i didn't i was not the architect of stonehenge but uh i i do have to say that when paulo was talking earlier yesterday he started talking about 1968 well i'm sorry 1968 is when i started writing code so i was having my life flash before my eyes as he was talking in fact i was actually one of the people in ibm at the time in the early 70s where they experiment on with waterfalls so i was one of the victims of waterfall so this is this is why you see a shell of a person today so my story actually starts not back at stonehenge but really kind of in 2004 and this i was i was working with an application and i was in thoughtworks at the time the application was uh at one about a million lines of j2e code and at the at one point in time it was the largest sun certified jte application in the world it was a leasing system quite complex uh sort of a bellwether application and it has a very very nice history associated with it um it started very very well it was really a project that agile was actually tried out in thought works uh among other things martin fowler and and ward cunningham came and consulted on in fact martin fowler actually joined thoughtworks after working on this uh the book enterprise application patterns that book comes from a lot of the work he did on this application so in some ways it's a very influential application in our industry so it started really well so my part of the part of this application gets in 2004 and what i found in 2004 and now i'd gone to india to work on this because that's where most of the developers were working on at that time what i found in 2004 was a million lines of code about 2 000 tests but 70 percent of the tests accepted test passing was considered victory they were happy and i'm like but that's 30 failing and it was a different 30 every day and they thought that was okay uh there was a bug database without a thousand entries in it it was like uh where are these things it was like oh these are just the bugs i'm like okay uh it was they were still working on it so they're still changing it but not fixing the bugs or not getting the test to run and writing new tests well it was like why bother the old ones don't run anyway so why bother writing new ones now again this is an agile shop and it wasn't exactly what i expected to find but this is what i got started with and just to sort of give an indication i i do like my metrics to some degree i you'll find me as a process minimalist i don't really like processes but but i will dust off metrics where they make sense and one of my key metrics i like keeping track of what's going on in a project is count the new unit tests those have never worked within a project won't realize but i write a lot of code in a project i get in there i write code because frankly there's no secrets of your writing code you can try to manage a project and the programmers will may not tell you the truth but you're a developer in there as well there are no secrets so i count unit tests because i know if we're writing unit tests we are in fact getting more functions more function with more unit tests it's easy to count them and so here's the graph of unit tests across five months now anybody tell you about systems that go up and down like this means you have a highly unstable system so this is a five month timeline and it sees it fascinates from you know sometimes even less than 40 all the way to almost 200 it just seems to vary all over the place week to week now the really scary thing is those numbers at the side so you take an average week of 80 things how many programs do i have with this project well it turns out i had 40 developers this is how many tests they're writing in a week 40 developers will write 80 tests in a week which means on average they're writing to a week which means we're not really writing them in a bad week obviously not doing much in fact i could probably find the three programmers if they were doing something else that week that's why we're down so clearly things were not working well so being curious as a developer about why can't you write tests for this stuff i decided to run an experiment and so basically i wrote myself a test it says i'm going to write this test and the concept of a loan is a fundamental aspect this is a system that does leasing systems and the idea of the loan was a it's a very nice encapsulated object it was you're part of the original design it's really great idea but for some reason they decided that this little number 50 which should be encapsulated you shouldn't be able to get to that number directly people decide yeah i should get to that number and so you wind up finding about 40 different places and 37 different classes that reach through and grab that number which made it really hard to understand what's going on so in my little tests here i i basically said okay it's a very easy unit test i'm going to go in on a saturday so i don't get any of the serbs i'll mother by myself i can do this and five hours later it's not null so clearly we're broken um and this was kind of the thing that oh yes it was a really nice system now what happened what happened to this thing that martin fowler and ward cunningham and all these really brilliant people put together this really gorgeous system and great conceptualizations what happened to it and so um one of the things we realized very quickly and i sort of brought some colleagues in is one million lines of code this application is not one million lines of code in fact one of my colleagues said and very cleverly it's really a hundred thousand lines code trying to get out that you could probably if you really wrote this thing from scratch it's not that big it really wasn't that complicated this is my colleague jeff baywell you'll come back just hear a lot about jeff bay because he keeps making really wonderful observations uh but that was his idea about it and lo and behold i think he was right now again what happened what went on here and to some degree the question is well were they just lazy programmers were these guys in india just being lazy about that but i was in india i was watching these guys they were working hard they're very bright guys it wasn't a matter of being lazy well maybe they were being sloppy about it and you know perhaps a little bit of that but what didn't really seem to be true either uh maybe they were inexperienced and it turned out yes they actually were quite inexperienced these were university hires they'd really not been exposed to agile practices they hadn't been exposed to what good programming looks like and some executive and thoughtworks had decided was a really good idea to send this application over to india where we have programmers and let's just work on it and they destroy it but not intentionally so there's a little bit also going to work is that the indians didn't feel they had ability to say no if they were being told by their their us counterparts that you have to do this they would just do it when in fact noah had been a good answer sometimes now there's a name for all these things that we use a lot in our industry we call it technical debt but i will say that i personally believe that is another way of saying one of these other things so i hear people talk about technical debt i'm really saying what was it about your system what was about your organization that allowed you to create this problem in the first place putting a fancy name on it doesn't change the underlying issue it should not have happened in the first place so we'll come back and talk about that because somebody at some degree we've i've got a system now that makes that go away to a large degree all right so now we start evolving into this so now i'm actually been assigned to china because that's where i was needed at the time and where i've actually gone into a chinese bank and we're consulting with a chinese bank to bring a service-oriented architecture to them now i gotta say at that point in time i didn't know what that meant i was fortunate i was working with a colleague jim weber uh he published his books about this stuff in fact i think he's the chief scientist now for neo4j so i i really chance to work with a very bright guy and the challenge the chinese banks were facing was they have all these banking sort of things they want to do just like everybody else wants to do in banking mortgages and loans and they have bank accounts they want to do all these things and unlike some of the western banks particularly the u.s banks which you can do all these things from your when you're working with a teller they also want to do these things with every other interaction technique they have with the customer they want to be able to go to an atm and get a loan now in western i can tell you the united states that was not possible but in china they said it was absolutely feasible in fact they had their even their smartphones back then were able to do some of these transactions that we in the west were just at that point dreaming about so they really wanted to get to these systems in every possible way and we were brought in basically to to propose an alternative architecture to try to get basically one of our clients uh a foothold in these chinese banks to help them develop software so we basically decided what we really need to build is some sort of pub sub bus that in fact we wanted to sort of take advantage of the fact that if a customer is in front of me i want that channel that has a customer to screen back into the into the cloud i have a customer and i want all these other things that are interested in customers that sort of wake up and say oh if you have a customer well we should probably get him a loan or home sorry he owes us something on the credit card or maybe we want to refinance his mortgage we want these little services to wake up and sort of say oh let's do that now how you say that back to the customer that's back to the channel but the thing that you want to do as an interaction is sort of broadcast by these applications and so it sounded like a really nice structure to put in place and in fact the client rejected it uh they decided they really wanted to go buy a system from citibank in the north north america because citibank is western and therefore it must be better the fact that citibank wrote this system in the 1970s and they wouldn't support your tellers or anything else apparently was lost on them so it was a nice idea but one of the things i did walk away with was in fact going back to jeff bay and his 100 000 lines of code i began to believe there was another way of thinking about this application i was beginning to believe that in fact it's really twenty five twenty five thousand line of code little services trying to get out that if i went back and restructured that application i would probably restructure it differently than what we were talking about before they didn't have to be a stonehenge so that's 2005. uh now i've been been sent back to the states i'm working with the medical systems company uh of course i've got some new ideas here just the fact that they always rejected but i'm sort of got some new ideas i'm always dangerous with new ideas um and we had a system where the the this company was in the medical systems business they made you know x-ray machines and cat scanners and mris and all these things and so they had the idea that we should actually run hospitals because if you run a hospital then they'll buy your equipment so they want to really run the whole hospital and they were basically trying to rewrite their platform they decided that the whole platform was written to c so they're just going to rewrite in java and it's like but that's not going to be different than the c implementation and we're again trying to suggest some alternatives so we suggested this pub sub again it says you know you have all these things that are trying to collect data and you have all these other parties that are interested in the patient is interested in it although a lot of cases you're not allowed to tell him things but certainly you're a doctor you know the staff running the emergency rooms or the or the critical care units and of course accountants they seem to love things about this as well so we propose again sort of a pub sub base that says we want these instruments to publish little nuggets of information about what's going on and we'll sort of route those nuggets to interested parties so if you're getting an mri it's very interesting and and probably your doctor would like to read it and perhaps therefore the critical care unit needs to know what's going on and of course the account loves this because there's you know ching to ching we get more money um and we started looking these little nuggets that you would publish and one of the things we started thinking about is first of all we want to publish conclusions i don't want to publish the raw data because you know i don't want to send just x-ray images over to the patient or certainly not even to the critical care unit i need to send conclusions what did i figure out as a service publish the conclusions and so again if you have a cat scan and it was jane getting in the cat scan then the conclusion reached by the team doing the cat scan may say yes there is concern and that may regular alarms on the doctor's system it certainly may bring little alarms in other places but publish the conclusion there's a reason to go back and get the original data fine find that later but publish that but it's also important to say this information is not always good knowing your blood pressure was was perfectly fine six months ago does not help you today however probably your your blood type that probably hasn't changed in the last six months so different data tends to have different longevity as you publish in the cloud and we thought it was very important to age this information out so this is a little refinement we had in mind certainly if you're in critical care situations heart rate and some of those things are things you want to be taken maybe every few minutes so again this was our idea about how to do these things uh again the client said no thank you we just want to rewrite our java system so uh when i was putting this presentation together i actually began to figure out that i've been very unsuccessful in my career but i still get paid well which is kind of a good trick um all right rolling right along it turned out right next door to this medical assistants firm was this financial firm that does you know funds and stuff in the u.s a very successful firm in fact so successful they don't make a good client good clients or clients that are desperate they let you try things but a client that is being successful is very very hard to get change in there because there's all sorts of things that try to stop you like why should we do this and you can't really answer it on faith this is one of those particular clients uh i turned out i decided i didn't want to work with them because of that reason but my friend jeff bay uh he got sucked into it anyway um and what he did was he said well we're not going to rewrite your system but we'll prototype a new sort of solid architecture he in fact was trying to get them to run faster they were on you know once every three or four months we do a release sort of thinking and he was trying to get them into a different place so he was prototyping a new system architecture it's not the real thing but a prototype and i create these things that i call the bayesian service principles and named after jeff bay and it was two profound observations jeff made first of all he says it's okay to run more than one verge another service at the same time and when i heard this i was like jeff this is wrong we got we've got a new service we got to make sure everybody's using it he says no you don't and i try to explain to him why and i couldn't come up with any reason which generally means you know it's a bad idea i'm stuck in the stonehenge age i'm back at stonehenge again so he said that and it was like okay that's that's kind of interesting and then he said something else really strange he says he only allows the system to deploy one service at a time that's deployment you cannot deploy two services only one and you then put another one so of course he was trying to say you can deploy a service and then you probably knew it for the client that works with that service you need to make sure it's that order and obviously you just get it makes it more difficult by the way to replace it with a version 2 and kill all the version ones because that's lots of coordination result of these two really simple ideas is that we sort of have broken the dependency upon that old system we've broken the coupling to that old code yes we may have somewhat duplicate services running around doing almost the same thing but jeff would say so what and again i didn't really have a good answer for that now assist these guys were basically deploying about once every three months with jeff's little rules they were deploying about three times a day so he had a profound impact about how that how you do that stuff just by basically going through and breaking some assumptions about how we had to do things uh so very quite profound so i would consider that actually uh somewhat of a success but he was a prototype wasn't a real system but of course gave me even more dangerous ideas so i took jeff bay again and he and i headed up to a large manufacturing firm automobile in particular and we're trying to put a system together for those guys because they had to replace some parts and existing cars uh and basically some parts were getting somewhat antiquated if they need to upgrade the parts i can't really say much more about it but it makes sense when you hear the domain um and they and they and it turned out there's just a lot of variations on information you were getting so sometimes a call center would say oh i found a car it needs to be replaced sometimes a customer call up say i have an element i have a car that needs to be replaced maybe it happens to be the garage it goes to to get repaired they notice it has that part needs to be replaced so it turned out we were getting information from all sorts of sources and lots of lots of different formats and different information so that was sort of the challenge now this this this company had a uh an in-house vendor uh basically that uh yeah i don't remember what it was but again in-house vendor had it and uh they estimated to change their existing systems in order to accommodate this water replacement was going to be about 18 months 15 to 18 months they could modify the legacy system to do that the business thought that was actually infeasible we needed to have this thing done with much much much shorter time so good desperate client allows you to go play so again unacceptable business so what do we do in that environment well i came up with this what i call the pinball model i'll be a little clearer when i talk about it a little bit but basically the idea was i have these information packets and basically i have a i need a lot of information in order to do the replacement order to get the replacement successful but i'm starting out with different little tidbits according to where it came from so maybe i know the name of the person and when he bought the car maybe i know the address of the residence that has the car maybe i'm really lucky actually have the car vehicle number but i have this all this other stuff and we start what we really want to do is we want to sort of take this little pinball and bounce it around the various services and get more information until it's all filled out so that was the theory we just want to kind of balance this packet around trying to get more information until it was full in which case now we have a successful order so how do we kind of do that we started defining little services and we tried to find a service as small as possible so this responsibility was basically just fill in a little bit more information but just one tidbit each so i want service as small as possible but it still does something interesting so for example maybe i have a service that's able to inject this you know aaa information of some sort but it needs to have a vin and it needs to have some other information in order to calculate that and then maybe i know the service that can figure out the event but it has to have a name and maybe another service that needs needs could get a name but it has to have an address so what i did was we started taking these packets and started pinging them in these services so you know event tends to be one of the key things i want so i i sent this little packet off to the vid service saying here uh you know can you figure out can you figure this stuff out and he says well no no i can't figure out your vin because uh i i don't have a van but uh but if you want the vin you need to go to this other service and he'll help you get the vin so we go down the other little service and you say oh yeah i can help you but give me your name well i'm sorry i don't have a name but there's this other service over here that can figure out your name you have your address and so we send it over there and say oh we have an address so let's fill in the name and then we sort of bounce it through again and we kind of go through again until he gets more information and more information more information and he finally gets filled out completely so that was the idea interesting enough as we looking at it of course this is a legacy system it has lots and lots and lots of databases i mean oh my goodness the databases but one of these we decided was we wanted to carve up the databases in terms of responsibility any given service can work with some tables but they're the only service allowed to touch those tables the only service and it turned out the most interesting services were services that tagged around join tables because joint tables seem to be where the action is when you sort of marry two static pieces of information together and making conclusions again think back to the medical industry we're trying to figure out conclusions and publish those so we took the database and we sort of carved it up and we found okay what they say table's this knee okay you're the guy that owns these tables and we segregated all the tables very nicely and basically we were able to take that system live in nine weeks so client was expecting you know you know 15 18 months we went live in nine weeks uh needs to say client was very happy about this uh reputation was enhanced uh and again desperate clients you let you play some things so i was very happy with that in fact i would consider that my success some of the some of the uh observations we got from that again adding on to my new dangerous knowledge um i began to think about when i used to talk to even jim weber back in china about this where's the service value what does the boundary of a service look like and i started drawing back on my own background my own background i've been doing object modeling for oh god it's almost 25 years now so it's kind of my natural way of thinking and i began to think that these class these services were just like classes in the traditional sense that they have some behavior and they have some encapsulated data but they encapsulate data is really the tables that they're allowed to own and they don't want to share the data with anybody else good encapsulation again but they still need to do something not just spit the data back out to anybody who asks they need to reach conclusions very much like good services so and it turned out these things got really small that when you carve it up this small it actually got quite tiny i also have a small talk background and it was people have talked about small talk and influence and stuff like that that was my first object language so i was kind of spoiled but one of the things i would learn and was taught actually actually taught as kind of a kind word beat the crap out of me was kind of another way of saying it but my teachers basically basically crap out of me about you these are not subroutines these are methods in fact if i called it a subroutine i had to pay them 25 cents and if i turn use the word i'm calling this call you know can't call that's the subroutine thing 25 cents please you're invoking a method you're sending a message it turns out that way of thinking really translates very nicely into service thinking so after being beat up for many years in object modeling i began to think this way as well and again the encapsulation turns out to be key in fact making sure nobody cheats and looks at somebody else's tables was kind of key to making this work now we weren't without a little problem so this is actually a new way of thinking about things but we did have some issues come up that were i guess not too surprising thinking back on it but again we're always surprised we don't think about them ahead of time one is is we couldn't figure out what's going on you got pinballs right pinball has been dancing all around and sometimes the ball would just keep going around and round and round and we were waiting for it to drop out and would never drop out or sometimes we go back in there and the ball's gone it's like where'd the ball go uh and so we had we sort of had to start doing some you know tracking and logging and and how do you count cycles so we started trying to refine some of these things some things that tools do a much better job of now but that was a little bit of our loss that wasn't the biggest issue the biggest issue turned out to be the programmers themselves and i had a it was an extremely bright team uh i think the guy who started cruise control originally he was on the team i had jeff bay on the team and of course he's a brilliant guy as well so i had a lot of brilliant guys on that team but they never had try to carve a problem up this way that they never tried to turn up in little services and encapsulated around the concepts and and dancing information around they really haven't thought about a problem that way so i had a rolled off release one they sent me off to london uh release two came along and these terms started getting bigger there's a little more sharing of the tables so some of this stuff kind of began to fall apart not that it didn't work in fact we kept delivering very fast certainly compared to the old cycles we were still getting rave reviews within the client the client still loved us but to some degree that pure implementation i had in mind originally my pinball model was beginning to sort of crumble a bit there was a technical day at thoughtworks and they actually this project presented a poster at that and got an award they got second place for award for a technical poster but the subject of the poster was what a bad idea this was so i'm a little insulted by that so i'll go back and say we delivered but maybe it wasn't necessarily a good idea because of some of the issues we came up with all right roll the clock a little further i'm off in london now oh by the way going back to this so now i have a new view of this again i'm a little smarter or i think i'm smarter i now begin to believe in 2006 that there's no such thing as a 5 000 line code service it's way too big that maybe 500 lines of code is a reasonable size for a service and so i've now refined my my thinking even further all right so now i'm in in london i'm working at ford ford's by the way one of the sponsors here um a great company to work for it's really hard to describe what they do because if you look at some of the products they have energy comparison phone comparison uh browser plugins you know little toolbar plugins we sell we sell parent cages online we have a warehouse and suffolk uh doing that stuff it's kind of really we actually have a controlling interest in two perfume firms uh it's really hard to describe other than the fact that we are probably very technology oriented and we like to market on the internet other than that there are almost no rules for what we will do so these are my our experiences of that because of course i got all these ideas now starting back in china going back to the states and now coming in to forward and you know first probably get unveiled is we are we are having these we're doing in advertising and in internet advertising google's spitting out reports that you like you wouldn't believe about how you're performing and taking that report data and analyzing it is very key to becoming very profitable because if you actually go out there and advertise on internet you're probably going to lose money about on about 8 of the 10 ads you've got out there i mean the numbers are really horrible about how effect and effective it is but the trick is to turn off those bad eight and keep the good two and how fast can i do that makes all the difference in profitability so of course now i got great new ideas of how we should build a system to accommodate this business so uh i said pub sub model in fact i was going to use something called linda spaces originally came out of mit there's a version of linda's spaces built into ruby called rinda i think it's still in ruby now it's still in room we call rinda spaces not that i know anybody's ever used them i was trying to use it but uh you know we'll see how that story turned out so i decided well what we want to do is have again let's let's build our database but let's make sure we segregate the tables and not let these tables be shared because we're building a brand new database i don't have to have sharing of that let's have little services little agents that are sitting there analyzing this data and pumping out to people who are interested like these marketing people so i want to be able to pick up a service pick up this data analyze it and pop something out to this agent saying here's something you should care about it's very much like my medical systems thing all over again and so uh we start to begin to automate these recommendations so the more we understand about what they're doing the more we can sort of take this and make a clever service that says oh here's some new keywords you should try or here's one you should turn off may we even turn them off for you as we continue to evolve this system so again i was very excited by this we were using an offshore offshore teams for implementation uh again i ran across the problem that the offshore team did not understand the architecture i was trying to implement and being all short made it even more difficult and i had a challenge that the uh the cto the organization was a former oracle guy who says oh you just need a big database and all this problem goes away so uh reject it now interesting enough we come back a year later same company and we started to basically implement an agile card wall because one of these i did bring into the company was agile thinking not just agile thinking for software development but agile thank you for everything in the company whether it's marketing and how marketing programs we're going to have and we'd have story cards for that and we show them the customers and we'd have you know retrospectives with a customer about what stories were playing and we counted points we put agile everywhere so we we built ourselves an electronic wall to sort of track it very lightweightly but one of the things we started adding into the card wall was we started basically doing some back-end processing these google reports and started publishing alerts to the card wall to tell people to use card wall that something interesting has gone on in other words the system we rejected the year before we started implementing again and in fact we continue to use that system we actually went to hadoop clusters we run five to seven thousand jobs a day you know aspire was 600 500 600 different services running across that period all publishing into this sort of architecture and so to some degree we finally have delivered a system that was along the lines we talked about now observations and recommendations from in revelations sort of from that experience first of all one of the things that was interesting is these little services became almost disposable entities we didn't really care about keeping it around for a long period it was useless we threw it away and wrote it again they became literally very disposable entities there's no reason not to keep it if i don't have to and and it turned out we loosely couple these things so we really want them not to have a relationship to each other so maybe they share a database maybe they pull out a common reports maybe they pass for restful packet adjacent packet each other but fundamentally they don't really care much about each other's internals now the only thing i didn't like about that is they still knew that this one ran and then this one ran and then this one there was a sequence that was sort of implicit in their relationship so that coupling did exist we also basically got rid of unit tests so how do you get rid of unit tests well first of all we're talking about 100 lines of code if you need a unit test to try to understand 100 lines of code to help you design it then you're probably in the wrong field so we would we did that but what we did do is we required these little services if they could not run if they couldn't find a little guru report if they got some garbage from somebody else they should raise their hand and say excuse me i can't run and we would take notice of that in fact big big monitors in the in our room would go red we'd go chase down what the problem was that became our basically our concept of replacing unit tests because most of the time our services worked fine only when they broke and usually grow so it's a very strange reasons will we go chase down the problem a unit test would not have helped us find that sort of problem now a side effect of that was we also got rid of our acceptance test you say well you got a complex system you need acceptance tests well i would say think back to india i mean i had a complex system there many lines of code i could write acceptance tests and get them to run it's complex system so it's going to be complex give up on that but what we do like to do is we like to monitor business metrics by business metrics what i mean is how much clear many clicks do we get how much money do we make how many parrot cases did we sell yesterday we decided that turned out to be a great acceptance test first of all it's not running once when i deploy it's running all the time and so when that when those metrics start going down we are suspicious something may be going wrong we just go check into it maybe it's our software or something we deploy maybe it's just bad luck maybe we have another oil crisis who knows but it's suspicious uh there was a paper given at the go-to conference and our host last year and a guy was proposing basically you should write your only real acceptance test is business metrics that is the true black box test i don't have to understand anything about the internals to measure the business if you measure the business it's a very robust sort of test and we found that to be true as well the side effect of that was we don't care what language you write these services in let me tell you the programmers got excited by that they started playing with all sorts of things we now play with closure we don't play with node.js you write in ruby maybe you write something else we don't care it's 100 lines of code well what if you pick up somebody else's code in in its enclosure it's like it's on a line of code i understand it rewriting ruby i don't care and so the programmers got quite excited by some of this stuff and it drives a lot of the motivation as well so where we stand today um one of things we inherited we bought a company a few years ago you switch which does energy switching to uk in the uk you can switch energy plant energy providers the same way we switch cell providers in the rest of the world so it turns out to be a fairly lucrative business but the system we had which supported lots of sort of switching capabilities was a legacy you know microsoft stack i mean it was absolutely gorgeous microsoft stack in its time web servers app servers sql servers uh i interface here or any interface there everything was exactly as designed but we want to think about that differently because it was getting to be 10 years old we had almost no programmer it was ever around when it was written nobody really understood how it worked anymore getting difficult to change but we had two new influences on how we wanted to do this we didn't turn the same team loose you turn the same team loose and say rewrite it you get the same architecture we didn't want that so we went to the other team that was doing the little small services and we got a couple of those guys and brought them over and they had two influences first of all we began to believe that events are more important than entities we think about entities when we talk about object modeling there's a customer there's a sale these are things that you want to hang your hat on but according to sort of the event world they're not really there so what's really there is it turns out and this comes from uh you know one of our guys reading the end of plex which is about obviously about google that google basically report just requires all sorts of little events if you go to a web page searching something you recorded it if you have to page down to find the search they record it because something's wrong with it if you click on an ad they care about it or if you click on the third entry and not the first entry what if you click on the respelling that they suggested that's important to them now they don't know why it's important but they record everything because it might become important in the future we want to take that approach other influence was we thought we actually our old system couldn't record everything because it couldn't have enough room database was too slow we had to pick what we wanted to sort of save we don't want to lose the old stuff the old history is very important to us again going back to google and here we're influenced a lot by closure because closure basically has the idea that data is immutable i mean once you create data it's there you don't can't change it i mean if i was standing here at the beginning of the presentation that is a fact the fact i'm standing over here now is also a fact but it didn't invalidate that fact that's a perfectly good fact as well let's keep both the facts again we're influenced by that thinking so we took the old structure which was basically you know as you would expect you know towers of stuff stonehenge all over again and of course we have a different stonehenge for each of our services we had all sorts of data we collected in runtime and of course you need a reporting database it was not nearly that clear reporting database went back and forth and back and forth and eventually gets back to the reporting database it almost never worked it was every day there was something in that pipeline that broke and we didn't have reporting but that's what we were stuck with so our new goals basically um we want to keep the history because we think history really is important uh we want to be able to run real-time analytics because it's a real-time business this is a business in fact that is all about making sales and making transitions we weren't able to figure out until end of months where we made money that's in this day in the world and we also basically want to make sure our data was easy to change our system so we created ourselves a set of signals again google like so our little our services begin to collect things like oh you just gave me a postcode or oh here's your email address or perhaps here's your real address or or you just hit a page and or maybe our performance of our system is running a little slow little events we just start putting these into a cloud and it turns out when you start grouping them together you get really interesting things going on remember there's no entity here where is the customer where is the sale it's kind of scattered across the events stream of events but we started seeing interesting patterns like if you take the postal address and the email address you probably have an opportunity to sell an energy plan but what happens if the email address changes that probably means somebody new has moved into the house new opportunity but what if the postal code address changes well that guy is moved so maybe another opportunity so we started backing up and saying oh my goodness if we collect all this little bit of data i now discovered a use for it i didn't pre-discover the use for it that's the google thinking just collect everything you'll figure out useful things to do later so we started doing that so we have guys that are producing all these sorts of events and pushing them in there we have consumers we're doing a lot of stuff with r doing statistical analysis uh yes we do like math uh but we also write some real time things little sometimes little perl scripts sometimes little ruby scripts just collect data and stream out very interesting things now we decide we park all this data into kafka kafka is a project that's been been given to uh apache from linkedin so this is the linkedin tracking system we're tracking all that stuff relationship stuff that that linkedin worries about and it tends to meet the need of a cloud of events very nicely because these events are related in various sundry ways very much like linkedin is so we use that to put that information in there and kafka allows you know streams to wake up and keep playing from where they left off so we've been discovering some interesting things we're discovering cross-selling opportunities we discovered that you know if you're coming in on the red site we we pass traffic to the blues blue guys and the blue guy is passing good traffic back so we're cross-selling products across this we're seeing the patterns of cross-selling so now we know how to reinforce that with better advertising and page length we're also seeing patterns of user journeys we can actually now map just by collecting this data running through various analysis we we build all sorts of very pretty little maps that tell us you know what's the city of our universe what pages are really getting hit hard where do we need to reinforce ourselves so we're collecting a lot of information about that and one of the side effects and this comes from another presentation i make called programmer anarchy very much like the presentation you heard on the first day about github you know we're basically running that same sort of environment but to some degree we've killed acceptance tests unit tests refactoring patterns all those things are things we do to keep a big stonehenge running insane that's what we need to keep it sane in that environment but if you kill that environment you're writing 100 line of code services all those needs go away it has it actually was i was quite surprised by this basically our architecture has now influenced our processes in very very fundamental ways and again i was again surprised that that's a result so where do we stand in terms of technical debt in this world well if you think about all the reasons we had technical debt to some degree the cause of technical debt the impact of technical debt you know reaching through and grabbing that number 50 and using in 37 other places it's hard to do that with a service i mean services by their very nature you just can't grab their instance variables their service over there they have adjacent interface they're restful and so it turns out it's very very difficult for 100 lines of code what if you do write a sloppy one it's still only 100 lines of code rewrite it or maybe it's wrong rewrite it it's not a big deal it's only 100 lines of code you have not impacted the entire rest of the universe and so to some degree this architecture has also reduced the risk associated with technical debt i no longer worried about programmers not understanding enough about this so it's actually helped that problem quite a bit as well so conclusions and i say so far because i think this is still a journey for me uh and the industry to go on but some fundamental principles of microservices they are very very small i mean 100 lines of code is probably actually a big service these days and that just hits all sorts of things you do need to make sure that decouple as much as possible we're starting to decouple even flow things now we're starting to do a little more pub sub and decoupling that as well multiple versions not only not only is it okay we probably encourage it because again uh if the old we can actually monitor since it's in the cloud this service has not been invoked for three months let's turn it off see what happens see who gets upset um certainly the services monitoring each other very important but it's also a key responsibility of every service to publish important events if you did something interesting you should publish it don't wait for somebody to give you a requirement for that that says we have a need for that data if you think it's interesting publish it we'll find a need in the future and finally the whole concept what's an application in this world i have no idea what it is but that's a real term that does not fit anymore so that's kind of gone away as well i think we are in fact building living software systems my analogy here is is basically um that we really have a long-lived system the system this uh somebody says we're building are now two or three years old but almost no service is much more than six months old in fact six months old service is actually quite an ancient service it's very much like a human body i mean your cells that you have here are probably not the same ones you were born with yet we're still i'm still me and we want to think about our systems that way don't worry about the individual parts the entity as a whole is going to survive quite a bit now we in in four we are very aggressive about deploying on average we're from a software perspective in fact sometimes more often than this i think last time i measured we're deploying something in production every three and a half minutes uh that means average on every program is probably two putting something in production every twice a day that's on average so we're just pushing out stuff again it's a little hundred line things the risk is minimal the interaction's minimal we got the business metrics on the other end to make sure things work we're just pumping out see what happens pumping out see what happens um we just got to understand first of all it is a complex system and if you think you're going to have a giant test that ensures it's perfect you're kidding yourself so accept that and write your tests around business metrics and again it had a radical impact to their hierarchies the hardest part will be in fact getting programmers used to thinking this way because this is not we've been trained to think in stonehenge and this is not stonehenge and so we find that if we drop a programmer into the team that's doing things like this they absorb it pretty quickly but if you had to start out by yourself and try to think this way you will probably get stuck uh there's a theory of corporations i got a business degree among other things and the idea of a corporation is we we assume in corporation theory that you're running the business that's going to be a business forever we're not trying to do anything stupid to kill the business and i think the same thinking needs to be applied to our software that we need to think about software as being something that's going to be running forever think about your system that way don't think about something that's going to run for the next five years and we're going to turn it off i think the statistics are out from a few years ago on average we plan for systems the last 10 years in practice they live 17. so we're not even very good about guessing how long systems left but assuming it'll be there forever so that's my story of my journey microservice architectures um i've come a long way since the stonehenge part i think um i i begin to carve up systems in very very tiny pieces i'm finding it much more fun uh because i can write a little tiny piece and get it deployed and i'm feeling like i deploy something every day feeling really good about myself versus my old ibm days where i worked on projects that delivered five years later not particularly satisfying work all right appreciate it thank you uh are there any questions thank you very very good talk so uh someone i know who's um experimenting a lot with something that could support this architecture right like very small like sort of actor-based stuff right his his basic thesis is that you can't support like adequate uh processing in a system like this because the time that you spend sort of deserializing serializing those messages between the services is going to inhibit what you're actually ultimately going to be able to compute did any of these services that you guys worked on were they computation bound were you worried about actual resources no not really most most of our time is spent basically getting getting reports off google the analysis that report is very straightforward we're basically time-bound relative to bringing the information across the net and so we haven't gotten anything that that sort of has that characteristic to it and we're also basically in forward we choose businesses that have fast feedback cycles fast business feedback cycles we want to engage in a business that builds a product that takes five years to figure it out we would avoid that sort of business so to some degree we are working in in rarefied environments uh but i i think that i'm comfortable with that i'm okay with that thank you of course punch cards yes so once you start breaking into small services naturally the question arises even before the cpu bound the question of dependencies and how do you deal with all the dependencies that might arise between the services unless you are in a very specific case where there are no dependencies but um you know there are dependencies that's why we have flow dependencies among them so i can't analyze the google report unless somebody's pulled the google report but that's it's relatively simple sort of things like that the business domain is in many cases not that complicated in terms of its business flow and somebody said before you know you can't make the system any simpler than the business itself but working some some businesses that are fairly simple and straightforward i put some ads out there google tells me how well they did i pulled the report analyze the report you know change my value on those ads and those sort of cycles are pretty easy to put together now the knowledge of how the system works lives within the team that's working on it we never write it down but amongst them they do have the understanding of that and that's why we can roll people in and out of that team because the knowledge stays in the team but you know our businesses are some ways not too complicated you know parrot case you just you know order a pair cage sell a pair okay ship the parrot cage take it back uh you know not the most profound businesses okay thanks i'm curious because you mentioned linda doppels which is a very old idea from the early 80s and during my phd i worked on that a couple of times but at the end there's a fundamental problem of scalability because trying to match very high number of toppers usually was not there are many ways you can kind of you know match them how do you solve that problem because this is complicated oh i i didn't think it was that complicated for the domain we were working in at the time uh because the number interests were really quite clear now i i'm not to say we wouldn't have had a problem if they actually said yes but of course they said no so i can i could sit here and say it worked fine but we don't really know um i i thought it was important though that we'd be able to you know account always once every event that had had to do with everything so that was easy to send those over there i think the biggest part of the render space challenge in that system was we did did have multiple subscribers uh and they were interested in slightly different things but you would definitely write a for example a physician a desktop or physician agent would basically if he saw something critical come across i went flashing red lights on the screen and stuff like that but that's because it's already been marked critical so i didn't want to over design the data because certainly medical field is where they got crazy with xml once upon a time and built some really elaborate stuff and i didn't want to touch that at all i want to make a very very simple straightforward conclusion sort of based you know urgent critical normal and then just that's all i care about and it was still run ranged around the legacy system which had holes in it so i just have to be you know try to stay with that but i don't know because we didn't finish implementing it hi i i can see how the system you're talking about the architecture would work well for systems where the data flows in primarily one major direction did you ever try to apply this to a system that requires something like user experience feedback where there's things like error conditions that have to be bubbled up uh immediately back to the user uh to some degree you know the the web sales with your switch we're doing switching that's basically a web-based front-end highly interactive stuff it has a bit of those characteristics but we're not talking about uh i would say anything more than an e-commerce sort of sales kind of concept so i would say an untested that field uh you know one of the things i'm doing i'm starting to work i started work last monday for an online newspaper so i'm now going to take a lot of these ideas and apply them to a new domain you know online publishing stuff to sort of see how much of this stuff works i'm comfortable it definitely all worked for us and forward but ford is a a relatively strange place in some ways uh so therefore we definitely take advantage of that in some ways so i wouldn't necessarily imply it works for your environment so um the 100 line services is really catchy and i actually do enjoy and i want to move to this thing for just us in general but what about the orchestration and discovery and routing that's certainly not a 100 line service is it well first of all i think you're doing orchestration you're building a god class so i i don't believe in orchestration uh you know working jim weber that was that was the era of orchestration and basically you're building this god class that we you used to have in our big systems that said everybody flows to the god class he decides who gets it next i think that's a dead idea i mean that's not redundant it doesn't scale it has all those other issues i'd much rather sort of distribute the intelligence that's why i want to publish into a bus that says i have something interesting and you subscribe because you're interested and i want to decouple those two here i don't wanna i don't wanna orchestration i'd rather have decoupled interest and pumps up stuff so that was a deliberate choice on that side um i do believe if you get get stuck in trying to build this god classes or god services or where you call it orchestration or whatever yeah you're probably going to go down a rat hole that's going to kill you i don't think i answered the whole question though oh yeah i i've already made that mistake thank you very much i'm not going to try that one again yes so especially when you start talking about data flow systems uh how do you deal with different time scales and events on different time scale systems because a lot of times events need to be synchronized across services so what do you do if you have for instance services that require some i guess a similar idea of temporality uh we don't tend to have that in our industries that we work with i mean the closest thing we have to sort of synchronization is i can't obviously analyze a google report until it happened but that's sort of the highest level temporal defense that we have we have very i don't think i can think of any case where we have two things that have to be coordinated and therefore before the next thing happens but if i did i would probably you know drop them in some non-relational store and if the service would keep waking up and not finding it didn't find it in time he raises a little handset i can't run um but in general we wait we prefer the system to fail than try to do feedback prevention it goes back to our whole if you think about it tests themselves are not lean in any way whatsoever because tests are there just in case something goes wrong anytime i use a word just in case or it may be necessary i've now moved outside the lean agile world i'm doing something but almost by rock i really want to be just in time in these sorts of things i'd rather have a system fail fast than do defect prevention and so that's why my unit tests have gone away because i have i'm deploying every you know twice a day i have my business metrics to tell me when i screwed it up i don't need to write this test uh so to some degree the temporal things i would try to set up as a failure in my system first before i try to try to do the coordination uh but again we don't tend to have that in our domains at least not that i can think of and i got to shut up apparently i guess i saw the handwave okay i'm afraid we have no more time for questions but uh thank you fred thank you [Applause] [Music]