Building Resilient Live Service Games with Python: Lessons from the Trenches by Willem Thiart

Hi. Hi everyone. It's really good to be here. Um, so I'm going to do a presentation in three parts. I'm going to explain what is demon's hand. Um I'm going to show you a really cool card DSL. So domain uh specific language and lastly I'm going to f finish with some resiliency. Okay. Um so what is Demon's Hand? It's a card game. Um live dungeon multiplayer. So there's other people playing with you. Um, scoreboards, live scoreboards. We've got emotes. Uh, the dungeons live, so there's lingering presents. I've got a nice world map. So, uh, over here you can see that's the hand. This is the board. You take, you basically just drag your cards to the board and stuff happens. Um, availability. We're on Google Play. says, "Really good." Um, going to go on it. Uh, uh, I mean, I'm old. I said iTunes Apple Store. Um, and Steam was everyone, uh, everyone as a gamer actually wants a Steam version. Um, cool. So, architecture. Um, sorry, it's a little bit of a AWS tinge because I I like it's what I know. So uh but the Python icon should be bigger. So we're going to focus mostly on Python today. Okay. Um and uh sorry I think the realization might be a little disappointing to some people but um the the client is is is written in Lua. Um we use the default engine. Um but in the presentation I've got a good news for you. Okay. Um, so we've just got two forms of connection. We've got the old web socket. Uh, it goes straight to AWS, has a really nice, um, interface and we kind of like mangle it a little bit. So, it goes on a queue, nice FIFO queue, and EC2 just gobbles it up and starts doing uh, gameplay logic. Um, so game danger logic goes to on the on the queue and all HTTP just follow like meta data and stuff like um if you want to see what's the world map HTTP you want to get scoreboard HTTP um everything that's not gameplay logic you just just do HTTP um and I like serverless function as a service so um there we go um just a question This does anyone got like an idea why we did EC2 instead of just lambda or not bikes? What's up with >> um RAM RAM games breathe RAM. Um Lambda doesn't really have a RAM, you know, because um a lambda executes and that RAM's gone. you can't really trust it. So, I would have done everything in Lambda, but um you know, we we got to use EC2 because of RAM. Um websocket over SQS. What's going on there? Um so, we love async. Um everything's async. It's just the way to go. We got a nice infinite loop over here. Um and we just pull things out of the FIFO queue. Come on. Um, and that's how the game logic works. Um, so, so, uh, over here just got a nice big fat gather and all the game simulation just gets pumped through and then we get the IO out and then we post the IO at the bottom. Um, this is not the actual algorithm, but uh, it's executable pseudo code. Um, I think recently free threading's come out. So, I think I'm going to do that next. Okay, cool. So, HTTP function as a service. All the stuff that players want to see and want to know about goes through a JSON API. Um, so good. This is actual code. Um, so Python just makes it really easy. So, okay. So, so, so even though we're using lure as the game uh code as the client side, it's a thin client. Okay. So, um, COD, the client has got no idea how the cards work. It's got no idea how the monsters work. It's got no idea how anything works. Basically, Python the server tells the client what's going on. Um, so it's so we got all these events. It's event driven. So, you know, we say draw card and then the card pops up, pops into the hand. Um, we got effects and stuff like that. Um, you see, you see that targeting system over there? Um, client doesn't know what it is. The server basically says um that card over there uh can hit that that monster over there, that monster over there, and you can move there and you this is also an option you got. Um so the card has got no idea about the game logic. It just gets a card from the server and the server says this is what you can do with it. Um it's really good because I can spend more time in Python. Um so Lou is a thin client. This is an example of a really critical uh message we send to the client. So, sorry I got to squeeze it over to squeeze it. It's kind of weird when you see Jason like that. Um, but yeah, that's an event. That's update hand over there. So, I the service says, "Hey, update your hand. Um, you got two cards today. You got cleave um and cost three and you've got a move card um and you can only hit those two targets. Okay. Uh and over here with cleave, cleave can do can go to that target, that target, that target. And um that stuff is affected by cleave because it's it's it's a uh I'll show you later. Cle's actually in this um surprise Google Sheets. So, where do we put the game rules? Where where do you put the DSL? We put it in a spreadsheet. Um why not? Um I used to be an accountant. So, um it seemed intuitive. Um yeah, this is the spreadsheet um part of it. You can see we're at the bees. So there's a lot more. Um you can see like we got like target of all. Got this DSL happening here. Um we got nice dice roll notation. Everyone loves dice roll notation. Um and then we just got the effects. I call this on effect. Uh it just tells you what the card does in a DSL. Um you get used to it. Um and then this is how hairy it gets. We've got like some sort of hook system. So you've got okay played on turn end um on turn end you draw one um stuff like that. So stuff's happening on this this uh this this uh all these hooks. We got hooks for everything. Okay. Um add a new hook. If you can't if it if it's tricky to implement something, you just add a new hook. It's fine. Um all right. Um, so I'm gonna show you the DSL now. Um, what does greed do? >> Draws two. I always forget what it does. Um, so over here we got uh just draw two. Um, that's the DSL. Um, got another card, coin flip. Uh, discard one. Um, draw from the deck. Um, so it's quite nice because it's actually uh sometimes it's more tur than the actual uh English which is really good. Um, it gets a bit more interesting now. Uh, we got a do. So do lets us do two lines like that. And we got another like a little if statement. Um, I think that double if might hint at what's going on. Um, but we'll get there soon. Um, so, and then we got the nice column for the damage. And that's how you implement uh, scavenge. Um, smite three. Um, so it's got a little bit of a if there's an undead monster, you use smite, you get a bonus. You get a super effective bonus right there. Um, what's going on there? What's going on there? So um [clears throat] so so like do you know what the dunder methods are for? They're for overriding. Okay, you should override. Don't that like like sometimes it's like oh why is this overridden? It doesn't work the way. Just override it. It's fine. There's no judgment. CFP. >> Um, well, nothing else to say. Just explicit is better, right? Um, so we just list all the stuff over there. Um, I like this one because you can see that the DSL is doing a little bit of like some sort of set magic. So, you basically say like this big set of monsters. Um, just add the status to them like that. And then, you know, now they're frozen. So, that's a status effect. I I think is quite nice because you you basically say like this thing impacts monsters. Um we'll do we'll show you more of that later. And there's a cleave. So So it's kind of like this type of action going across. You can see that the uh like it gets pretty cool because now you can pipe stuff. So now you've made a set. Um so the target with the when you when you tap the card on the left hand side and the right hand side the effect takes place. Um the most ambitious crossover since then 2010s all these uh functions crossing over. What a what what an ensemble. Um really lots of fun. you just say for each over there. Um and uh yeah, just a portion 51% uh just makes sure that you catch um what was the reason for that? But there was a reason for it. Oh um that wasn't supposed to be on the slide. Um game development, am I right? Okay. Um so on effect over here um chance so uh 70% this happens um easy peasy just just just try to remember this okay so chance 70% and then the effect happens over here um so surprise the DSL was actually just eval so all the stuff I just showed you is actually just execut ex executable Python um I love it it's so Um so here's an example of an implementation. Um chance is actually just an object, just a class. Um every single function has an execute method. Okay, so the system is just pumping out events. Okay, and these events uh there's a little bit of filtering happening, but we send them to the client. The client gets it. Uh we give it to the client. So the whole system is just generators generating events and giving them to the client. Um really like it. Um and for Eval, here you go Evvel, here's chance. And we just do this little silly thing over here. Um so that's all it is really. Um got a question. Why did I put the mutation decorator here? This is this is a custom decorator. What why I put it here? What about this is mutating? What does this what is what is chance mutating? Yes. Yes. Yes. Um in a card game, the random number generator is a resource you need to be fully aware of. Um if you're not paying attention to the number generator, it just gets a bit confusing about like you can't reproduce problems. So you got to plum the seed at the beginning of the game. Um, so that's why good answer. Um, so this is just another example. Nothing to it. So the little do thing, you just you're just iterating over the for loop over there. Um, and you just execute. Oh, over here you just get whatever it is and you just execute it. So everything is just generators. I love generators if you can't tell. Um but um it's interesting because like I don't have any async here. Um that's because nothing is allowed to do async. I've got a slide about that later. Um I'm not supposed to do any IO. Okay. So in the game state, no IO. Not allowed to do IO. I'll talk about that later. Um all right, the draw. So this is the draw function where you draw a card. I don't need to read it. Don't need to read it. Just need to know. Um, it's complicated, but you just hide it. The DSL hides it, right? So, whoever implements uses the draw function is like, oh, just draw one, draw two. Um, you can draw with all these other thingabobs. Um, just makes it easier. So, so this is where the DSL is really nice, everyone. Uh, you have super complicated game logic hide behind a DSL and, uh, it's just so much easier to reason about. Um, so this is just the deal the the card set. So we can do stuff like that. Um, the dund's been um overridden again. Um, overriding all these dundas. I love it. Um, you do some extra stuff like that. Uh, nothing to it really. Nothing much to say about that. Um, so this is the implementation of the card set. Uh, we got a pull function. So everything is just pulled and because we've overridden these guys we can like um get you start pulling from the or set and the answer set and you just get like a single stream. So we got a lot of card sets. Um just add as much as you want. Um makes it easier to reason about when you've got all this stuff. Um yeah. Yeah. lots of card sets. So you can like and and pipe those. Oh, so um does anyone suffer from happy code? C code that when you look at it, you just you just you're just really happy. So I really like how these worked out. I just found like it's quite um yeah, just this is so good. Um, okay. So, now I'm just going to move on to some resiliency stuff. Um, I want to talk about or JSON. Just one slide about or JSON because it's that's how good it is. It's just a nice swap in replacement. So, does everyone remember the the Google Sheets? Well, we're pulling it in here. Um, but we immediately loads it with all JSON. So, good. So much faster. Um if you do lots of JSON just just just just switch to um JSON watch out watch out for encoding though. Um UV loop. So because we love async there's this thing called UV loop which is a async event loop uh just just implemented a little bit faster. Um and this is it. This is this is this is all the calling code I got. Um, we just got UV loop install and for the rest for the rest of operation, we're just using normal async IO. Wonderful. I think the fact that it's only one slide shows you how good UV loop really is. It's not much to it. Um, it's a lesson. You can replace Python's uh batteries with whatever you want. Okay, it's just so good. Um, this I think that's a monkey patch. So, you can do it in Python. No problem. I think another language is a little bit harder than that. Um, HTTPX. Um, I love it. It's great. So, oh, AWS, right? So, um, we use Boto Boto 3. Um, but it's not async. It's not async. What's the point? Um, and like there's weird stuff going on in Boto 3. It's doing this P shape stuff. Why is it using CPU? Um, so, uh, I know I know there's other like other async libraries, but if you in in gamede div if you want things to go if we want to squeeze out performance, sometimes you just got to hand roll it. Um, so what we did was uh we took some of the services we liked like the as services and we just rewrote them in httpx. Um, so this is this um this is my favorite database. I love Dynamo DB. Um, not sponsored by AWS. That could be. Um, but yeah, all you got to do is just use the methods you love, make a nice interface. So you make your own interface like that and you just start implementing stuff. So So over here you got to have a baseline, right? So you get bo 3 and that's your baseline um implementation. Why do you need a baseline? Because then you you got to measure against something. Uh who who cares about the HTP X implementation? If you don't know how fast the Boto 3 one was. Um and if you want, you can go crazy. You can use Gevent. Um it's really nice. I am. So listen, if you spend time doing something, you can always make it a little bit faster. Um, just because the libraries exist, this the the ones that everyone uses, doesn't mean you need to use them. You could you can actually just roll your own. Um, it's okay. It's okay. Um, so I've fallen back in love with ABC from import ABC. Um, so good. I thought like I'm not going to use that stuff. It's it's it's it's solid objectorientated role play. Um, no, it's really good. Really good. I really recommend it. Um, so in the previous one, I showed you how you can just make a new implementation. It's really good for experimenting. Um, um, if you want to see like, oh, how is the system working? You just look at all the interfaces. You see how it like interfaces with everything. It's great. Um, oh, that's a little bullet point about LLM's. Um, yeah, you just tell LM, hey, uh, look at that, that interface over there. Can you make a new implementation? And, you know, maybe it stays on the guard rails inside that interface. So, interface is a really good, um, API. Um, so, isolate IO. Um the game loop doesn't have any IO inside it. Um which makes it easier uh to reproduce issues, you know. So if you got like a bug, it's easier to basically reproduce the bug. Um and I like having the IO outside because you can start batching it. Um, sorry, another Dynamo DB thing. Um, so scoreboards. Um, actually really easy. Uh, you just save your score. So you put the item there. Um, you got a partition. Can you just say like it's a score or whatever score. Um, and you just Z fill. Well, if your if that's how your database works, if your database is stringly types, you you z fill it. Um, and you just fetch it like that. um actually works pretty well. Uh I I guess we could slap a cache over here and call it a day. Um but it's it's it works okay. Um everyone should use PI instrument. It's so good. So good. Uh because we're using that SQS infrastructure, I can just say, "Hey, turn on the profiling." And then you play the game. Play the game. And then you you turn it off. Turn it off over there. And you can just fetch the profile from S3. I I think that's a good profile. I think so. I'm not sure. I'm not actually good at reading them, but like um you don't have the big fire thing. You got like it's like something seems like that async is working. Um and la last slide. Um, so there's these really cool pips that came out a few years ago actually. Um, PIP 73, PIP 738. Um, you could put Python on a mobile phone. >> Why am I not Why are we not doing this? So, um, uh, there's this bus tunnel like in Wellington, and it's I don't I haven't ridden it for a long time, but, um, what happens to your connection when you go through that thing? Um, so so you can let players like run the game on the phone. So, you put you put Python on the phone and then, uh, default is talking to Python and nobody knows. Um, and maybe this is like a little bit of research I got to do, but because the server is mostly a random number generator, theoretically I could say to the client, "Hey, hey, have a 100 random numbers." Okay. And then it should be able to or you could cheat. I guess you could pull the random number generators out and figure out how lucky you are. Um, but when you're out of the tunnel, then you just push all the input and the server because it's deterministic uh could figure out like how well you did while you're going through that tunnel. Um, cool. Thank you.