RCE 117: PMIx

[Music] welcome to another edition of rce again this is brock palin you can find us online at rce-cast.com where you can find over 100 episodes about research computing scientific computing and other topics i have again here jeff squires from cisco systems and one of the authors of openmpi jeff what's going on hey brock today we actually have one of my colleagues from the openmpi community here today to talk about what kind of started as a sub project but has grown into an entire community and project uh unto itself so i think i'm just gonna jump right in here and say ralph i wonder if you could introduce yourself yeah sure my name is ralph castain i'm a principal engineer from intel corporation uh been working on openmpi with jeff for a long time now and we are we started this pmix community that jeff asked me to come here and talk about today all right so ralph uh you you said the keyword right there pmix uh why don't you give us the two minute version what what is it yeah so it pmi x stands for pmi exascale process management interface exascale and basically what it is is a it's a standardized way for applications to interact with the system management stack like the resource manager and things like that you know to request services of various types and uh and be able to get a response back so what is a uh history of pmix because i think there's some other pmis out there yeah there are um pmi uh originally started you know quite a while back now as a way of having the uh providing a way for applications to wire themselves up so they'd be able to exchange addressing information basically to say how do i talk to you um but over time what happened was that the uh the needs grew to where the application needed to be able to interact over broader topics and so uh pmix kind of grew from that as a response to that to say let's give you more ways to interact okay so you said a second ago that this is dealing with resource managers i assume you're talking about like torque and slurm and lsf and all the others that are out there but what exactly does that mean what what does an application need to get from resource managers so there's there's really two things or two types of things that they do first off when you're launching a resource manager can provide you with all kinds of information about your job that's really helpful when you're trying to optimize communications and collective operations so they can tell you uh you know where all the your your peers are located uh what addresses they have etc that's information you can have at the very beginning so you don't have to exchange it afterwards then the other things that you you know that you want to do is for example you might want to ask for additional allocations of resources or uh or maybe you want to spawn additional processes so there are these kinds of services uh you know what is what's the status of cues and things like that that applications as they're evolving really want to be able to do and so in my limited world view here i i think of this is applicable to mpi applications and so when you talk about wire up and communications and things like that you're referring to mpi applications of any scale honestly from you know two processes to two million processes that when they start up they need to exchange mpi addressing stuff to get their ethernet addresses or their infiniband addresses or their whatever type of networking addresses so that if i mpi send to you then i know how to open a network channel to you that's the kind of stuff you're talking about yeah in part that certainly is is in there but it's also applicable to non-mpi processes so um let me give you an example say i'm in a cloud and uh and i'm running an application it doesn't have to be an mpi application any application in the cloud well there are things like for example when i get started i can use pmix to communicate to the cloud manager that i'm willing to be preempted and that now can be a service kind of thing you know where hey if you're willing to be preempted you get a different rate on your charges so i can now announce to the to the uh to the cloud manager that i'm willing to be preempted and i can use the pmix mechanisms that the cloud manager can tell me hey i need to preempt you now and wait for me to say back that i'm ready so i go ahead and checkpoint my job or do whatever i need to do to prepare for it then i get to tell the cloud manager back you know i'm ready to be preempted now so there's a whole bunch of things like that that you can do that's got nothing to do with mpi [Music] so is this almost like a operating kind of like a message bus where different clients can basically say this is what i'm able or willing to do exactly right uh one of the uh mantras we have in the pmix world is that pmix does nothing all it does is it communicates your request to the local system management stack and returns the response back so that that management stack always has the right to say you know nice i'm glad you asked that but i don't support it and you have to have your in your application some you know mechanism for dealing with a not supported response but but all the things you know people talk about with flexible workflows and the ability you know to manage their own environment better stuff we just provided the the hooks by which you can do that so is this done like a core plus extensions like if my client doesn't know about something like how it seems like you could keep adding more and more information that you could be announcing that you're capable of doing or communicating with how do you make sure that things can remain compatible between adding more and more things being mentioned or announced well we uh we adopted an architecture that says uh we have very very simple apis and select job controls where you announce for example that i am preemptable there's just one job control api and then you can provide a an array of key value attributes to that to that api that actually you know describe what the uh the operation that you want to do so you know if we want to extend the kind of things that you can do like for a job control i want to add something beyond just you know announcing that i'm preemptable or whatever we don't change the api we don't add another api we just give you a new key value attribute that you can pass and so that that's how we maintain backward compatibility uh we just we don't we have just a policy we do not add apis unless there simply is nothing that fits and then we'll add one so what exactly is the big deal here because way back at the beginning and again i'm i'm admittedly taking a limited world view of mpi applications but i understand this applies well beyond that as well but we used to just do ssh and pass things on the command line like if i want to start a 32 process or 128 process mpi job we could just do ssh and maybe do that a little smartly maybe via tree or something so that it wasn't linear why why do we need all this extra control stuff um of what used to be a relatively straightforward process well okay so there's there's two different parts to that well one is you know if you're looking at an mpi job which i know is is your your focus jeff the uh the problem is there's only so much you can pass on on a command line and so as your job gets bigger or as the amount of information that you want to pass gets larger you just can't fit it on the command line anymore and so you need some secondary mechanism for for making that uh for passing that information around what we used to do was uh was the pmi approach right which was to say well we'll take you know everybody will since we have a limit on on the command line we only use the command line for what we have to and then everybody simply broadcasts that and there's some out of band mechanism by which it gets exchanged and that has all the scaling issues and so that's why we went this way is saying well let's let's get back to that basic thing give a mechanism by which the resource manager can convey that information to you outside of the command line limitations okay so this is a way for applications to talk to the resource manager but how does that address that i'm process a and i want to communicate with process b and i therefore i need to know some kind of network address for process b how does that work so uh yeah so the way we did that in the past right was that process a would discover a network address he would broadcast it to everybody and process b would then receive that broadcast say oh okay i could communicate to you what we've done is that when the resource manager is getting ready to start the job we've given them an api a function they can call that will talk to the network and find out what are the addresses that are going to be used by the different processes what nodes are they going to be on and what addresses are they going to be assigned to and then we include that in the information that's given every process when it first starts up and so when the when the process first starts up process a starts up it can ask what's the address for process b and it already has that information there so is this something for end users or is this just something for like creators of mpi libraries like openmpi and other types of higher level tools that users interact with it's just between the creators of those middleware projects and the resource managers or is this something that a user can directly interact with yeah it's really both i i this so the libraries uh you know open in mpi libraries the openshm libraries etc uh they embed interactions uh that that use these pmix uh interfaces to do their basic you know wire up and and other operations that they do but then the application developers themselves are using it because they're the ones who know how they want you know their application has certain workflow requirements you know they want to be able to allocate more new resources or whatever they want to do they use it directly themselves and then you are seeing people embed it in tools so you know for example if i might have write this tool that allows me to launch jobs it says certain command line options and everything that i really like in the past you'd write that but it would be specific to the resource manager that you were locally using now you can write it with pmi x instead and take that same software and simply move it from one resource manager environment to another without having to change any of the code okay so this is really talking about just having a portable standard api that multiple resource managers and middleware developers can develop to correct okay so this does not necessarily replace something like let's say the task manager you know tm spawn inside something like one of the pbs derivatives out there because pmi x doesn't actually again do anything but now openmpi doesn't develop to tm spawn and all the other different interfaces out there they just developed a pmix that's exactly correct yeah openmpi calls pmix spawn and pmix takes care of the abstraction for it so just to nail this down then you're saying pmix is a library is that right it's actually three pieces uh first there's a there's an actual standard and uh and that just defines the apis and and uh and defines a set of attributes you know key value attributes uh that we all agreed that we would support or at least recognize but that says nothing in it at all about implementation and people are always free to implement it themselves the second piece is an actual reference implementation that's a complete library both client and server and that fully implements all the all the pmix function calls so again all it does is communicate so the client function calls communicate to the server function calls that then call the relevant backend resource manager functions uh to actually do something then the third piece is we actually have a reference server it's uh it's it's a runtime if you will it looks just like a resource manager that supports pmix with the exception that doesn't have a scheduler in it and so if somebody wants for example to develop some stuff pmix-based code in an environment that doesn't yet support pmix itself they can run this reference server in that environment and it operates just as if it was sitting in a pmix uh resource manager except for like i said it doesn't do scheduling so those three elements are what we mean when we talk about the pmix community okay so you're then also implying here that the resource managers out there are also supporting pimx directly is that a correct inference they are um some of them already do uh slurm for example has been doing it for uh for a year and a half now and uh ibm has been developing their job step manager uh and that that is you know really based on pimx uh our pmix all across the board and others are coming along at various stages so yeah eventually we we do hope that uh that most of the if not all of the resource managers out there will in fact provide that support directly now that's actually a a pretty fascinating place then that if applications have um one you know a standard that they can write to to do their large scale applications like mpi is one of the biggest but there are others out there and then also the runtimes have a standard to talk to the backend schedulers well that would also be fantastic because as you and i both know maintaining um you know the mpi implementation to talk to all these different resource managers who have different abstractions and different apis and whatnot it was kind of a nightmare so since this is a a fairly large disruption in this community how did you manage to pull this off well um the you know being having some role in openmpi obviously put me in contact with a lot of these people to begin with like you said we had to talk to them because of the infra interfaces we had to provide so there was some personal contact uh involved in that as well but the uh you know the real thing was that this was a need that all of us that were involved in these communities you know the resource manager folks etc we all knew we needed to do something and some of the resource manager folks had already been starting to write proprietary responses to it which was what was causing a lot of consternation in the user community because you kind of had to lock your application then into that environment which is you know something people really don't like to do so it was a recognized need out there and the real key thing i believe in that that led us get the adoption by the resource manager guys uh was the the stipulation that that the resource manager always has the right to say not supported so uh so you know if you don't want to for example say you don't want to support a particular backend capability you just provide a null in that function pointer and the pimx server if it gets a request for that will see the null on the back again and just return and not support it for you so you don't have to do anything uh if you if you want to support that cape that that function but maybe you don't support every option that somebody could pass well you have the right to be able to look at those options and say you know i'm sorry i don't support that and reply back so i think that not supported ability uh was the thing it was one of the key things so a second part of this inference uh of you know the the resource managers supporting pimx natively are they writing their own code uh their own implementation from scratch or are they using your reference implementation and then as a consequence of that is there a standardized network protocol that you use because then you can divorce the software implementation from what is communicated across the network so uh they're all free to write their own code so far nobody has done that and there's no indications that somebody wants to do that um the we did not we did not standardize the protocol between the client and the server so if somebody writes their own and it it's incompatible well that's you know you just have to make sure that you link against this the the same client library that they used you know they have to provide both a server and a client library but the apis would be the same to be compatible like i say so far nobody has gone that route it's a lot of code to write and nobody has seen a proprietary or a competitive advantage i should say in implementing their own the competitive what's happened is the competitive basis between the resource environments has shifted from the apis to what level you know which apis they support on the back end and which ones they don't okay so one of the goals of pmix is you know you've got exascale right in the name of it um what exactly does pmix bring to the table if it doesn't actually do anything that gets you that extreme scale well the it's really again in in two pieces if you will um the exascale uh requires uh that you be able to you know be first off you have to be able to launch the job in a reasonable amount of time if you if you just took the current method of broadcasting and sharing things you know next scale machine might take you know tens of minutes to start up a job that size and that's obviously that's something you really don't we would not like to see so um part of that is is the ability to then have the resource manager share the information that it already has i mean when we went back and looked at what was actually being broadcast around it turned out that more than 90 percent of that information the resource manager already knew but it didn't have a mechanism a standardized mechanism by which it could share it to the application so either the application had to come up with a you know a resource manager specific way of dealing with that communication or we had to standardize it so the library could be portable and so we took the approach of saying look the resource manager already knows a lot of this stuff let's go ahead and just give them a standardized way of communicating it then the second half of that was we had to go to the resource manager guys and say there's an additional 10 percent of the information that we would need and if we had that we wouldn't have to broadcast anything and we had to get them to agree to provide that ten percent and and uh that's we were able to do finally is to get get a a commitment from them and so we created a list it's on the web uh that has here's a list of all the stuff we need and the resource manager guys are going through and filling that list in so aren't you just moving like all the startup time from the mpi you know like runtime you know wire up whatever you guys call that very first step to get everything going aren't you just shifting that problem to the resource manager how you know normally it's not the individual nodes on you know resource whatever the resource manager has run into individual nodes it has a lot of this information right like how i i'm not seeing yet quite how this actually benefits if if all the information still just kind of held in one place on one part is just now into resource manager okay well let's let's take the uh the endpoint information as an example so uh the way it works today all right in the past has been that uh the application process starts it it it discovers a resource let's just let's use sockets as an example it opens a socket and gets a socket id uh socket number and then it has to broadcast that because none of its peers know what socket it's listening on so one way you could address that was be to assign static sockets to your processes and then you don't need to exchange the socket information anymore because you can compute what socket they're on but the problem is you might not be the only application running on that node and so you don't know what sockets you can actually take so what we now do is we say okay resource manager you use the pmix plug-in for the for fabric that plug-in will manage a pool of sockets based on its knowledge of what's being run across across the different jobs across different nodes and it will use that to assign static uh socket numbers for this app for this application those numbers then are included when the resource manager sends its launch message out to the to the compute nodes that information goes along with it so instead of having an exchange the daemons are given before they even start the political processes they are given all that endpoint information and they just simply convey it down to their local clients and that eliminates the need for each client to broadcast that information so if you go through and you look at what what pieces of information the different libraries are asking for you make that laundry list up and and then you ask the resource manager the workload manager when it sends the launch message out to the individual nodes you ask it to include all that information for every node in that message and so now you just use tcp sockets as an example and that's kind of a baseline but uh many more high-end hpc environments for example use other types of networking and there are several available so this i just want to clarify that tcp was just an example you can do the same thing regardless of the backend type of network right that's correct and we already do um support those at least at least all of the uh the most popular ones uh but basically yeah that's right you you know the the uh the network interface support in pmix and the server side is just a set of plugins so there are plugins you know for all your favorite flavors of fabric and we in those plug-ins we've worked with the network manufacturers to get those plug-ins available and those plug-ins are all now capable of creating those addresses for you and so just to drill down on this a little bit more so not only are you making the data available let me just provide one thing that was kind of an inference there was that when you say it's sent out to the daemons you mean it sent once to each uh let's just say node or server on the network even though that's kind of an amorphous term but like once and then if you've got 20 or 30 or 40 or more cores on that server that damon receives the information once and can locally give that information to all of the processes that start up whether it's 20 30 or 40 using local ipc not networking ipc right that's one of the wins that is one of the wins that's correct all right and then additionally on top of that you also do some compression types of techniques in the launch message that's sent via pimx right well the pimx doesn't send the launch message the resource manager does we just provide the information for them that we say look this information needs to go along uh here's a payload that has all the information that we need you to take along you'd be surprised the information is not as big as you might think our typical launch message uh our pmix-enabled launch message from like slurm for example is less than a megabyte because there's so much uh you know we compress obviously anything we're going to provide up to slurm slur just gets it as a as a blob and just sends it along but you know slurm use you know other and all the resource managers do they use like regular expressions to describe where the processes are located and things to try and keep the the launch message down so it really is only about a megabyte in size or so even for an exascale size machine ah that was that was my next question so a megabyte for how many like what what scale how many processes are you talking there well we've been launching uh in our in our biggest test cases uh a million processes on like 30 000 nodes and that that's about you know about one one and a half megabyte launch message so it's not very big [Music] so what about use cases besides mpi i know that's where this kind of started but you also talked about you know on cloud how you could say you're preemptable has there been any implementations um that kind of touch on one of those other examples uh yeah uh there are the um and i i apologize that i cannot i'm not at liberty to give you you know names and details but because those companies haven't taken it you know public yet but but there are people working on uh you know cloud uh interactions like we talked about earlier there are also people working on uh different kinds of tools you know that can take advantage of it you know debugger tools etc that can use these kind of interfaces to do more than they do today for example on a debugger today you get a a node level representation of where everything is and there's a limited amount of you know of information that you can provide because the interface is limited so but with a pmi x interface you could ask for example to show the nodes in a network-based layout where are they on the network relative to each other the processes i should say where they are relative on the network to each other you could ask the fabric for traffic reports and show where choke points are because the interfaces allow you to be able to make that query to the system management stack and be able to get that kind of information back now you said something a minute ago that i kind of want to dive into a little bit you said you were testing at a million processes how do you test at that scale well we have uh friendly users at uh who at facilities that have these kind of big machines and they will generally take a little bit of time out and run some tests for us which has been very much appreciated we also have ways to simulate scale so for example we can launch multiple processes on a given node and make them look like they're sitting on different nodes and so one of our our collaborators was kind enough to do that on amazon where he in fact takes a small number of amazon nodes but makes it look from a pmix standpoint like a much much bigger cluster and we can do some scaling tests on that that may not be you know fully uh realistic in terms of a cluster but gives some pretty good scaling law measurements um so we have ways of getting that information even when we can't get a hold of the big clusters so could pmix help in heterogeneous environments by this i mean what if i have you know hybrid systems are popping up all over the place where you have sometimes you have accelerators and nodes you might have fpgas are coming back on the market as alternative types and you know like for me at the university of michigan our cluster has different machines of different architecture types with different accelerators is this a way that a given application could basically optimize for whatever machine it landed on by asking the system do you have these things i know about yeah you could query what you know what resources are available to me there are there are coordination mechanisms in pmix that might allow you for example to say well if i i don't see or i have only four gpus on my machine how many of you got on yours oh well maybe then we use a coordinate we handshake and play you go ahead and run something over there so there are these mechanisms in place i i don't think that everybody that we have really fully understood everything we can do with pmix the community has been trying to come up with you know just trying to enable people to experiment with it and then the expectation is that people are going to do things with it that are beyond anything we had in mind but that feedback will come in and then we'll be able to perhaps offer better mechanisms for them to do some of those things but i i think this is we've never had this kind of capability before to be honest with you and we're kind of feeling our way at the moment to say well you know is this is this useful to you and how might you use it uh and and hopefully you know the community have a chance to try and test those things out so you had mentioned earlier too that you know you could ask the fabric you know how congested you are in different places and with more complex fabrics especially on the bigger system that's definitely useful but even on regular systems we might have you know in you know fabric islands or you might have a dual fat tree or something like that is there a way or is anybody doing anything where you could effectively choose say an optimized collective operation between hosts based on you know effectively the fabric architecture which again the application can find out about by asking through one of these standard interfaces i i don't think anybody has done that yet again i think that's one of those things that what we're looking to see is you know given the given people these tools that they can do that we expect researchers in particular to start asking those kind of questions and i believe that there are groups in fact that are starting to ask exactly that kind of a question about the collective optimization uh but i don't think they have published anything yet and this really only goes one direction right now and by that i mean if an application encounters an error condition that is could be system related uh think of like the pbs health check i can't use the pmi interface to send information back to the resource manager or to some sort of metric system or something like that uh well you can there's there's two ways that you can do that uh one is you could just raise an event and uh and we support complete binary uh payloads and we actually take care of heterogeneity for you so you can put binary numbers in there whatever you want to do and then the event when you raise it you have the ability to pass however much information you want in there so you could raise an event to the resource manager saying hey i saw something here's a complete blob of information about it um and then the resource manager can do something with that obviously you have to have some kind of a of a agreement with the resource manager that hey you're going to listen for that event and b you have some idea what you're going to do with it but uh but you also have the ability to uh to log so one of these people asked for was they said well you know if i see something or if i just want to even log that i have a certain amount of progress and i want to stick that in my job record so every resource manager keeps a record of the job that you could go back and look at see what happened you can actually insert messages into that log from the application and that way you know there's a when you get the official uh job record of what the job did you how much resources it used etc those messages will be there for you so you can record them for yourself or you can try and communicate them to the resource manager now one thing you've been very consistent about uh through this whole discussion is you've been saying a lot of we and the community this and the community that who is involved in the community well uh you know intel obviously through me and then there's uh melanox is with us on ibm uh those three are probably the biggest uh code contributors at the moment but you know we have a list of others you know fujitsu is involved folks from wrist we have livermore and los alamos from the national lab community that have been involved uh and then there's been you know slurm the sket md guys the altair pbs guys have been there so there's there's a and i'm probably leaving some folks out that i'll have to apologize to afterwards because there's about about 12 to 15 active members at this point in time and these are members who like you you mentioned uh the first several of them were code contributors to the the open source pimx code base itself but these other members are helping to shape the standard right i mean it's not just about contributing code right that's right that's right the uh every everybody participates all of those people participate in the uh in the standards process which is based on the ietf mechanism so there's always an rfc that that has to be you know backed up by a prototype implementation which is usually in the reference library it doesn't have to be but it has to be visible and then it goes through review it goes through uh uh comments and then uh we have weekly teleconferences and uh at those weekly teleconferences we schedule and and have a a review of those those proposals and then those things get accepted or rejected usually accepted into the standard so all those groups participate in that okay so this really is quite a bit more than just ralph's little toy this is a full-on community that is full of all the interested parties across this vertical right yeah yeah you know it really is a lot more than that than me and uh and i should give credit to everybody that's involved is it's a lot of work from everybody's standpoint you know part of it what you have to understand is that you know it's you can create this channel by which these two parties can communicate but there's a lot of work behind the scenes that has to happen that the resource manager guys have to agree they're going to provide and then like for example you have to go to the fabric people and say hey guys we need you to provide this this endpoint information blob etc that's work on their part that they have to do to provide that information or if you're going to talk to the file system guys and you say well you know we need you to be able to pre-cache files for us when we tell you or tell us how long it's going to take them to retreat to be retrieved that's work they have to do so it's really a collaboration across all these different elements that makes it possible so it seems very flexible what's the strangest or probably in this case more unexpected use of pmix you've seen so far uh the one that surprised me the most was a request really from the uh from the cloud some people working in the cloud world where they wanted to be able to loan i think i missed this earlier they want to be able to loan resources back so uh they actually have workflows where they they you know they need a an envelope of resources uh eventually at some points in their computation or their work but there are times when they don't need it don't need all of it and you know so i i had never anticipated somebody actually loaning resources back to the system and getting them back later that that to me was a surprise that last point was one i really wanted to get across it really involves all these different parties actively collaborating because that's what's really different in my mind with pimx versus before is it's the first time i'm aware at least of you know the resource manager the fabric the file system the library guys language library guys all getting on a weekly telecon collaborating on how they're going to orchestrate this application environment all right so there's a lot of open source code here um you must have a fairly permissive license that you distribute this under because some of these resource managers are closed source and proprietary what what license are you using we use the three clause bsd so it's uh it's absorbable by people using you know gnu license as as well as you know proprietary people they're all welcome to use it so uh ralph thanks a lot for your time where can people find out more about pmix and get involved the probably the first place to start would be going to github everything's on github for pmi x so the uh the reference page for for pmx itself is pmix.github.io pmix or you can go to the code repository itself uh there's a group because we have both the uh implementation as well as the uh reference server there and that's at github.com pmix okay ralph thanks a lot for your time thanks ralph yeah thank you very much for having me you