RCE 115: PBS Professional

[Music] welcome to another edition of rce again this is brock palin you can find us online at rce-cast.com um yeah you can find links to our twitters our blogs and all the back catalog of over 100 episodes on scientific computing i've again here jeff squares from cisco systems and one of the authors of openmpi jeff thanks again for your time hey brock so we are recording here just about on the eve of the eclipse i think this episode will probably be published after that so everybody will have seen super exciting totalities including me and including our guest today yeah and an office full of tech people you can only guess that you know around half of our office is taking that day off to go do something related to the eclipse so yeah it's actually an excuse to absence it at my kid's school so we just have to say we're going to the eclipse and that's good enough that's that's actually not a bad idea it's like the one time to be able to see it and now where you are it's much closer to where you are than where i am up here in michigan yeah we we just drive two hours south and it's right here in bowling green kentucky so we're just gonna drive down on the highway pull off on some country road style and like picnic style spread out and bask in the glory of the totality so we have with us the cto from pbs works at altair uh you want to go ahead and give us an introduction hi this is bill nisberg really happy to be here with you guys um i got my start actually back at nasa where we developed the pbs pro software uh now i'm the cto for pbs works at altair so uh can you give us a quick rundown of what is pbs pro sure pbs pro is a software you know it's funny i always um end up at these conferences where i can't tell whether someone's selling software hardware or services so i always start with that it's software it's middleware so it's operating system little level middleware that does job scheduling and workload management um if you're in hpc i don't have to say anymore if you're not in hpc probably should say a little more do you want to be in hpc today yeah well you said one little phrase there let me ask you you said it's operating system level middleware what do you mean by that phrase um so i i sort of think of software as as three layers so there's well there's the hardware you know people know what that is you go buy it um there's the operating system everybody knows what those are hopefully uh and then there's applications that you use like gmail or you know like some uh computational fluid dynamics application that's that's blowing air over airplane wings and then in between all of that stuff in between the applications and the operating system is this stuff called middleware and it does you know the extra stuff the operating system doesn't do okay so then what is pbs pro's relation to that you said it's it's middleware let's pretend we're not in hpc today so that we give our non-hpc friends a little overview here okay awesome so um pbs pro is software that you use on large hpc systems so clusters or clouds where you're connecting together lots of different nodes together lots of different computers together and the simple version is that it sort of does traffic control um scheduling so what happens in high performance computing is a lot of the work is super computationally intensive so it uses a lot of compute power and it tends it ends up being cut up into little pieces and handed off into the system in what i would call jobs and then there's also a lot of people say competing for that resource and so there's a lot of jobs from a lot of people and pbs pro does the traffic control the scheduling the the watching the system to make sure it's still running um does that make sense yeah it does but clarify something this doesn't serve a role in terms of you like your programming api right this is managing the resources but not serving a role like mpi or co-array or something like that right uh that's right so we take pre effectively pre-packaged what a job is to us is sort of a pre-packaged thing that you might run on your own from the command line so you might run say some structural optimization code say you ran op destruct to you know test the the flexibility of your cell phone display or something and you could run that for the command line you'd say blah blah blah blah blah and it would virtually crash test your cell phone into some cement but it might take three or four hours so and you might not want to just drop it once virtually you might want to drop it from a lot of different angles a lot of different heights and so that ends up being say a thousand different jobs that you might want to run and so instead of running those thousand different jobs by hand one at a time what you would do is you would package them up into jobs and you would submit them to pbs so pbs is not the api layer it's it's really at the application layer and it's it's very agnostic to application so i just used optis as an example but it's it's anything that you could run basically on an hpc system okay now pbs in itself has a long history and you even referred to it uh in the beginning back there um give us a little bit about the history and the evolution of where this came from ah well um actually there was a system back in the very early super mainframe computer days uh in the sort of the cray days called nqs i think it stood for network queuing system and the group actually the group at nasa ames even before my time created nqs and as parallel computers were sort of looking looming on the horizon the folks who were developing nqs i believe ran out of space in some internal data structure and realized it was time to actually start from scratch so this was back in 1991 i think and actually started a posix standards effort to standardize what batch computing looked like um and out of that came a posix batch standard for batch computing but also came a reference implementation which became pbs and then became commercial pbs and now is open source pbs okay so uh scheduling seems relatively straightforward i have resources i lay stuff out that's a very naive look at it what makes scheduling difficult that's a great question you know um i think if you're just looking at your own problem so if you're say one organization um and and you just have one set of of users who are competing for one machine scheduling actually isn't that hard and a lot of people actually continuous continue to sort of build their own scheduling system or even just do something simple and you know yell over the partitions and say hey it's my turn do you mind if i run some big job what makes it hard is once you go from sort of a group of people who can yell over partitions to you know people who are all coming in remotely or you go from say a few hundred jobs a week to a few thousand or you go from a few tens of machines to a few hundred thousand machines um just the combinatorial explosion makes it hard and then hpc uh is full of really strange and unique things like um you know nodes without disks or nodes with xeon phi or gp gpus or networks that have a topology that you might want to map something to because you get better performance if you have a better mapping i mean all of that plays into scheduling okay so that was a good history but uh what about uh torque wasn't torque one of the open source derivatives of this somewhere along the line as well uh yes it was actually i mean here's the here's the whole the the family tree was uh first it started out as the portable batch system uh which got abbreviated pbs um in 2000 um actually i and the group of original developers left nasa to form a commercial company around pbs and we um coined pbs pro at that point and also opened pbs a few years later a fork of open pbs became torque i think torque is still alive people are i'm actually still bumping into it so turkey is still alive um openpbs kind of died um we kept going with pbs pro our group was acquired by altair in 2003 and then just last year at international supercomputing we dual licensed pbs pro so that now pbs pro is is there's an open source version and there's a commercial version oh and what are the differences uh between the two ah our goal is actually to try to eliminate um as much as possible the technical differences between pbs pro and pbs pro in fact we we decided to choose the same name instead of resurrecting the open pbs name because our strategy is um to try to kind of bring the two worlds together that have formed around some some using open source workload management job schedulers and some using commercial uh workload management job schedules and we really wanted to try to bring those two worlds together um the i mean that obviously we we have a commercial version and so the how we how we differentiate on them is that the commercial version kind of like i guess the way to think about it is um more along the lines of say um the way red hat does say fedora and red hat enterprise linux right so you you get all the features exist in both um although actually you get tons more features in fedora you just get tons more bugs and if you want something you know if you want something stable and supported and that you get regular updates you go with the red hat enterprise linux so that that's sort of the philosophy we're trying to try and do that makes sense so how has scheduling changed over the years because that's not how hpc started um you know well there were many different form factors of hpc but um a typical one that most people think of is you know small clusters of linux machines back in those days it was pentiums and the like and whatnot but now we have entirely different form factors like you talked about with gp gpus and xeon phi and their thousands and thousands of cores and remote users what is your perspective and how has the software had to evolve to handle these kinds of scenarios so i mean so scheduling has gotten well actually i think scheduling's gotten more interesting of late um in the in my early days which is too many years ago to talk about um scheduling was pretty easy there were cpus and memory and that's about it and and you just had to do some some counting you just had to say how many cpus are allocated okay don't allocate anymore um then we actually got parallel machines which made things interesting um in fact we got some weird parallel machines we got this cm connection machine and the intel paragon and ipsc860 and stuff and it became kind of you know all over the place then it went to sort of these white box linux clusters where everything kind of looks looked the same for a while um a little bit harder than just cpu's memory because now you had nodes and nodes have cpus in memory so that made it a little harder and but today boy it's it's fun there's gpus there's xeon phis there's fpgas there's multiple different kinds of network architecture and actually one of the more interesting things that we're doing is scheduling not just based on sort of matching and use of things like licenses that you buy once and you use for a year but power where you know if you turn it on it uses power and that costs money and when you turn it off it stops using power it doesn't cost money uh you guys doing anything with ephemeral resources this i'm specifically thinking cloud um ways that you can burst or run dedicated what are you guys doing in that space um we are actually it's a really nice um uh yeah analysis it's very close scheduling for cloud is really close to scheduling for power which is really neat right so when you turn on a cloud resource it costs you money when you turn it off it it saves you know you don't pay money uh if you bought a machine when you turn it off it saves you money when you turn it on it costs you money um so the the scheduling problem is really um very very similar which is great um and so yeah so we're having a lot of we're doing a lot of stuff with cloud i mean you can also a cloud you can build a cluster in the cloud and so pbs pro works really well for for managing resources in the cloud if you want to share them a lot of people just want to use them for one job and then turn it turn it off if you want to just use it for one job and turn it off then we have a lot of folks who are doing what's called sort of hybrid cloud hybrid computing or cloud bursting where maybe you have a cluster inside your organization and for peak demand or for you know particular kinds of applications uh you you send them to some cloud where it's either cheaper or faster to get the results and for that you know pbs pro can automatically connect to say azure or aws create a small cluster for the job run the job tear it down send the results back and that's that's what cloud bursting is is uh pbs pro also handling things like data staging and other things because normally your shared file system scratch uh you know reference genome set stuff like that don't also span across the land or slow across the land does it handle that kind of logic so it can i mean here's this is an interesting um issue problem uh in hpc i think in general and in hybrid computing so the quick answer is yes of course pbs you know can move data around that's not that's not an issue you can say hey before you start my job please stage this data in or please stage this data out um when you're done with the job um the big problem is actually a lot of these applications have a lot of data and it doesn't make sense to stage it in or certainly well staging is usually free for some clouds but staging it out can be actually pretty expensive um and so um people who are designing their cloud strategies i don't think it's a solved problem yet they're sort of trying to decide what they want to do so if you have a huge genome database and you want to do cloud computing you're better off storing it you know you're using amazon or something you're better off storing it in amazon than you are trying to access it or copy it in and out now i want to jump back to something that you mentioned uh earlier you said that you know you have the open source version and the closed source version what drove you to what was the rationale behind that decision um because you see a lot of impassioned arguments on both sides of this coin of the open source or the proprietary and things like that and you have explicitly and deliberately made a choice to well not fork because you're trying to reduce the technology differences between the two but you've made the choice to be on on both sides uh why why did you do that well um it was actually a hard decision actually if you if you uh got a view into the internals of of of alter you could see that we had a lot of a lot of long discussions over it but um in the end here's here's what we here's how we view things which is that so in my tenure in hpc i don't know i'm curious what you guys think about this but there's always been especially in workload management these two worlds the the public sector um universities in research and the private sector like you know fortune 500 companies and the public sector loves open source and the private sector it sounds like they hate open source but they actually just want to get they just want tools that work with support and stuff and so they gravitate towards commercial software at least in workload management that meant um that nothing could reach critical mass or nothing has i don't know if it could maybe that's the wrong way to say it nothing has reached critical mass right i mean look at the open source um progression of of tools in workload management uh so nqs was actually open source back in the 70s okay that was before open source even had a name but that was that um you know as that fizzled out pbs actually became pretty well known then torque became pretty well known then grid engine actually took over uh condor for a while um and now slurm is the big thing in the open source side but i'm not i don't i don't feel like anything's ever been able to take hold and stay and and my view on that is because nothing's been able to bridge the gap over into the commercial side and then reach critical mass so our our idea anyway i hope it works uh is that if we can have a dual licensed tool that plays really well in the open source world in the public sector world and also plays really well in the private sector commercial world maybe we can bridge that gap and reach critical mass and how's that been going because i would tend to generally agree with your assessment of open source versus just give me a tool that works um for the different audiences and things like that are you seeing good adoption on both sides of that um yeah we've had a i mean we've we've had a long time we've spent a long time in the commercial world so we have really good commercial set of customers couple thousand commercial customers all over the world doing all sorts of neat stuff um and in the last uh year and you know it's really hard to count um open source um users as as you probably know with openmpi you sort of have to do statistical analysis or see how many people show up in above um but i do know how many people are sort of contributing on the open source pages and stuff and we have about a hundred um sort of active contributors right now and okay look uh to be fair you know 30 40 of those are from altair still uh maybe even 50. i don't know actually it's hard to count who's from altair because everybody's using their github id not their alter id um so it's it's actually going very well um so far well you actually anticipated my next question was what exactly is your definition of open source are you just throwing code over the wall or are you actually building a community and it sounds like you're trying to at least build a community and you are having success in having people actually contribute new code right yeah yeah no the the goal is is to create uh one one pbs that everybody likes um one pbs pro i should say that everybody likes and and the only way to do that is to be sort of very aggressively open on the open source side um and then you know very nice and supportive on the commercial side and so we're trying to do both um and so whenever we're making in fact some of the decisions that we've made um we've made to try to make the open source um more open so inside altair we actually as part of the whole effort we reorganized ourselves to act as a contributor so if you if you go to the if you go to pbspro.org which is where everybody should go you can look at the show notes i'm sure for that you'll see and you wander around on the various open source sites uh you'll see that we're acting as one contributor and we're following all the rules that are posted and and you know people are even even people internally are complaining to other people internally saying i'm not sure you can do that because the community might not like that so it's actually kind of nice so support models i mean you covered a lot of that but here's an interesting question because it's now open source it also has the ability for a person who wants a support contract to also modify it locally and not contribute it back uh have you ran into that situation and if so how do you handle supporting something that's been modified non-configuration of actual source code modification by the uh by the person who wants to buy support yeah no that's a it's a it's something we struggle with a little bit i will say that we've been really lucky um even with our closed source version for many years we actually had a lot of source code customers for example nasa ames as a source code customer and they wanted a lot of support and we supported them but they modified the cac out of the code i would say that right now we haven't run into that problem and we've been really lucky of the people who've been modifying things that they're um pretty intelligent smart folks and you know maybe pbs pro only attracts really smart people and and you should if you're really smart you should use pbs pro that's totally what it is that's that's yes make sure all the listeners know that yes exactly really it's totally it's fun with gold thread well let me ask you this then so if you are just a contributor does that mean that there is some third-party organization that holds the code or are you just treating yourself like what what is the license that this code is under including the stuff that you get from contributors okay so the the we're using it's dual licensed so it's a regular commercial license that you'd expect from commercial licensing um it's available in that and then the open source stuff that you download on github is under a gpl um version three license it's not held so right now altair still holds all of the intellectual property obviously though it's agpl so you can get a copy we did that because we really want the dual license to work we really want to be able to take um the innovations that happen in the open source community and move them over the commercial side and vice versa move commercial to open source we also did that because it's really new for us and we're learning and we didn't want to you know take some path that we couldn't undo um in terms of like you know if we went into say the apache foundation we decided oh no the linux foundation's where we should have gone ah so we wanted to sort of dip our toe in um before we before we dive in and what kind of response have you gotten from the community you said you've got uh upwards of 50 external contributors or so um what's what kind of things are they submitting are they submitting just oh here's a little bug fix where you have a typo or are they submitting like genuine new features or how's that going um a little bit of each i mean actually we we released it at international supercomputing not this year but last year so a little over a year ago and within three days somebody submitted a port to i think it was debian um because it didn't compile and debian so they they submitted a pull request for that we're like wow cool okay that was fast um but we've also gotten uh a handful of bug fixes um a couple of new features that actually make it a little easier to configure pbs um we actually got a huge dump of kerberos code um but then that was withdrawn because the person who contributed actually moved from one organization to another i don't think they had the support to finish the process which was a little disappointing um but we're hoping we're hoping that the original um organization you know hire someone new and then they come back but that was like 30 000 lines of code it was crazy we were a little worried about that because we're like we have to okay the right now the the you know we're trying to structure this like a real open source project so there's a set of maintainers right now they're all tear hopefully one day we can expand that and they review all the code before it actually gets before the pull requests or get accepted and we're like 30 000 lines oh someone's gonna have to review that so since you've gone open source is there somebody who is like a household name that picked up the open source version um wasn't a customer before using the commercial pbs works version like somebody new who somebody would know i wish i could give you a better answer that um the the biggest contributor outside of altair right now is uh uh cessnet ces net in in chechia um the the people using it we we i did see someone come by or one of the guys um at isc this year saw quantum the the big manufacturing company come by and take pictures of our booth and they asked them hey what are you doing and they said oh well we use pbs we use the open source version of pbs pro like okay i don't know how you find out that people are doing that but but that's cool so um uh you know i i think i think we're we're spending all of our time on technology and a little less of our time on on marketing the open source and i think i'm really glad that you guys set this opportunity up for for us because i think um people still don't know about about the fact that pbs pro is available open source in some sense because i do bump that about i would say about now half the people i bump into say oh it's open source now really okay open source or not uh what's the you don't have to name any names because i know there's a lot of companies that are you know protective of how much resources they devote towards high performance computing but what's the largest system let's go by core count um managed by pbs pro uh well that's easy the the biggest system is still nasa ames pleiades the one you saw in the martian if you got that little tiny bit that they scrolled by i thought that was great um the movie that is uh they have uh you know they keep changing their machine because because it's a it's a whole bunch of well now it's hpe hardware and they keep rolling in new hardware so um when i looked a couple years ago there was 12 000 nodes and they were scheduling 250 000 cores some of which were virtual as one system i think they're actually down to to like 10 000 nodes but up to more cores because they replace some of the old stuff with higher core count new stuff so in the 10 000 node range 250 000 plus cores which is pretty big okay and then what's the strangest use you've ever seen um we've had a lot of people on here something they never expected to happen what have you seen someone tried to do you know i you guys um nicely sent me some some of these questions beforehand and and this is the one what is the strangest use of pbs pro that i i've been racking my brain about um because most of the folks are using pbs to do scheduling and it's not too strange i have seen a lot of strange uses of features of pbs like we'll design a feature of pbs to do one thing and somebody else will will use it to do something else i was at a weather site where we we had expected for example them to use advanced reservations to schedule their weather models right you know the weather models run like say four times a day every six hours so we have this advanced reservation feature where you can set aside some resources to run that model and you're sure they'll be there they didn't like the reservations the way we designed them so they were using them in this really weird way where they were making reservations that were like 24 hours out and then the way the scheduler behaved is it would make sure that that reservation was available later and that created a hole in their system for backfilling that they used for something completely different so so people use features all over the way that we never expect but i don't know that the whole software suite has been used in a in a surprising way or at least not yet so let me rip off of that and and uh go off something you said earlier in the conversation that scheduling has gotten a whole lot more interesting recently with new architectures and new topologies and all kinds of things like this what is your most interesting feature or or your your favorite feature from that perspective that is just needed for the new and growing evolving complex hpc scenarios um i'm actually really uh excited about some of the stuff that we're doing in power i think in part because i think it's really early days in power management um in in terms of sort of limiting power running for power i mean if you look at so one of our bigger systems in japan was the tsubame 2.5 system which was all you know big green system run by tokyo institute technology um and they had this real big problem which was uh they were only allotted 0.8 megawatts for their system and so they had to run it you know with a power cap and so it didn't it didn't matter how many cpus are using how much memory do you need you know how would this they also had this extra um almost orthogonal issue that they had to manage at the same time and so i think what's making i mean what's making scheduling interesting and what what's what's cool inside of pbs now is okay now how do you take how do you take these different things that you're trying to do so you're trying to maximize utilization you're trying to you know get turnaround time as as short as possible you know maximize throughput but now you also have to minimize power use while still you know making people happy on those other dimensions and so there's just there's so many dimensions now to play with that it's it's become a really interesting problem and a really interesting system so how do you play with that though do you just power machines down or do you power cores down or do you change sea states i mean what kind of things can you do um all of that but we try to stay out of we try to stay out of exactly what we do and we try to separate out sort of some the policies of what people want to do from the mechanisms of how they do them so uh you know pbs is really good at counting so for example if you want to keep under a threshold so i'm under some power cap you can allocate power and then you know jobs can ask for a certain amount of of well okay it wouldn't be uh yeah it would be power sorry i keep getting mixed up with power and energy i'll let i'll let the listeners look those two up if they're not sure what they mean but even i get mixed up and i've been in this world for a long time so but to to cap power you know jobs can ask for a certain amount of power and then um we're good at counting so we can make sure we won't go over that power limit um but if you also want to say run some low priority job at a lower power you know we have facilities to do that um how you set that lower power whether it's a c state or frequency or or what you're doing that's sort of a mechanism and and we let you plug in you know whatever you want to plug in to to pick the right mechanism for you we also have a facility for looking for when nodes become idle so if your cluster isn't 100 used all the time you know maybe you have you know sunday nights it's not used or something we can go automatically go out find nodes that are are going to be idle for a while and shut them down or just put them into a low power state again that the mechanism of how it reserves power is kind of up to you um some of it is what we're doing some of it is what we're rolling out one of the so we're i think i would say we're still in the process of moving a lot of stuff that only exists in the commercial code over into the open source project and a lot of this stuff is in the commercial code because we did it in concert with for example sgi when they existed as a separate company or cray and so now we're we're in the process of rolling that out and figuring out how to make it general and acceptable to a wider community and putting it in the open source project so bill besides these power things uh what other new features are coming in the future for pbs pro um we're doing a lot sort of in the the exascale space um meaning um connecting to other things in the ecosystem um but also scalability a lot of scalability stuff so we have a sort of um i don't want to call it a prototype because it's running in production but we have some uh really great throughput code that takes pbs current version of pbs up to ten thousand jobs a minute throughput and and throughput um and we're gonna roll that into uh the open source project and then into the commercial generally available code we have a another feature we did only that is only exists right now on cray systems that lets you run schedulers multiple schedulers in parallel so you take one pbs system so it still has one database and sort of when you do a status you get one status and everything but you can cut the system up and run multiple schedulers with different policies for different parts of the system that and that also is both speeds things up and then also gives you a little more flexibility about how you set scheduling policies so exit scaling so those are the exaskelly stuff that aren't power so i already talked about some of the power stuff um in the other part of our our world we also have some some non-open source stuff i don't know how much about the rest of pbs i should talk about but with pbs works we've focused a lot on the user experience and we're just about to roll out i think in the next couple weeks in a beta version of of the next version of pbs works which will be generally available in the coming months which is really tuned toward listen with a real focus on user experience so a real focus on you know for engineers um being able to just do what engineers want to do without paying any attention to hpc behind the scenes um or real focus for assist administrators to sort of just just get their job done like just get a snapshot hey is things working things are working great i can go away or oh something's not working let me click on that red thing and see what's not working and fix it so along the same lines there has been a bunch of resumed discussion over the past i'd say even two years or so about mpi integration with job scheduling systems and it seems to have gotten deeper and broader uh over the past couple of years is this something that the pbs pro community is working on looking into talking about any of these kinds of things um definitely talking about we actually started working on some fast job launch stuff by working on i don't mean writing code yet but sort of doing design work and and figuring out what we want to do um with the uh pmix community um i mean the idea is uh better i mean the idea will with mpi is so pbs picks a bunch of nodes to run your job on say it picks a thousand nodes um it knows what they are um and then it and then it hands control over to your job if your job happens to be running mpi the first thing mpi does is goes well what thousand nodes would i given and let me find out about them and and that takes a little bit of time and so the kind of integrations that we're trying to do now and this is where pmix comes in is is eliminating that dual uh find out about the universe and just letting pbs pass down to mpi hey here's what the universe that you're given looks like you don't need to find out again so maybe you can take a little less time on startup so so that's some things there's actually other other things in mpi and we've only talked to people and we on the commercial side anyway we get driven by uh what people want to do you know next and not so much why what people might want to do 10 years from now i'm hoping actually on open source side we get some more of the researchy stuff coming in from that but we don't the the discussions we've had about um you know growing and shrinking jobs um in fact we have some code that we put in that we're actually i didn't talk about that we're also putting into pbs to shrink shrink jobs but also uh you know handling resilience so you know your 10 000 way mpi job loses loses 10 nodes you don't want to kill the whole thing and start over it's been running for a week you know how do you handle that so we're we're dabbling with that but i don't think we've we're not we're not to the point where we're writing code or there's an obvious solution okay bill uh where can people find more about pbs pro and pbs works um so pbs the the s place you should remember for pbs pro uh the open source project is pbs pro dot o-r-g uh and that has pointers to all the other sites um because there are a few other sites like the community bullet board and the contributors portal and the issue tracking system which are all over all over the web in github so pbspro.org for pbs works you should go to pbsworks.com okay well thank you very much for spending your time with us thanks bill uh thank you and just you know please uh go take a look go sign up for the announcement list at pbspro [Music] you