EasyBuild Tech Talks VIII - AVX10 for HPC, A reasonable solution to the 7 levels of AVX-512 folly

okay we're good to start um welcome everybody to the E easybuild Tech talk um it's been a while since we've had one but this this one in particular should be a very interesting talk for a whole bunch of reasons um we have Felix lir today who will be talking about ax10 for HPC so floor is yours Felix thanks um so ASM for HPC with the loose title a reasonable solution for the seven levels of ABX 512 falling you're going to detect a little bit of a theme of light humor a little bit of sarcasm here and there but fundamentally we're sticking to the world of x86 today so x86 64 that's Intel xeons that's AMD epics and a little bit of consumer platforms we're going to be taking a historical view as well going all the way back to the 70s uh for which some of the people in the call Yours Truly included weren't even around and then building our way up from all those Legacy instruction sets to understand where we came from for why we are where we are today and then along those lines there's going to be an understanding of what are we covering where we came from but also why we got to where we are in terms of the way it's implemented right because one of the things we're going to be talking about a lot today is what's an Isa so what's an instruction set architecture what's the what's an extension to that and then why we got what we because a specification without an implementation is just a piece of paper that you can throw and Chuck in the bin kind of like a paper that no one ever read doesn't actually move much of anything forward so the first question is what is an instruction set architecture I've got a I I asked around uh to a lot of folks some of which are more private some of which are more public some of which designed ABX 512 and I got a bunch of different answers but one of the things that you're going to see clearly in all of these kind of question quotes is it's the language that at the fundamental level your computer interprets we read it as Assembly Language in a sense but fundamentally it's the instructions and how those are specified So when you say I want to do a plus y well you need an instruction to do that you also need to say okay what registers is though are those in if you're doing floating point you need to Define what the rounding behavior is but fundamentally it's the language of the chips right and as I is a mapping between integers so what you see as an OP code or an operation code the hex and binaries of computers two things that programmers can understand it's a vocabulary um I thought one of these was particularly cute and Isa is like a musical note it makes an orchestra of transistors perform a symphony of compute then uh Andrew Richards uh CEO of code playay um Now by Intel put it particularly well as well where it was is a contract between Hardware and software designers to let us work together uh this is supposed to be Steve Canon of Apple for higher level programmers and Isa is the stable API for the CPU and then this is a favorite Tom foresy uh unofficially the father of AVX 512 put it best and Isa is the train line to which Hardware designers chain their screaming future colleagues to and that's because fundamentally when you have an Isa when you have an instruction set your man dating that all your chips implement it but once an implementation is out in the wild once you have those instructions that's software you can't change all of your software all the time right so once it's in your chips have to support it so if you mess up during the design phase oh you're just stuck with it you just have to accept it and move on so that's kind of our intro section one is when we had isas from the 70s and the original 8088 the original 886 and there you had floating Point numbers or simd right and we'll get to more of what simd is as we move on but floating point I think everyone here is familiar so talking about the original Isa of x86 it's not going to solve all the problems of today right something designed in 1978 trying to look forward to the what's going to be useful in the before the IBM PC even existed that's what the 8086 did right it was a small 16bit microcomputer with an which used an internal 8bit bus to save on costs right and the parallels with that are similar for HPC type things to fortran77 in a lot of ways where fortran77 when it came out was great mind-bogglingly Advanced right did a lot of really good things for the time but ask someone today to write fortran77 instead of of say Fortran 2018 or something like that they're going to look at you like you have three heads like no it's not happening so the original 886 and the 888 which were the test more of a small micro Eda chip and then finally the 888 which became the ibmc they didn't have support for floating point they just didn't so when you if and when you went to do any maths at that point whenever you needed to do any division algorithms well that algorithm is N squared and half fun right so we're going to be doing looking at some code today you don't need to have written a lot of C you don't need to have written a lot of assembly to understand what's actually going on here so what we're doing is everyone who's written a lot of HPC codes or even a little bit of HPC code at the high level we're going to be looking at how do you implement a basic ax plus Y in double Precision function so the idea is so if you read it perform Dax spe double Precision ax plus y so function to add the elements of two arrays we have a function it's C C++ but everyone should be able to understand it we're taking in an integer n so how many things we're doing and then basically we have three arrays and then we're going to Loop through them so do while I smaller than n and we're going to look at that and say that Z is going to be updated to have the value of X+ y multiplied by Z right a simple ax plus Y and then int is just like the the main is stuff that you do in CS 101 class one now if you look at the actual code here in terms of the assembly you're going to see a lot of moves and pushes and calls and so on and then a few ads and a few movement instructions and essentially what's going on in that code and you don't need to read all of it the important part to get here is that we're having to simulate and loop through all our floating Point operations because we're dealing with floating Point numbers and to the surprise of no one in HPC we actually care about the decimal point for which floating point is a really good idea so realizing this in 1980 we got a new kind of it wasn't at that point it wasn't an Isa it was actually a co-processor called the 8087 or lovingly refer to these days as the x87 and we got floating point on x86 before it was called x86 so that co-processor that x87 supported three major data types it supported fp32 or single Precision floats double precision and fp64 and it also supported this weird fp8 type which never went somewhere but that we're still stuck with today because as is the theme it's it existed and software used it because it was useful at the time so so now we have to live with it and one of the things that's one of the things that happened with the 8087 is that under the hood even if you said oh use an fp32 and you're only using 32 bits of your 80 bit register it actually would extend that fp32 to an 80 bit which means that if you go to compare a 32-bit float addition today to the ones that you got from the 8087 you'll actually get different results because the 8087 was doing everything in 80 bit floating Point Precision right so you'll actually get different rounding Behavior different offset behavior and so on but one of the things that the 87 did really well was it it inspired what would become the Bible of floating point for the modern era right the deao specification is i e 754 uh then it was just i e 754 now it's i e 754 1985 and then there was a 854 87 spec for decimals but the long and short of it is if you have binary representation of floating Point numbers it uses I 754 as amended so here's where things get kind of cool on the left you'll have the same code as before and now you'll see all these little F loads or F ads and F M multiplies right and F store pointers fstp is f store pointer and the idea here is you're looking at of hey with a single instruction I can do my ad I can do my I multiply I can do my loads in my stores I'm not having to extend I'm not having to do any shifting Behavior not having to dissect parts of my floating point for what part of it is the exponent what part of it is the manessa so on and so forth and then you're just looping through that and it's significantly fast orders of magnitude faster in fact now at that point we were doing one instruction at a time and one data type processing at a time remember that part I said where fp32 you you would use that in a whole 80 bit register so your entire floating Point Unit you have 80 bits but you're only doing 32 bits worth of compute at a time it's not really efficient those days we had other ideas for how to get around that but it wasn't as popular con concept then comes along Intel in 1997 so 17 years later basically half of people's careers and they introduced this thing called MMX or the multimedia extensions and the idea here is that you have four data types in 64 to in32 H 2x something why is why would that be the idea was we're going to use the register file the registers from the 887 but what we're going to do is we're going to put multiple D pieces of data in those registers at the same time so let's say that you have a loop that's doing the same thing to multiple pieces of data so you could say that you're doing the same instructions at at once but then you're doing multiple data points at the same time and so long as say you're looking at a linear Loop and expanding as HPC often does for when you're looping through massive arrays of data you don't have a data dependency between entry one and entry two in the same array right so what you can do is have one instruction compute multiple of those at the same time now on with MMX being multimedia focused and especially back in 1997 and them wanting to reuse the 80 bit registers of the 8087 fundamentally no floating Point okay we use the same register we can inter leave some floating point and some integer operations but you're not doing all much so you have to choose are you doing simd or are you doing floating point and then here if you chip supported MMX well MMX doesn't support 64-bit floating point so you end up falling back on the 8087 floating Point Unit so your code looks exactly the same right it's identical so even though your chip is 17 years newer or even newer than that you're still relying on the floating Point Behavior from nearly two decades ago your chipped a lot faster because at that time we were just adding larger floating Point units we were getting more chips and running them significantly higher clocks but fundamentally your assembly your actual code your actual Ops operations are doing the same thing little bit of a Spider-Man beam in there because your code is identical because you can't use MMX to do any of your Dax P operations well that's a little bit silly so so what if we did floating point and simd at the same time now you're cooking right then we get ss1 in 1999 the streaming SD extensions because we love having nested acronyms within everything ss1 however only supported one data type it added new registers and they were 128 bits wide but fundamentally it only supported 32bit floats it they were I e754 32bit floats but for HPC especially then when we weren't as worried about memory bandwidth and going to nor to lower types because of that it just wasn't all that handy right you could do some algorithms to use fp32 simd to approximate fp30 or FP 64s and shift between them but honestly it just wasn't worth the headache at that point once again historically when you were looking at chips we were doubling the frequency every few years right mors law was well and truly active at that point it made more sense to just wait a year and expand your cluster than it did to ah we're going to do some really weird assembly stuff but then the year 2000 the New Millennium we get SS 2 streaming cyy extensions 2 and we finally have real general purpose Cindy overall SSC now supports two FP 64s at a time we're doubling the size of of the floating Point Unit and we're also because we're no longer relying on the 8087 we're never falling out of specification because of rounding behavior of the 8 bit internal floating point we now have consistent reliable 64-bit floats across our chips that implement this and then of course you have because the idea was same instruction multiple data they also make it more general purpose so yeah we have our four FP 32s from ssc1 but we also now now integer operations which on the more consumer side of things or for say large loads of data was really handy so now this code looks a lot bigger but the idea here is if you you'll if you jump to where uh in red you'll see xmm pointers right here we basically have two Loops where you see all those word pointers xmm word pointers the idea here is that you have one m one main Loop that does two of the FP 64s at a time you're doing add you're adding packed doubles together so you'll see add PD that you're doing the add operation you're adding two numbers together and then the PD is packed so one register multiple data pieces of data inside and then within that the D is doubles so double Precision numbers problem here is that you don't always know when you read if you're when you're running near the end of your array if you're going to be aligned on having an odd or an even number of data points so at some point you might have only one left and when you're writing general purpose code you don't know what you're going to run into so if you look to the right of the XM work so bottom of column two you'll see a lot of the same instructions as the one on the left but instead of being the packed variants these are the single variants so add SD so add a scaler double register so this means if you have one trailing at the end well you have a tail Loop to handle the residuals when you only had one of these entries not a big deal but when we get to up to bigger and bigger registers which we will that becomes more relevant a note here on compilers for package maintainers for people deploying software you may have heard optimization Engineers re uh research software Engineers people that want to make your code go fast for you talk about make sure you tell your compiler what you're running on because fundamentally ssse 2 which is also when we went from 32-bit to 64-bit on x86 is still the default Target still today remember that part about an code and soone designed in 19 in the mid 70s was not great for stuff we're doing today well well by the same token why are you targeting an instruction set that is now 23 years old that's old enough to vote and so on and drink everywhere right come on SO fundamentally make sure your compiler knows what it's running on right when you your users your packagers or your libraries don't provide an MR Arch or telling the compiler what architecture you're targeting the default is to fall back on SS2 and I might get some Guff for this but it's hard to believe technology has come a long way since 2000 right so it doesn't feel like it sometimes I'm here talking in 2023 as a 26 year old guy talking about Assembly Language that feels like straight out of something from five decades ago but technology has moved on so we go on and 2004 rolls around and we have the streaming simy extensions version three in 2004 really cool instructions B it's like a do it at this point H add xmm1 and2 is horizontally add packed doubles so the idea here is instead of two registers on top of each other going vertical in your pipeline what you're actually doing is you can say okay I have this register of two data points I want to sum those and then I have this other register and I'm going to sum those and then I'm going to store them in one register so you can see how this can especially um like a really common uh application for HPC these days is things like artificial intelligence and so on and one of the applications that those rely on a lot is an all reduce application where you're basically summing up and multiplying every single part and you can do this for your Matrix decomposition as well where you're summing and then you're bringing together this doesn't help for a Dax P example but something like it will ssse and that thing that will is ssse 4.1 and S 4.2 in 2004 here we get dot products oh my gosh which for some of you might be familiar a DOT product is fundamentally the same operation as a fuse multiply ad which in turn is the same as ax plus y right so here 2004 we have double uh dot product of packed doubles where you're taking two registers and then this thing called an immediate eight basically a control register to say hey if it's this value don't bother multiplying just pump it straight through the register now when it comes to software you'll notice I said the default earlier for ssse 2 because we never know exactly what implementations are and we don't want to Target all of AMD and all of Intel and all of via and all of Centrix and all of centur all at the same time all these different x86 minor and major vendors the folks at Intel red hat and I think it was Oracle came together to say we need a specification for general purpose versions and they decided that version two would be everything up into and including ssse 4.2 so that means a generic CPU that has all the way up to the SS 4.2 instructions basically that includes everything since 2013 including say a a really tiny embedded microprocessor inside of a set top box that's going to be running SSC 4.2 so in terms of our Dax P example this is going to be important I do want to have a quick note here S 4.2 wasn't as clean as it looks um at that point people were realizing after not having extended the instruction sets for a long time between the 1980s and the 1990s it became the thing to just add more extensions and add more extensions and add more extensions so there was a lot of experimentation that sort of made it into the market but never caught on so I'm just covering all my bases here so AMD came out with their 3D now extensions to that used the ssse registers we had SSS E3 uh that that did a few interesting things ssc5 was another one where it was trying to treat more underlying questions of floating Point sorry I'm just checking per time here Etc none of them survive become anything particularly relevant but one of the things that did happen at this point is that when we moved as mentioned to 64-bit operating systems we doubled the amount of registers we had we had eight now we have 16 of those registers you can thank the find Folks at AMD for that all credit is due when intel was trying to bring us down the itanium route that was a bad idea universally accepted at this point and we moved to amd64 better known these days as x86 64 so uh I do want to take a very brief pause here that was a lot to get through and and very quickly do we have any quick clarifications not more questions for the end but just quick clarifications of high level if you do please raise your hand in Zoom or post a message in the tech talks Channel and slack I'm not seeing anything popping up Felix so I think we can continue that' be yeah that's great um perfect so let's go on so section three Advanced floating Point same instruction multiple data or Advanced floating Point extensions AVX Advanced floating point so here we realized hey CPUs are starting to really especially x86 at the time is really taking off we're not doing our sun Micro Systems anymore we're not doing our cray uh vector and our cray Vector processors subasa from NEC is less and less of a thing so as a general purpose processor we're also doing comput on them at this point so avx1 in around 2010 comes on the advanced same instruction multiple data extensions targeting mainly HPC and here what happens is we're still at 16 registers but we're jumping to double their size so instead of two FP 64s we now have four of them if you're doing single Precision math you're have eight of them but the thing is this is something that Intel could do relatively easily at the time because they are a cisk a complex instruction set computer versus the risks that you see with arm and risk five and myips and so on so they can just really expand the entire instruction set and not only expand it but say hey we want to do a lot of really interesting things we need an extra bite on our instructions and that was the instruction of Vex the Vector encod it's not officially called this but the vector encoding subset of x86 where they added an extra bite as a prefix on the instructions and then suddenly hey we can address twice as many registers right they can be twice as big and so on and when you look at the assembly for these sort of processors targeting them when you make sure to to turn on your compiler optimizations for those targets you'll notice a lot of ad packed doubles becomes V ad pack doubles or vector ad pack doubles Vex ad pack doubles right the big thing is the V means hey we're dealing with larger registers or we're dealing with a more advanced instructions in the first place most of the time it's both it's always one of the two whenever you see the V so here you'll notice compared to our Loops from earlier on the left still the same code on the right you're seeing these VM unpack doubles ymm zero so here what's happening is the compiler is unrolling our code so unrolling our loops and then prefetching all of our memory at the same time so not only are we low are we doing more math at a at the uh per iteration of our Loop the compilers realized hey memory pressure memory bandwidth is a problem so what what happens is I can use these larger registers to just do a single read of memory at the at the time so instead of loading one fp64 another fp64 another fp64 another fp64 just for my ad and then doing the same for my multiply so three times that you can say I just want one load one instruction to grab everything and then at the same time we're realizing is hey we're what's known as an outof order execution CPU so your actual instructions under underlying your chip we can actually we don't have to execute them all in order which means that we can do a lot of prefetching and while we wait for that memory to arrive we can do the compute for another part of our array and another part of our kernel right you can do these things simultaneously and change the order because fundamentally uh yeah D times DX yes sorry I'm looking at the chat it's not technically Dax it's close enough for everyone here if you're Bloss libraries yes we can we can chat about that anyway fundamentally we are doing larger vectors and that was great for HPC but fundamentally x86 is not only an HPC Target at the end of the day we're actually targeting a lot of consumer a lot of general purpose Enterprise a lot of data center applications and to the shock and dismay of everyone including yours truly not everyone does everything in 64 bit floats so avx2 comes along Advanced c 2 Electric Baloo keeps those 256bit vectors and then expands that to be six and a half data types for your registers the half is important gets to what I work on personally but we still have the 4X FP 64s we still have the8 FP 32s but now instead of relying on on SSC for all of our integer operations now we have 256bit Vector support for 64-bit quadword 32-bit double words 16bit words and then our bytes and you can do 32 of them at per per iteration of array that's a lot that's a lot and then I'd be remiss if I didn't mention the 0. five at this point with between avx1 and avx2 we had a generation called Haswell or sorry ivybridge uh and with ivybridge Intel implemented this thing called f-16c uh which was floating Point 16 conversion and the idea here is that I E had finally updated the floating Point spec in 2008 to add fp16 as a an official specification but at that point everyone was like you can't actually do anything in fp16 for actual arithmetic but it might be useful if we store floating points so that way because you're you have a defined definition for what converting an fp32 to an fp6 looks like hey we can convert between the two and then to save memory bandwidth what we can do is we can have all of our memory B fp16 load all of that into our registers convert that to 32bit do all of our math at a higher Precision for our time steps and then from there once our math is over we can go and we can store that back to memory as fp16 but fundamentally you can also at the same time to F to F16 C we get fuse multiply ad instructions and the fused multiply ad is interesting because it's like the dot product instructions from earlier but you're actually doing it in one step and you have a higher internal accumulation Precision so you don't lose anything because you're doing it in two steps you don't have two rounding steps like floating Point normally has so here's our assembly in our code once again you're probably getting familiar with this and if we once again go to the ymm pointer part you'll see VM on uh the vov is basically memory prefetching but now we're using ymm zero all the way to seven and those are only those are there to do all of our memory caching and then we have this VF mad 213 pack doubles which is little has a vector fused multiply ad registers two one and three of that contain packed doubles so what's cool here is that before we were always doing do your multiply then do your add right or do your add then do your multiply whereas here we have one instruction and it's taking in three pieces of data you can never do that before right you could do two floating Point registers with packed data and then like an immediate integer like a an little 8bit thing it's not all that useful but here we have three full-fledged 256bit registers coming in at a time and once again because of things like the branch predictor the outof order execution engine because we know this fuse multiply ad instruction is coming we we can go and do the memory prefetch transparently in the background so we only effectively do our memory prefetch for two two of our operands and then at the when the instruction comes we're just in time to fetch the third operand for that instruction now once again these are really big vectors right we're storing four FP 64s per register so on the right side you'll see we're still using those VF mad instructions or the very mad instructions uh but you'll see it's SD instead of PD so it's a scaler of double Precision so basically we only have one entry in our Loop and that's just to make sure that we're not fetching memory out of bounds we're not accessing something we shouldn't be accessing so it's oh now we can decrement the last part you'll sometime it's not quite the correct term but you'll sometimes see this referred to as tail latency or tail the tail of a f of a multiply instruction scen so it's worth at this point comparing the two so you'll notice the top part is nearly identical the end is nearly identical but when we're doing our actual math hey we're doing some prefetching here some pre-etching there but we're actually combining all those instructions into a single instruction and when you look at the actual implementations of these instructions on the actual CPUs running the map you'll see each multiply instruction was typically like four maybe five clocks and then that add instruction was another two clocks but something that CPU designers did that was really clever is that they essentially made it so that the fused multiply ad variant of these instructions was actually using the multip the multiplication floating Point units so you actually got the ad for free once you did this so you all so you actually cut your execution time to do the math by two 2/3 to four to six instructions total for your fuse multiply ad instead of four to six for the multiply and then one or two for the ad itself my mouth is slightly wrong there but you get the idea and then just to break that down a little bit more just so it's a little bit clear we have our main Loop and we have our residuals we have our main Loop and we have our residuals there you can really see Vector fuse multiply ad of pack doubles so once again you'll notice earlier I mentioned we have levels so that you can generically Target devices Target sement all the way to abx2 including fp16 and fma3 fma3 is the actual instruction set for three operand fuse multiply at are considered x86 64- V3 compliant the important thing about these flags is that they're recognized by your clangs by your gcc's by your ixs your a PS and so on they are generic compiler LEL that you can pass your compiler to once again make sure you aren't struck on instructions from the year 2000 and this is pretty much every HPC CPU for since 2014 there's a few weird embedded chips for once again like set top boxes that don't have avx2 or AVX but we're this is an HPC talk with a mostly HPC audience for your all intents and purposes you yourselves can always put x86 64 V3 maybe not in your home Labs if you're running some really old say s sandybridge era zons but they don't last forever guys one thing here is that it doesn't include tuning so once again we're looking at the specification of does it support it not how does it implement it just a a side note there so once again that's a lot to get through I wanted to open the floor for any quick clarifications so again if you if you would like to clar clarification here please raise your hand in Zoom or speak up in the tech talks channel in slack it looks perfectly clear and certainly to me Felix so wonderful I realize that especially looking at assembly for those unfamiliar can be a little bit intimidating so always want to provide the option we are rapidly running out of time good Lord okay so section four larger Advanced floating point simy so this is where the title of the talk came in AVX 512 or this uh ax10 a solution to the seven fols of ABX 512 ABX 512 came along and we're going to talk about it but fundamentally it did a lot of things but one of the mistakes it did in my opinion opion was not requiring contiguous sup sets of all previous instructions for your implementation which meant that you could pick and choose what instructions you added that were from AVX 512 which means that when you go to support all these different AVX 512 devices you end up with this mess of somewhat overlapping somewhat suet CPUs and supported instructions not all of which were supported at the same time many of which came out at the same time same Years and we're all all say Zeon third generation or Zeon second generation but didn't support the same instructions so you could have one Center in the states that bought third generation zons and then one Center in say belgum that bought third generation zons yet those third generation zons can't run the same code right and we'll go through all these various things one of the things you're going to see is AVX I'm going to refer to AVX 512 not as AVX 512 I'm going to be referring to it as AVX 3 to ax9 because that's sort of what happened it's not quite true but it's close enough for unless you're actually writing an instruction set architecture other note I'm ignoring Zeon fi Zeon is a really cool device but fi and laraby just aren't relevant for the topic of discussion they're good for historical context but they're just not relevant here here um if you're interested in what became fi what was the lar ofy project at Intel I highly recommend Tom foresight's talk smack me to ABX 512 the life cycle of an instruction set the links are in the extra detail extra resources on the easy build landing page for this talk really interesting stuff we just don't have time to cover so AVX 512 the original version AKA what I refer to as AVX 3 Advanced simd 3 big vectors look at me so what we're getting here is you may remember abx2 we were at 16 256bit vectors Vector registers so these are already pretty big and that's a lot to keep track of every time you're contact switching for example right but AVX 512 in its original form ax3 gave us 32 registers and also doubled them so we're getting four times as many as much total bits for our registers and the original version supported four things it gave us eight FP 64s eight double Precision floats 16 FP 32s eight time 64 you can read the slide you're all smart people I'm sure you're literate so but the big thing here is we have more registers to do our math and store data and do our prefetching with as well but I said I'm ignoring F I'm ignoring it because abx3 itself never ended up in a single chip what I would consider real ABX 512 or version 4 added a bunch of other extensions to make it actually viable in a big CPU and something you could actually install specifically right now we're looking at a die shot from sky like X the 10 core variant so there we get six we add on top of what we already had we add 16 times 32-bit integers 32bit unsigned and signed integers and 64 times 8 bit integers and unsigned integers you'll notice throughout the talk I was talking about integers but we never had the unsigned variance well finally AVX 512 brings that to the table but fundamentally from for HPC it's more of the same as abx1 we get we still have fuse multiply ads and we have bigger registers great we can do more math per cycle so that point let's look at that you'll notice at this point it's pretty much the same as before but now we're looking at zmm or zmm registers depending on the world you're from and the idea here is that these are the same registers as our ymm which are actually the same registers as our xmm we keep just expanding and making them larger and longer and adding more of them right but in terms of our actual instructions we expanded them from Vex to Extended Vex or evex but they're effectively the same right we're running the same sort of instructions then we got version five which was originally uh Canon Lake but that Canon Lake never really took off it did some really cool things with instructions though it's essentially one data type called the integer fuse multiply ad um to your point Kenneth that's actually because the a the B's and the C registers were used for the main general purpose registers so we went to the opposite side of the alphabet but anyway um so here what happened was hey if you actually take a look at the underlying structure of a floating Point 52 bits of a of a double Precision floating point is actually just an integer right and there are applications say like signal processing uh 4A transforms and so on where it can be useful to do everything in integer maths but at the same time we want as much Precision between steps so that would be your multiplies and your divides as possible before any rounding from the integer side right so what happened here was they said you know what what we can do is we can give you an integer version of these big floating Point units and now it's a 52- bit integer and it uses 104 bit internal accumulation kind of cool and then vbm was uh Vector bite manipulation instructions which just aren't relevant for HPC they're usable for some kind of lower level data processing applications signal processing stuff but it just wasn't relevant for HPC ABX 512 version 6 came along and here's where we're looking at it's the age of AI in HPC where the big AI folks are running a lot on gpus but when they need to run big clusters they need the memory and they just want to process on their actual CPUs so Intel came along and implemented what's known as a brain float which is fundamentally the same as an i fp32 it's the same thing but what happens is AI cares more about the range which is at the front of your data type less about the Precision who knew AI was not precise shocking I know but fundamentally you're truncating and after that all of the rounding all of the other considerations of rinf flat are the same as fp32 so they added this Cooper Lake had an in a limited installation Market it was mostly the big cloud vendors uh those folks so AI firms using HPC love this but classical HPC where it's floating point or it's nothing maybe a little bit of single Precision just wasn't relevant for us we got version 7 at that point uh which was not any data type expansion but what we did get at that point was integer FMA for unsigned integers and uh signed integers for 8 bit and 16 bit so for those looking once again I feel like I'm talking a lot about AI but that's what the conversations are these days when you're looking at inference workloads so you've trained your model and now you want to deploy it there's something called quantization where you trained an fp32 but you don't want to do all that level of math for single time steps when you're looking at thousands of time steps steps you need the Precision of a big floating Point Unit but what inference in AI workloads does is we're only running one Epoch right we're only running one more time step of our model for the most part details so what happens there is that you can reduce everything to their bare minimums with eight bit and then that's where vnn or the vectoral neural network instructions came in where you had these little tools and there you had AI using HPC used it for CPU inference whereas brain float was more for training the model in the first place and then bbmi 2 was expanded some of the uh Vector B manipulation instructions and then beyond that bit algorithms where things like uh population counts and other very low-level manipulations that just once again they're just not relevant for HPC uh ax51 version 8 would be vp2 intersect or and this one's actually since been deprecated um I asked around a lot because it looked like it wasn't happening anymore and then uh Runa sigal from Intel uh helpfully commented on my one of my tweets where I was asking what the heck's going on with us and said no it's deprecated or yes it is deprecated so this one isn't super relevant but it was its own discreete generation version nine finally floating Point numbers yay HBC so here we got 1 plus I data types uh because floating .6 never does anything this normal way we had to get a half data type support for storing but not for adding it now we get it in full so now we get I 754 compliant binary 16 and we get to add it we get to multiply it we can do whatever we want with that it expands even beyond what we had with fp64 and fp32 where we now get dedicated complex plane instructions for encoding so for things where you're doing transforms in and out of the complex plane instead of having to do data manipulation steps so some exponents and so on you can actually do that all in one instruction so now not only can you use it for data processing and for some lower level uh Precision stuff so say the the store and then expand stuff I was talking about earlier but some kind of higher level Edge maths can be done completely n16 excuse me a lot of talking so that was version n so as of now once again with our generalized Target that your compiler can actually use if you implant all the way to AVX 512 F Foundation BW bite and word CD conflict detection used for unrolling loops and conflicts and so on DQ which is is double words quad words and VL which is Vector length which means that your ABX 512 instructions can run on your big 512 bit registers but they can also operate on your 256bit and 128bit registers you have all of those things which is approximately version three and version four together those are considered x86 64 version 4 compliant that's basically everything in HPC s from Intel since 2015 AMD since Zen 4 which is uh sort of started seeding to some labs and some users in 2021 but actually made it into General production in 2022 and I think there's one or two people in the room that got to mess around with uh centaur's uh ABX 512 implementation it doesn't matter I'm not going to talk about it just like five but the long and short of it is that you have really powerful data processing and lots of massive vectors so avx1 that was a lot about x512 for title for talk that's about ax10 what in the world is ax10 Section Five no more subversion subverting expectations the goal of ax10 is to put an end to this I hope ABX 512 dies a painful death in that Intel starts fixing real problems blah blah blah blah blah famous lenus torvald Al quote about why he thinks especially at the time that yeah ABX 512 should die because at that point it was purely an HPC thing we weren't normal people weren't getting much value out of it because you could only have AVX 512 on these big large systems and these big large CPUs for the longest time right 32 times 512-bit registers doesn't sound like a lot but supporting that in your floating Point units supporting that whenever you have a context switch and then need to store like load and store those registers that's really expensive and that's hard to do in a small laptop CPU that's supposed to only draw like five to 10 watts it's just really hard to do and the goal of ax10 is also to put an end to this where uh this is from y cruncher the pi and other kind of mathematical constant uh benchmarking application where you're having to detect okay I have the foundation instructions I have the conflict detection but I don't have the extended reciprocal instructions and I don't but I do have the variable length instructions but I don't have the four input fuse multiply ad of pack single instructions from Zoni and on and on and it's meant to take an end to this which is some code I wrote for uh like a little Benchmark application where you're having to individually test every single flag for hey do I have AVX at all do I have ssse do I have have AVX 512 do I have AVX 512 fp16 do I have F just the fp16 conversion instructions right you're having to do all of these routines one at a time for your optimized Bloss library or your optimized lapack Library just to see hey I think I have this chip but I need to check every single flag at run time and at compilation time to make sure that I can actually run this code because the last thing you want is a user working walking up to your cluster and that CL just giving you an error that's that's not fun and the last thing we want is for that to happen users deserve better right so that's why we have to do these routines but these routines are a pain in the butt and then this is from open Bloss where this is some some this is like on Tenth of the detection code for figuring out what chip we're running on and we're doing it by detecting what instructions it supports and then later on it's like oh is this Al their like desktop or is this Al their like mobile or is this Skylake X or is this uh Cooper life right and this from Jeff Hammond formerly of uh argon then Intel and now Nvidia where it was an entire sweet Mass deployed in open source for detecting what's on my chip what AVX 512 does it support how many fuse multiply ads does it actually support right I think you all get the point ABX 512 because of its nature of not being a contiguous super set ends up where it's spaghetti and you never know and you can never trust what you actually have so you end up with all this extra infrastructure code that is always spaghetti because you're always having to re-implement it every time something changes and something gets added and if you don't get it at this point I don't know what to tell you so ax10 answer the darn question what is it ax10 is actually avx1 NM okay okay I know give me a second we're going to get there what does the point n in ax10 NM mean simple it's actually really simple n is version number you have ax1 version one two and three in the future period something gets added you increment importantly if you CPU say it supports n plus4 so let's say version five down the line that means it supports version four and it supports version three and it supports version two and it supports version one guaranteed you don't have to think about it it just supports it clear super sets what does the m in ax10 NM mean simple m is implementation size of your vector Registries valid options are 512 256 and sort of 128 I I'll get to the 128 later but by and large you'll notice earlier I was talking about how it's a lot of state to keep track of right 32 time 512 bit vectors is a lot for a consumer CPU or a little embedded chip that's going on a small little robot it's just not going to work you don't have the power you don't have the space to implement what you want to implement here so you're making a small chip do the 256bit version period and importantly here if you have code for 256 bit abx1 as long as you're at the same version number or your code is compiled for the same version number your 512 bit CPU will run that code all day long no questions asked same way that all abx2 CPUs can run smaller ss4 codes okay what does it include version one which which is the only version that's out so far and that we know about publicly is all of avx2 which implies all of avx1 which implies all of fuse multiply ads and ssse 1 till 4.2 and all of the instructions from AVX 512 Foundation conflict detection variable length double quad words bite and words integer fuse multiply ads Vector bite manipulation instructions one and two population count bit algorithms vectoral neural Vector neural network instructions the long one galwa field neural instructions V bf16 and fp16 we have all the instructions from those Generations but not the 512-bit vectors the ital the italics and asteris on that last slide are very spooky on today Friday the 13th so what's up with that because ax10 is meant to allow instruction support on smaller chips it keeps the instructions from ABX 512 but not the register sizes isn't that just gonna go talking to people about this before this talk came about it was like isn't that just going to make things worse and like all things in life yes and no it depends it means it does mean you can run a single check and know what is and isn't supported great less spaghetti code it doesn't mean you need to check for local implementation size fine but it does mean I'm checking two things instead of checking nine different things and trying to build a tree of what that does and doesn't imply for targets it's a good thing say it again most of the instructions from AVX 512 are great for commercial data center as well as local client data processing they don't do much for classical HPC apps right but that's we're not the only people in in the industry we're not the only people using computers right so ABX for HBC beyond the spec what is it for admins and users it's a single compiler flag to enable all the goodies in the history of ABX 512 right for programmers developers research software Engineers Etc it's a pair of CPU ID flags that you can check in your assembly functions to check for kernel version and targeting and implementation size kernel here is a fancy math SL Library name for what your Bloss function is actually like right underline it's not the Linux kernel or the windows kernel name overloading is fun this is one another case of that is it out yes section six apx 10 for HPC in 2023 so ABX 10.1 512 is coming officially next year in with Intel's Granite rapid zons they've been public about this it's coming soon but that's next year don't care how I want it now good news AVX 10.1 512 is the same from an instruction set standpoint so the actual binary instructions as Sapphire Rapids so Zeon fourth generation and the upcoming retooling of sapphire Rapids known as Emerald Rapids which is coming out in December per Intel for those with uh Intel Dev cloud or that are affiliated or have access to argon National Laboratory argon leadership Computing facilities Sunspot supercomputer which is based on which is a subset of Aurora your codes for spr are the same Isa as ABX 512 or ABX 10.1 512 they'll run the same That's all folks uh I do want to have some brief notes um to add some content here some context avx1 256 are coming soon I personally expect this isn't under NDA or anything like that it's just from kind of what I've been watching on the mailing lists expect the 256bit version so for your developer laptops uh that you'll be deploying across to your Fleet for the it managers in the call probably in 24 and I expect AMD to support avx1 in five tall bit uh in probably 25 brief note on AVX 10128 in general uh yeah I'm I'm at time here so ABX 10128 is a silly silly silly idea uh should never have been allowed to exist and I've actually asked Intel formerly to remove it from the specification um I wrote an entire companion article that was originally slay to be part of this talk but there was no way I was going to have time to go through that so um for those interested in like microarchitectures and so on there's a really good publication called chips and cheese uh and I post a companion article uh to that Artic um to this talk on that publication there's a QR code here the slides will be available for everyone afterwards and there's a link in a QR code if you're interested uh in terms of me uh I'm Felix uh generic Canadian programmer guy wearing a flannel because it's past October and that means flannels are out um you can mostly find me active uh at fclc atm. HBC doso on Mastadon I'm also on Twitter for now I'm mostly migrating my way off at Felix dcore uh and if you really have something uh important or critical you can email me um first name. last name 123 at Hotmail because I've had that email since 2006 or something uh and then these are the two profile pictures you'll normally find me at um I do want to acknowledge some stuff mainly um I want to say a big thank you to people that provided quotes at the beginning uh various CPU Architects both working on x86 and working on risk 5 and working on arm Who provided context and ideas on CPU implementation details once again a spec without an implementation is useless and implementation without a spec is also useless you need both uh various spec lawyers as they call themselves Who provided insights and chips and cheese for hosting the accompanying article uh and finally I want to say a big thank you to easy build in general for hosting me and allowing me to kind of say my piece on ax10 and share that with the community uh abx1 General is really exciting because it makes all of our codes a lot simpler to just say hey just plug in this flag once it's out and then everything just kind of works right um we'll go to the this says closed session Q&A but uh I think we're going to do a few open questions uh if people want those questions be part of the recording before we go into a closed session uh so this will also include questions from the live stream for example so uh if we want to open the floor uh go for it all right thank you very much Felix this was a very great uh talk thanks a lot if people have questions in Zoom you can raise your hand uh we will allow to unmute you so you can ask the question yourself uh you can also just post it in uh in chat and then we we will raise the question to uh to Felix and of course also via the tech talks channel in the easy bu slack questions are welcome also for people watching the YouTube live stream all right we do have uh one question already uh what does the assembly code look like or AVX identical um if I go back up bunch of slides to uh abx4 when it comes to you're doing a double Precision ax plus y it's identical uh the idea with ax10 is not to give us more things to worry about it's to take all the things we already worry about and simplify to a single Target one thing to worry about no more no less right I think that answers the question um I have one as well have two actually so if if nobody else is coming up with questions I can ask a couple um so one of them is you you've showed the various levels let's say of AVX 512 up until 9 UM and you've also mentioned that V4 is basically AVX 4 right yeah basically so there's there's a gap in terms of instructions that are supported where the kicker is ax9 where there's lots of let's say interesting stuff for HPC do you have any idea how much performance you lose by running a binary that's targeting the the standard V4 x86 Target while running that on a chip that supports ax9 are we talking single digit percentages or could it be a lot bigger if you say depends you probably will but yeah so no I'm trying to avoid saying it depends um so from an instruction level for double Precision math it doesn't matter abx4 is equivalent for floating point 64 to AVX 9 in terms of instructions because none of the new instructions from ax5 till 9 so all the various levels of ABX 512 affected double Precision other than the first ones that said you have it now right what happens though is implementations have changed a lot so implementations are you take the spec you put it in an actual CPU so Sky Lake X versus Cascade Lake versus Cooper Lake versus Ice Lake versus Sapphire Rapids versus emerald Rapids those all do very different things internally and then even within those Generations sometimes uh Skylake X for example the first generation of CPU ABX 512 was Notorious for downclocking so if you normally run at say 3 GHz running abx1 and abx2 codes you would run at like 2.5 across all your cores when running ABX 512 right and because memory bandwidth was such a problem and continues to be the loss in clock speed overall and some memory timing issues meant that you effectively lost out on performance and that you were actually better off using ABX 512 instructions on 256bit vectors right and that's changed over time especially since ice lake so uh Zeon third generation or third generation Zeon whatever the marketing is for that you basically at most you'll lose one maybe 200 megahertz on a 3.5 to four gigahertz chip so it's it's borderline negligible at this point so you're typically better off but there is no kind of one- siiz fits-all because you're not going to be running fuse multiply ads at all points right and the thing is once you jump instruction levels that's an extra pair of bytes at the front of each instruction to decode because you're running from ssse then you went to your Vex your vector instructions with ax1 and then your evex with ax 512 your extended Vex and it was yet another bite right so that's more instructions in your instruction stream doesn't matter that much two bytes versus a 512-bit load negligible but it's yet another Factor at the same time you have some implementations like amd's recent Zen 4 uh which if I bring sorry so zen4 will support AVX 9 no okay no um so I'm just bringing back to this slide here so what happened is Zen 4 is if you look at Sunny on the left in the yellow and then you look at Cooper Lake in the purple underneath golden Cove that is Zen 4 right so you have your vectoral neural network instructions those kind of fuse multiply ad of low Precision integers and then you have your brain float 16s from Cooper Lake which is roughly uh AVX I think I have them as AVX 6 and seven respectively so that's zen4 but zen4 has no support for actual arithmetic of floating point 16 of i e floating point it has the brain float 16s but it does not have floating i e floating 16 which are actually physically different like they're actually different types entirely um so that's roughly where zen4 is but you'll notice that from an Intel specification standpoint and an Intel CPU standpoint they never released something that was equivalent to zen4 that was which would have been Sunny Cove which is ice Lake and Cooper Lake at the same time right so that's kind of where Zen 4 ends up the other thing with Zen with Zen 4 is that they did something really clever where they said you know what we don't have the die space because we're sharing the same core across our consumer and our Enterprise CPUs um to go and implement this big 512 bit Vector because prior to Zen four Zen three and so on that's your r um and milons those were stuck at avx2 and they never did any AVX 512 right so to go then and say okay we're jumping from 16 256bit registers to 32512 bit registers that's a lot of silicon so what they said was hey what we'll do is our integer uh arithmetic logic units and our floating Point arithmetic logic units or floating Point units will Implement them as 256 bits and then what happens is you can take the first half of your 512-bit vector push it through and then immediately go and fetch it and put the second half through so some of you uh may be familiar with Vector processors from NEC or way back in the day your cray uh Vector processors where those would have large vectors and then they would pipeline them into a smaller floating Point units this is the same kind of core idea uh and sometimes it works really well but sometimes if you're doing HP DC only and you're just spending a lot of time in your general Matrix multiply routines or Matrix Vector routines your gems and your gy um then at that point Zen 4 doesn't look so great but for most normal people applications Z4 is between fine and wonderful and because of the power savings they were able to do with that style of implementation and some other in a lot of like lowle compiler tweaking you're able to get a good like 70 80% of the performance of an 80 of a full 512-bit floating Point Unit which gets near enough that it made sense and then you also get a lot more cores so performance looks better and you also have more memory bandwidth so that helps so once again it's a tossup it's going to be application specific there is no one siiz fits all but in terms of instructions it's the same all right that was a very long answer but yeah sorry really that's fine that's fine um there is another question in the chat so let's let's raise this one from mirin um if spr supports all of the AVX 10.1 512 instructions will we be able to compile something with the AVX 10-m Arch flag or equivalent and will it then run on spr as well so I'm guessing he's asking about cross compiling well yeah I think it it's to do with uh the slide that I said where it's like this uh abx1 uh probably this one where it's the same actual instructions so from an instruction level you're compatible but the flags that your CPU ID uh functions check your assembly routines to check hey is this instruction supported those are different now with that there is no reason that Intel could not issue say a micro code update to Sapphire Rapids and Emerald Rapids to say hey say that you support it because they actually do support it but there's kind of two factors that stop them from doing so with sapphire Rapids and one factor for Emerald Rapids so the two factors are the one that's shared between Emerald Rapids and Sapphire Rapids is validation even though they are the same underlying instructions before a CPU is sent out the door from the Fab and packaged and so on that design has to be verified against every instruction from every instruction set a lot of that can be streamlined when it's the same instructions but fundamentally you could say that Sapphire Rapids and emerald Rapids have not been validated from a we guarantee this processor will work when you do XY z um on the other and then specifically in the case of sapphire Rapids picture you have a cluster of sapphire Rapids zons with hbm big fancy one and then you've gone and you've deployed that and it's humming away doing all your Matrix multiplies for your physics or your biology codes and so on and then one of those servers as H as it happens dies so you go up to your HPS or your cray your HP CRS or your Lenovo or whichever other vendor you say hey I need a new one under my support contract send me a new chip the problem is then that new chip have gotten the update to expose AVX 10.1 but all your old chips in your data center do not have that exposed and then when someone's compiling let's say that the node the new nodes that do display the flag someone goes and compiles with Mr equals native which tells the compiler hey check all the flags you have because I'm going to always be running on this type of CPU and then your compiler goes oh okay sure works for me and embeds those CPU ID checks in your binary but then when you go to do your big high a high performance linpack run because you're aiming for top 500 because we all love bench marks um well then hey that CPU ID check that you have in your binary fails so now you have half your cluster that does support it half your cluster that doesn't because also a lot of clusters while we'll keep up with security bulletins we're not always running the latest micro code right um I I I hope that helps the answer is could Intel do it yes but there's some certification stuff that gets in the way and there's also some actual in Market problems to do so so I would say from a developer point of view treat it as a development tool do not treat it as strictly equivalent for production as a kind of high level CIS admin POV that yeah I think that's a very clear question very clear answer sorry uh we have Bart who has a question I'll let him unmute hi Bart hello do you hear me yes okay yeah I just have a question so at the same time as Intel talked about avx1 they also released uh they also announced apx as another new instruction set and and so I I saw people talking about having a V5 level that combines the two basically apx 10.1 and and apx but do you know if they will come out at the same time in New chips or one before the other um sorry I have to kind of tread on ndas are a thing um from what's been publicly stated ax10 is coming out first with granite Rapids next year uh apx I do not believe has been publicly announced for what shifts it's coming on um so do with that as you will from a high level standpoint when implementers and compiler folks get together for deciding what the next version five Target will be I personally see AVX 10.1 plus apx as being a very reasonable level um simultaneously though if you look at current x86 versions 1 through 4 those assumed the amd64 extensions and then various instructions on top of that so there's a reasonable argument to be made that instead of going that x86 version five should be apx you can say okay now we're starting over again and apx version one and then apx version one so happens to support whichever version of AVX 10 were on at that point um apx is sort of your Baseline general purpose register what's the core CPU that is going to be everywhere implements and then abx1 is very much your high performance simd instruction sets for doing all your floating Point math and all your integer big math units if you're doing compute then that's going to be an ABX if you're doing general purpose register stuff um like a lot of normal people applications are that's all going to be in your general purpose registers which was covered under x86 64 also known as amd64 and is now going to be covered under apx um side note opinion on apx apx looks really good uh clean clean implementation getting rid of a lot of old stuff from the 70s and 80s that has no reason to be there in modern CPUs it's a really nice nice specification okay thanks a lot um I think it this is a good time to wrap up the the on stream on recording Q&A um and thank Felix again for a wonderful talk for the people who want to and as long as Felix is up for that we can still stick around in zoom and do some more chatting about all of this stuff but we'll end the recording just before we go okay uh I did see one question um I'm not sure I ever caught the affiliate affiliation um so hi yeah um hi I'm Felix um I work in uh software optimization targeting HPC is specifically in reduced Precision um I was hoping to be affiliated with um with a very big institution uh but it's uh we're now as of today 23 weeks into me waiting on my letter of offer after having accepted a position so currently I am freelance and available for Contracting um and also if you have a place in your Center maybe we can chop um so officially freelance right now no official affiliation all right that's that's a that's a good remark to end with I think thanks again Felix uh we'll end the recording here we'll end the stream here but you'll probably be around for a little bit longer in Zoom yeah than thank you very much