RCE 114: NetCDF

[Music] welcome to another edition of rce again this is brock palin you can find us online at rce-cast.com where you can find the entire back catalog of over 100 episodes about high performance computing research computing and other topics again i have jeff squires from cisco systems and one of the authors of openmpi jeff thanks again for your time hey brock how's it going it's getting to be hot in the summer here i don't know if it's hot up there in michigan but it's certainly hot here in kentucky yeah the uh the humidity has been the issue here it's like swimming outside sometimes you think it's hotter than it really is fantastic well let's distract ourselves by talking about something interesting then okay so today we're talking about uh something that was actually on the proposed list for a long time but we only reached out to them recently about netcdf and we have um ward and russ here with us to speak to us so guys why don't you take a moment to introduce yourself [Music] okay this is russ i was one of the original authors developers of netcf at unidata with a guy named glenn davis and um he was tragically killed in an airplane accident about 1999 so after that i supported and maintained the net cdf library and utilities for several years and until and developed a proposal with uh ncsa guy named mike folk didn't to nasa to develop net cdf4 and recruited and hired some net cf development team members including ward and and wrote some blog entries about an ncdf and i am passionate about it still my colorado license plate has been net cdf for the last 10 years which is one of the geekiest license plates anybody has great this is ward i'm a computer scientist i worked with russ for several years on that cdf before his tragic retirement uh cdf has become a passion of mine my background is uh computer vision and machine learning uh but my work with net cdf here at unidata uh has been very interesting it's a something that's easy to be passionate about so what is net cdf well next udf was originally developed to kind of provide a standard interface between data providers and data users for scientific array oriented data and metadata and for portable data that was machine independent and platform or and application independent so the simple view is it's a file format and a data model and some apis and freely available software that implement the apis so you can read and write net cdf data and together those support the creation access and sharing of scientific data um there are some complications to that there's lots of different apis and nsgs involved over the over the three decades of use so that it's now actually got some variants but users usually don't have to worry about all those complications because of version compatibility and transparency we've always developed net cdf to keep to keep it backward compatible with previously written data and previously written programs so that when new new versions are released uh things don't break um and so i think uh well let me just mention a few of the apis that uh language apis that support nhdf access uh originally c and fortran and then c plus plus and java and more recently python and then there's also r and matlab and ruby interfaces and there's lots of third-party software utilities that's that can sit on top of an sdf for data analysis and visualization and management so what does the cdf part stand for in the name okay well the whole thing is network common data form it was not really format because we weren't really emphasizing the format we were emphasizing uh the api originally we wanted to be able to change the format underneath without people having to change their programs but we wanted to still support all previous versions of the format but people often call the cdf common data format and actually there was an original software from nasa called cdf and their theirs really did stand for the common data format and we met with them and used some some good ideas what they had was was a fortran only uh library that only was meant for vacs and vms machines and we thought that there were such good ideas in that that we wanted to extend it to see and make it [Music] portable for other machines and also create a single file format for it because the original nasa cdf was a multiple file format to store multiple variables in different files so so that's where it originally came from now you mentioned network is part of the name there but in the same breath you also say files so which is it or is it both it's really both um file files or containers for netcdf objects that are real simple variables multi-dimensional variables with their dimensions and some attributes but the network first of all means that there is a network format originally based on sun's external data representation xdr so that you can access data the same data on a network with machines that have different architectures and different ways of storing numerical and text data and also there's remote access to netcdf data using what's called opendap protocols open data access protocol that's been developed quite extensively with netcdf so that you can access data out of huge archives remotely small amounts of large data sets efficiently through open dap protocol requests that's all underneath the api so it's really no different than accessing data on your local machine except you give an url instead of a file name so netcdf is probably best well known in the climate and other earth sciences community how did that what historical artifact existed that caused that to come about well i'm in the historical artifact guys so i'll take this question too when it was first released it became an ad hoc standard for sharing scientific data and metadata among modelers in climate ocean and atmospheric science communities because it was at the right level for for representing that kind of data it had simple abstractions for variables and dimensions and attributes and those three things were very important uh because a variable like temperature on a three-dimensional grid and dimensions like latitude and longitude and time and attributes like what units are the data in were natural abstractions for the output of data models and for earth science data so it was a good fit to represent multiple variables on shared grids and even had the right abstractions to represent shared coordinate systems there were other reasons for its popularity then mostly these things were written in fortran these early models and c was was becoming more popular but um you know these fortran and c users uh didn't we're seeing the disadvantage of using fortran i o or byte oriented c libraries to write scientific data because it made their data not portable across platforms and languages and ncdf just provided an efficient portable language independent i o apis for fortran and c users and it had some other you know desirable properties too the data was self-describing it it had metadata about the data in it the file included ways to represent metadata of course it was portable it was scalable which means that a small subset of a large data set could be accessed efficiently you didn't have to read through all the preceding data you could append data to to a net cdf file without copying the the data set or redefining its structure so that was efficient adding a little bit of data to a big data file um it was remotely accessible as i've mentioned through these opendap protocols and this guarantee of of compatibility with backward versions of the software made it a good thing for thinking about keeping archives of data so i think those those were the most important things later on there was this development called cf conventions for netcdf metadata that became an international standard for representing metadata in output of models and forecast models and simulation models so that was also very important okay so what exactly then is the relationship uh between uni-data and u-car and net cdf what was the cross-pollination there uh so i'll jump in and answer this one so uh ucar uh is the uh managing organization for the national center of atmospher atmospheric research uh ucar being the university corporation for atmospheric research ucar maintains several community programs the ucp programs of which unidate is one and we have several other sister organizations all of whom support science and scientists in our community in our particular ways so unidata is primarily supports the community through development and maintenance of open source software uh netcdf being uh the most prominent software package uh that that unit data maintains yeah let me just add that that united has been around for about 40 years 30 years sorry providing data software tools and support to this community of the united community which is a bunch of universities also let's get into technology a little bit and i'll jump right into the probably maybe a little bit controversial one which is you mentioned backwards compatibility but then there's um net cdf 4 and its relationship with hdf5 can you talk a little bit about what the thought process there was and what you're trying to do there uh net tdf4 adds some of the features of hdf5 in a backward compatible way because it's a layer on top of hdf5 that also supports the previous versions of the format before hdf5 was used through apis that simply have extensions there's no there's no incompatibility with previous versions basically we saw that hdf5 from illinois had developed several advanced features like compression and and data chunking and we really wanted those and so we didn't we didn't really want to develop yet another format we thought well why don't we try to make try to kind of do a merger of net cdf and hdf5 by adding some more apis and using their storage layer underneath and that way we could get some of the advantages of hdf5 without creating yet another format and all the all the work that would involve and and it sort of worked i mean we we had hdf5 group worked with us and they had to add a few things that weren't there and uh we had to represent some some things that weren't there with uh kind of artifices that were built on top of hdf5 but the the result was uh pretty successful the net4 preserves the common characteristics of those two formats and takes advantage of the you know the widespread use and simplicity of net cdf and the performance and generality of hdl5 yeah i would add that in my experience the the net cdf4 uh it refers to the enhanced data model and enhanced file format uh it doesn't necessarily mean that that cdf3 has been deprecated or has gone away at all netcf three is now the uh referred to as the classic file format and classic data model uh and is still actively maintained and developed so the numeric naming convention uh can occasionally be a little misleading i've found yeah i i think that's right i think they're actually about equally popular now even though it's been 10 years since we developed an ncdf4 and etf3 is still very popular so given a lot of the functionality exists in hdf5 um why would someone choose using netcdf versus using hdf5 directly very good question that basically many users of net4 think that its data model and programming interface are simpler so it makes using netscdf netfn programs shorter and easier to understand than the equivalent hdf5 programs and that's not because it does exactly the same thing as hdf5 but has better interfaces it's because there's this trade-off between simplicity and and power and this ef4 intentionally doesn't implement all of hdf5's complexity and power but only a subset of the most important features but but there is another important difference that's not the only thing that is simpler and easier to understand it's it's it's an scf support for named shared dimensions this is an abstraction which was never part of the hdf5 data model and so an hdf variables that share a set of dimensions have this way to represent a shared grid or a shared coordinate system that's that's not anything that's naturally provided in hdf5 hdf5 is more serves as a container for all kinds of things and doesn't have the conventions for for representing shared shared grids or shared coordinate systems so um that's responsible for probably one reason people use netcdf for or even sdf3 instead of hdf5 when they when they want that capability and they want as simple an interface as possible and they don't need all the stuff that hdf5 has uh to add to that answer the other thing that the hdf hdf5 library is lacking is the ironclad backwards compatibility or archiving promise where we will never release you know that cdf will never release a version that cannot read uh data written by old versions of the library and that is not a promise you get uh if you're if you are using hdf5 directly um in fact we encountered something along these lines uh i want to say to middle of late last year with hdf5 where we the the two current net cdf developers had to scramble to to mitigate some changes in the hdf5 library which would have potentially broken backwards compatibility that that was our highest priority for for several weeks uh working around this change so in addition to everything russ said this net cdf provides this uh promise uh to give scientists uh confidence in archiving their data in that cdf directly okay so just to put that completely plainly um if i download netcdf today and install it on some modern os uh with a modern application whatever i can read uh with that one installation of netcdf uh data sets that were written 10 15 years ago with net cdf version one is that absolute correct statement absolutely okay it's not just that you can read the same data it's that if you have old programs that uh that created or read that data they will also work although you may have to recompile them and re-link to the new library to to keep them working sometimes they'll work without i mean if the format is changed underneath you definitely have to re-link to the new library but you don't have to change a character of the program all right let me go on a slightly different direction here being an mpi guy i have an mpi related question for you here there is a project out there called parallel net cdf or p and net cdf um but there's also an mpi enabled version of net cdf is there what's the correlation between the two so i'll jump in with pnet cdf if that's okay russ sure so parallel net cdf is uh an independent third-party project uh maintained as a collaboration between northwestern university and argonne national lab and it works with that cdf3 uh uh the classic library and data model and file format uh and it provides parallel i o uh which was not native to the net cdf3 code so i assume you mean the uh native mpi with net cdf and parallel lib hdf5 so when libya when the hdf5 library has been built with uh parallel i o enabled um the net cdf library at configure time uh before compilation will actually probe your hdf5 library to see if it contains the parallel i o operators and if so parallel io is just enabled and available through netcdf your which which is great because it lets your program which relies on that cdf uh achieve parallel i o without really having to change your code uh it is just inherent it it's used automatically because the underlying lib hdf5 i o is parallel enabled okay so this is mpi underneath the covers to affect the parallelism what about the other way around has anybody done the mpio apis with net cdf underneath not that i know of uh same here uh not that i've heard i think mpi is kind of a lower level library than net cdf it doesn't uh it doesn't deal with abstractions like variables and dimensions and attributes and so i'm not sure an mpi program could make that greater use of ncdf underneath so a file format is only as good as the ecosystem that can read it what are some of the other common tools people use with net cdf going from their simulation code to their visualization to archiving you know what are common tools that understand nets net cdf that uh people use i'll take a stab at this the um the library from the software that comes from uni data comes with three important generic tools that have lots of uses by themselves for conversions and abstract extractions these are called the nc dump ncgn and nc copy but there's lots of other tools as you can guess from a format that's been around for this long and if you look up netcf software on the web there's the list of i think it's over 80 freely available packages now that have been adapted to access sdf data and visualize and analyze and manage it and there's some commercial packages too there's about 25 or so licensed packages that use it and that's that's really too many for new users to have to choose from but there they can look at the descriptions and try to figure out what might be useful but there's some there's a few large third-party collections of tools that are especially suited to nco and i'll just name those now nco which are the sddf operators from charlie's entering his group at uc irvine ncl which is the endcar command language it's a bunch of really good graphics and analysis tools and a kind of a interpreter language that that deals with the variables and such ncf variables and then there's one called cdo which is the climate data operators from a group at the max planck institute for meteorology in germany and they each have their own particular strengths and the large collection of users so it's hard to say much more about them you have to have to use them to see what or look at them more carefully to see which one is most suitable there's lots of other single applications for doing browsing of net cdf data and nasa has has some packages that are a very good general mapping and analysis packages that's about all i want to say right now about that well i sorry go ahead uh i would also say just from talking with our users and community members you know for non-developers people who who just want to work with net cdf data the big three packages a company out of california esri has software that is commonly used for visualizing data stored stored in that cdf format matlab is another commercial software we get a lot of questions about or just that comes up in conversation but then also free tools like r and python uh both of which have uh net cdf hooks as well as the uh inherent visualization capabilities of those languages are also very broadly used but as russ said we maintain a list of just dozens and dozens of commercial and open source packages that speak not cdf yeah and you're mentioning in python i have to throw in one more thing here too because python's uh model for multi-dimensional data it is well very compatible with net cdf data in this package in python called x-array x-a-r-r-a-y x-ray developed by steven stefan hoyer is an open source project that that really brings the power of pandas using sdf data pandas is a is another popular package in in python it provides in-dimensional uh variants of the core pandas data structures and and in-member it provides in-memory representations for netfiles so it's really quite quite a good package to look at if you're going to be doing your programming in python and you want hdf access well so that brings up a related question here you listed off a whole laundry list of languages that the net cdf apis are available in how did you go through the typical quandary of exposing functionality in different languages are the bindings as close to identical in each of the languages or did you take an effort to you know like support pythonic things in python and uh see things and see and you know try to emphasize the strengths of the particular languages and could you cite an example well for the modern interface uh so for the modern interfaces the modern api bindings um unidata maintains three directly the core c library then the fortran and c plus plus uh apis which are just separate libraries with hooks back into the the core c library we also help maintain uh the python bindings though that is not a project uh we spun up from scratch uh only other languages uh of which there are many like r as previously mentioned um ruby uh pearl if you like uh any number of other languages actually come from the community these are thing these are bindings that we had zero involvement with creating and for the most part they exploit the features of the languages in which uh for which they're intended so that we don't try to make everything look like the c interface we originally tried to do that with a fortran 77 interface but later on for example when fortran 90 came along um an ex a user contributed a binding there that really exploited features of fortran 90 that weren't available and was much more comfortable for fortran 90 users and similarly the java interface and the the mental model you need to use it is quite different from the c fortran or python interfaces and it's very javonic if you want it's not like pythonic it's it's uh it was written by a sophisticated java user and so it knows about the idioms of that the language and the way you represent things i should also apologize to the java team for forgetting that the java bindings are also maintained internally right so what's coming in the future for net cdf okay uh unless russ wants to jump in i'll answer that so uh right now uh the the next step that we're looking at uh in the short term is extending uh the compression capabilities uh currently we we leverage uh z lib through hdf5 to achieve per variable compression in the net cdf enhanced file format but lib hdf5 provides a uh an interface for for adding in additional uh compression plug-ins so to speak my colleague dennis heimbigner has written an api that will let us leverage this uh and we're also designing some experiments uh to provide uh compression results to our community so that they can kind of see what they can achieve with different compression schemes beyond that with cloud computing having exploded the way that it has block storage is something we would like to be able to leverage with netcdf to be able to read from and write directly to block storage such as that provided by amazon and other cloud providers and beyond that largely we will be responding to the needs of our users because that is our user commun community is who we serve and what they need is is we try to get there at least before or at the same time as as them i guess i'd also say uh see the ncef github site um i think an sdf uh jumped on github uh sooner than hdf5 for example i don't i think they still may not use it but there are so many good developments and going on there and so many users who have been contributing that the the future is somewhat being driven by what's what people contribute and what how it proves to be useful and how popular it is so i think there are some plans out there puller requests are welcome and encouraged and any reasonable feature that is pitched and implemented and submitted via pull request will be given a full consideration let me ask another forward-looking question which you you may or may not have answers to but what do advances in hardware mean for you so faster cpus the advent of ssds you know faster access to storage uh faster networks you know do you use native network apis all these kinds of things that give acceleration possibilities um to the underlying hardware are the opportunities to use that in the in your implementation well uh currently uh you know the the the faster the underlying storage is to access uh the quicker the api that the library can retrieve data uh locally um if we are talking about uh uh data stored remotely via and accessing it via open depth the open depth api uh advances in network speeds and uh you know the underlying technologies and hardware there we will see in uh we will see better throughput net cdf the net cdf library is a storage medium it's not an analysis medium there's there aren't any operations to go you know for example request a matrix decomposition on data stored in that cdf uh and and because it's really uh primarily just file i o and a data model um there's nothing for increased cpu speeds or you know gpu accelerated uh programming there's nothing for it to really do that would benefit in that cdf at this point i would uh point out though that the ssd availability is actually uh kind of important if you're using if you're doing compression and chunking because when you re-chunk if data is written in a certain order and you want to commonly most commonly have people read it in a different order um and there are huge data sets for example you have something that's that's stored with all the data at each time and you actually the users actually want to take out time series at each point it's it's often that's about the worst case for accessing data that was written one way and you want to read it in different way and ssds turn out to be very helpful for that re-chunking of data to try to get it into a way that's that's not really really fast in one direction in one order and really really slow in another order but that is kind of uh pretty fast for any way you want to access it in along any dimension so so i wrote a blog about some experiments with ssds and how they could improve um very they could you could create huge improvements uh by re-chunking your data and the best way to rechunk it if you knew how it was going to be accessed was to use ssds rather than spinning disk just because you get much better performance for the kinds of things you need for re-chunking if you have lots of memory and you have ssds but that's about as far as we went with that so what about licensing uh what license is this library distributed under so i'll let you answer that one russ for historic purposes and then uh because and and then i'll have something that sure okay so you know data and sdf software from the start uh was sdfc and fortran and java interfaces were under a simple mit style license we actually wanted um commercial applications to be written with an scdf just to support it as an ad hoc standard and so we didn't really want to put any restrictions on its commercial use it should just be open source that's the sort of thing that mit style license gave us for open source later on there were some some issues about whether to use a gnu uh library license uh various versions of that and i think sdf java for example is available under multiple licenses including the uh live new license uh and the mit style license okay uh so adding to that um so yeah that cdf as russ described is currently licensed uh open source in the sense anyone can use it for anything uh wait which is how we would like that to be and that is how many other unit data product products and projects are licensed as well there has recently in the last 12 months been a push to adopt one of the big licenses one of the more uh commonly known licenses like a bsd three clause license instead of what we have now which is effectively a bsd three clause license but whatever the license changes to the spirit will remain the same it will be free to use for anybody be it commercial or open soft projects with really no limitations on how anyone uses it yeah i guess an apache license was another consideration which i'd forgotten about too because of the patenting issues but uh we we don't think there's any patenting issues with net cdf and and so we as far as i know decided not to use any of those apache licenses yeah are you going in a slightly different direction again here what's the largest data set that you have heard of that netcdf is used for so i sent an email out to our community mailing list when recently asking this very question and the response i got was someone who had a uh single single digit petabytes it was uh two or three petabytes of data stored in that cdf i think that was even single files that were stored in there were there were several petabytes right it was yes because there are archives like for example the uh ipcc uh climate data uh is is a multi petabytes i believe from the fifth ipcc report but but that's stored in millions of files it's not all just one one unit so this this person was it was uh just having a single container for petabytes uh yes i was impressed but that is uh that is correct russ it was in a single file and then uh an offshoot of that question that we like to ask a lot of our our guests here too is what is the strangest or the most unexpected use of your software that you've seen something that when someone tells you that they're doing it like okay wow we never thought that would be a use case um i have an answer for this but i'm curious if russ has one as well because he's got the broader view well i know that uh ed hartnett who's one of our developers always used to uh claim and so did rich signal actually that they did their taxes in cdf because it was so convenient but i'm sure that was a joke i there is some uh some some use of net cdf in um in some standards for storing uh what is it uh some there's an instrument that does spectral analysis of chemicals and they use uh it has nothing to do with meteorology or climate and the standard is based on the cdf but i guess that's not very strange um okay so the uh use case i'm thinking of is several years ago we had a support email uh from a gentleman who wanted to store all of his linux system configuration data in that cdf and had some very good questions about that and i was happy to help although he really never answered my question of why would you do that but i'm that's none of my business i'm i was happy to help him in in that same vein there have been people who became enamored of net cdf and said well why do i need relational databases i'll take my relational data and tried to store it in netcdf and that really kind of contorts the data model and it's not necesdf is not ideal necessarily for the kind of stuff you store in relational databases it doesn't follow that data model at all and and you really have to um contort things to do it to do that very well i think generally if something is well suited to relational database management systems go ahead and use those or but but for something that's closer to scientific data observational data or model data net cdf might be the way to go okay so you mentioned uh before we started recording here that you were one of the original authors of of this package here could you give us like the two or three minute history of netcdf how did it come about and how has it gotten to where it is today sure in 1988 uh actually we had some meetings among folks from nasa and the the cdf uh format that they mentioned uh and then uh some people from the university of new mexico who developed something called candace and a guy from a image processing company all to talk about issues with developing something like cdf that for unix and for um for other languages anyway the out of the meeting came the desire to develop our own uh software for this and not try to use the nasa stuff and we have uh support from the national science foundation who were the primary funder of unit data so we just developed this in 1988 1989 and that's when the the initiative version one came out in 89 it was beta version 88 and that gained a lot of popularity until about uh well actually all through the 2000s we talked about getting together with the hdf5 folks but uh we ne and we were certainly competing with each other and cooperating with each other but uh we still were two separate uh developments and then in 2010 2011 we thought of maybe making a proposal to i'm sorry i have to go back and change the dates in 2003 we actually got together with the folks from ncsa that developed hdf5 and tried to submit a proposal to nasa to develop this kind of merger between ned cdf and hdf5 that would put an hcf simple layer on top of the hd hdf5 underneath as a storage layer and that was funded by nasa and it's supported basically four years three or four years of uh development that never would have happened without that uh that grant which involved work both from unidata and from hdf folks so we'd like to thank them then there's just been so many contributions from the community of users everything from bug reports to uh actual big code contributions like for the fortran 90 or the python software some of the other language uh software so we're very grateful that the community is has provided so much of what net cdf is and why it's still useful okay well thanks a lot again for your time guys where can people find out more information about net cdf and get involved uh well a great place to start is at the unidata web page uh which is unidata.ucar.edu from there next you can go to our github page which is github.com unidata slash netcdf-c and from that landing page you can find links to the fortran c plus plus etc other uh landing pages as well as a lot of information about uh netcdf at the high level the philosophy and then the nitty gritty api details uh finally through the union data web page we maintain several mailing lists uh and joining a mailing list or browsing through the 30 years or so of archives is a great way to find out more okay thanks a lot for your time thanks guys [Music] you