RCE 118: MEEP

[Music] welcome to another edition of rce again this is brock palin you can find us online subscribe and find the entire back catalog of over 100 episodes on research computing at rce dashcast.com i have again with me jeff squires of cisco systems and one of the authors of openmpi jeff thanks again for your time hey brock it's getting to be i guess we're hitting mid to late spring here um and that seems like a perfect time to me to talk about electrodynamics don't you think of course it's always a good time to talk about that so we have with us uh some of the team that's behind me um guys why don't you take a moment to introduce yourself i'm stephen johnson i'm a professor of applied math and physics at mit i am artavan oscui i was formerly a student here at mit with professor johnson and currently leading a startup out of san francisco commercializing me called syphidis okay so can you give us a background on what meep slash sympatis i think is sympatis synthesis uh you know what is it so meep is uh a program to simulate maxwell's equations the equations of electromagnetism and there are lots and lots of methods computational methods for electromagnetism so meep uses one of the most general and basic methods called the finite difference time domain method so just discretizes space and marches maxwell's equations forward in time and so because of because of that it's handling the full time dependent maximum equations it can include lots of different kinds of physics and it's you know very general code uh with the trade-off this is not as efficient as maybe as some more specialized code that only solve one small class of problems so dr johnson you were on this show before uh about fftw which is probably what you're best known for there's a relationship between fftw and meep isn't there uh very tenuous so fhw is actually used uh for another electromagnetic simulation code that i developed called mpb which it is less general than meep it but it only solves one kind of problem it finds the like harmonic modes the resonant modes of maxwell's equations and that uses 40 transforms and and a fourier transform basis and also uses ftw and in fact f50w was developed in part four meep so meep uh solves a more general class of problems and so it doesn't use fourier transforms directly but it can it actually can call mpb for for part of its features and then and then mpb calls f50w so there is a connection but it's sort of a little bit indirect ah okay so but meep is actually an acronym right what does meat stand for so i think it originally stood for uh mit electromagnetic equation propagation we've invented lots of other meanings for the acronym like maxwell's equations for every person and other things like that but like most acronyms mostly people forget what it stands for and they just call it meep okay so you talked about mexico's equations or whatnot but break it down for those of us who are not deep into the science here what kind of physical phenomenon does this simulate i mean what how would i explain this to say my mother so you know maxwell's equations is all electromagnetic electricity and magnetism like circuits and also light and optics are electromagnetic waves and all sorts of things like plasmas and other kinds of things can fall into this category as well so mainly meep is oriented towards modeling wave propagation problems so problems uh like i said in optics modeling uh you know lasers modeling sensors um you know modeling all sorts of like optical fibers and filters and other kinds of uh you know phenomena also microwave effects so you can you can model microwaves and microwave devices and other kinds of things but mostly i think most of the people using meep are using it for optical and infrared simulations so how would that those types of simulations map into like a real world uh product and what people actually use i can give a few examples here um one kind of emerging area of display applications you guys might have heard of are organic leds they're recently been introduced by apple in the latest generation iphone x but also increasingly uh prevalent in solid-state lighting applications and one of the big challenges in organic leds is much of the light that's generated by these organic molecules actually remains confined within the device and so there's a lot of research within companies such as samsung lg nichia japanese company to design structures that can extract the light and to increase the energy efficiency of these organic led devices and so that's one application that involves quite a bit of optical simulations based on uh finite difference time domain tools such as meat and other solvers other applications involving photonics involve integrated circuit applications for data centers and so many of the listeners of this podcast may be familiar with bottlenecks related to data transfer within large data centers and there's been a lot of talk recently into transitioning from a kind of an uh copper interconnect technology to a light based optical fiber technology and at the core of that are essentially optical circuits that can convert electrons into photons and those involve quite a bit of optical design and uh simulation tools such as meep and others so getting light onto a chip from an optical fiber and into a circuit and off back off also solar cells like you know the if you're trying to make a solar cell more efficient very often what you do nowadays is you take a thin solar cell layer and you texture it or you put microstructures on it to to scatter the light and make it absorb more efficiently and to design that you need lots and lots of simulations so who are you know who uses this software so you mentioned a bunch of brand names in there and i work for a very large networking company too cisco um you know do we use these things to develop our hardware because forgive me i actually i don't know because i'm a software guy so i don't work down much more at the hardware certis level well i can say pretty much with high confidence that most large corporations developing display applications or sensors involving light and steven mentioned solar cells all of them electromagnetic simulation tools really at the core of their design and more recently there are applications involving augmented reality you may be following kind of developments in google glass and its successor daydream oculus for virtual reality all of these have involved really intensive research into kind of next generation display technologies and those involve lots and lots of optical simulations as well and now there are lots of simulation methods and lots of simulation software out there so sometimes it can be a little hard to track down who is using what so for example meep is a time domain simulation uh and there are actually several it meep is completely open source free software but there are also closed source proprietary uh simulation factors that use similar algorithms from a whole bunch of companies uh so you know and different people are using different software so initially i think me meep's initial foothold was more in the research world where you know when you're doing research and you're pushing the edge of of what's possible with optics you you often need to be able to get inside the code and really change what's happening and that's only possible with an open source tool uh but you know now our artivans company is working much more with a lot of different uh commercial commercial vendors that are starting to use this kind of thing so you've talked a lot about the organic leds which you know for things like displays you want to be in the visible range you also mentioned i think it was infrared um what about extreme parts of the spectrum is is this code completely generalizable like could i use it for extremely high frequency like almost like radio tran like like radio type frequencies well radio's not high it's actually low but yeah i got that backwards but um like yeah okay is it is it purely generalizable or do you kind of assume you want to stay and we're focused on this optical range you know from inner infrared you know to each side of optic you know visible but you kind of want to stay in that range now it can work for everything from radio frequencies to x-rays uh you know as a theorist i usually work in dimensionless units i said my wavelength is always one uh so you know in some sense uh the theory is the same whether radio frequencies are at x-rays the difference is the materials so at at really uh at long wavelengths uh you know microwaves and radio frequencies um you know where you have really really good conductors like metals is really high um then you know you can still use this code but sometimes there are other methods that take advantage of of that that really high connectivity to be able to be more efficient from metallic structures um at really short wavelengths in some sense once you get to start getting the x-rays it it almost becomes is overkill because at x-rays uh almost everything is transparent as you know that's why they use x-rays to look through you uh so there isn't as much to do in terms of doing scouting calculations meat can certainly handle it but it's it's like if if you're mostly modeling light propagation through empty space it's overkill to use a big simulation for that so you know many of the most interesting the reason the reason that i i mention uh uh uh you know most people are using it for infrared and optical frequencies is that that's where most of the interesting photonics design is going on at this space it if you're at microwave frequencies if you're ready if you're really long wavelengths 100 kilometer wavelengths but for radio waves then you're doing circuits right so i mean and it's overkill to do the full maxwell's equations for a circuit well and if you're a microwave if you want to trap if you want to make a waveguide at microwave if you want to send light along a channel you just make a metal tube you don't need a complicated geometry and again on the other end of the spectrum if you're at x-ray there's really not much to do with optical design because it goes through just about everything so the the interesting part is optical and infrared and that's that's why people need lots of simulation tools there so you mentioned uh mpb you know mit photonic bands which was another code you had done and you mentioned that meep is able to call mpb but you said that meep is more generalizable so what what does mpb do that you'd still want to call it versus just using meep why would you want to choose one package or the other so you know usually in and this is all of scientific competition there's a trade-off between generality and performance you know and ease of use so if you have something that just does one kind of calculation it can be very fast and very easy to use for that thing so mpb is one of those things that just does one thing it just computes what are called the modes uh the harmonic modes of a structure so if you have a waveguide an optical channel it will very efficiently compute you know what those modes are what the field patterns are that just propagate down the wave that we've got at a fixed frequency so meep is much more general you can have things that don't really have modes they have non-linearities or things are moving around in time so the frequency is changing it gets extremely powerful in general but the trade-off is that if you just want to compute the modes of an optical fiber you can do it with meat but it'll be slower and a little bit more annoying to use than something that's specialized for just that problem and so the reason that meep calls mpb is very often if you're doing a calculation for example you're coupling you're trying to couple an optical fiber into a chip and so you want to start off the simulation by sending in one of the modes of those fibers your initial condition in some sense your source is one of the modes of those fibers and so what meep will do is it'll call mpb to compute that mode and use it as the source and so mbb will tell it oh this is the fundamental mode of that optical fiber and and then meep will use that information and launch that mode in into the chip and then you'll see what it does the mode comes in and it bounces around or it gets converted into something else and so forth and people will do all those dynamics now one of the obvious challenges that we have in in lots of scientific computing is that uh you know on a single server you can do so much and so you know typical solutions to this are you know the the typical mpi types of solutions does does me support mpi do you guys scale up that way yes yes so in fact one of the things that our defense company does is is help with people do that feeling like okay so what's your what's the biggest scale that you guys have done or or i guess maybe a better question would be what is a typical scale that your customers run on well let me give some context to this which is the the range and the size of the problems that uh meat calculations uh involve can be huge we're talking from you can run calculations on your desktop kind of with a few cores all the way to hundreds of cores or perhaps even thousands of course and so that range and then the computational needs um really demand uh and kind of inspired us to kind of start this company a scalable resource like a public cloud infrastructure where you can on demand access a scalable resource that's sized just for the applications of interest and so particularly for a small startup or a small team that can't afford to build their own local cluster and maintain it having access to uh an amazon cloud or a google cloud infrastructure really enables them to leverage me and other open source tools in ways that would be very difficult if these tools were not available so how well so i that sounds exactly uh in line with my expectations of you know using amazon and google for their infrastructure one and it's perfect for exactly this but there have been some traditional challenges um doing hpc in the cloud such as the data transfer on and off and the lack of high-speed networking in the cloud how do you guys uh address that well the applications that we've been looking at uh recently haven't been pushing the limits of the uh interconnect capabilities of these resources yet so bottlenecks or communications bottlenecks yet of course they are there and uh it's just that the applications that we've been using uh have been fairly well constrained uh i would say um but certainly i would say that uh and of course having a dedicated hpc infrastructure for these kinds of calculations would obviously be important for performance reasons yeah and of course so some of the big calculations we do you you get time on a traditional super computer as well um you know a lot of the time you're running simulations on a few course uh that can even fit on a single machine and then you're if you need lots and lots of cores it's often because you're doing parameter sweeps or some kind of optimization where you have lots of lots of things that are embarrassingly parallel that don't even need to talk to one another uh so then you just need to run a thousand instances of them um so um but it you know it's you and usually in sort of data days of design work people are typically doing relatively small simulations and then every once in a while when they've got sort of everything working they put everything together and then throw it at a huge machine with you know hundreds or thousands of course okay you actually answered my question i was going to ask if you could actually optimize around the fact for total time dissolution because you do have the ability to sweep over frequency and maybe actually subdivide ways where you're not as worried about getting a single large one done quickly when you're more interested maybe across a range of frequencies or range of time or something like that where you could actually subdivide so it sounds like you're already doing that but so so first of all meep is is a time domain code so you're not putting in a single frequency so it actually automatically gives you so if you want multiple frequencies you put in a pulse in time and then you fully transform the result and it gives you all the frequencies in one simulation that's actually one of the big advantages of doing a tiny simulation in cases like solar cells for example where you're really interested in a broad bandwidth you know the whole the whole visible spectrum and more um you can get that entire bandwidth in one simulation um you also can't paralyze over time because this is the the time dimension is serial you know you have you have to do the earlier times before you can do the later time so you can't you can't do those in parallel so you can only really paralyze over uh space like you chop up space into into different into pieces and they still have to talk to one another or over other parameters for example you know in engineering design you're usually not just doing one structure you're you're looking at a whole family of structures and you want to you know see the effect of this parameter or that that parameter that the radius of this of this uh waveguide or the the height of that of that other structure and so you're doing a whole bunch of simulations in parallel and those things paralyzed perfectly of course so you've mentioned a couple different cloud providers there are are you finding that the companies you're working with or the people you're working with that they kind of drive that decision or are you providing a um almost like hey for the type of simulation you want to do we find this to work better over here what's kind of driving that decision about who you know like which different provider to choose um well to be honest we're just using amazon at this point um we've been asked whether we uh provide our offerings through google cloud or microsoft but we haven't yet gone in those directions yet only because aws is really cheap uh you guys and your listeners might be familiar with spot instances and so this is a situation where you can rent like a multi-core virtual machine for literally pennies per hour um and so and the scale of amazon's infrastructure is just so much larger than its competitors at this point where the cost kind of considerations are really foremost among our customers at this point and so they want really it's more important for them to access the cheapest uh low-cost machines versus the most high performance and most spec'd out uh computers okay i was i was thinking more along the lines of if that there was a an interest in like they had existing data existing infrastructure or existing agreements because i assume some of this work is proprietary when you're talking about working with companies but that really hasn't come up yet with the clients you're working with they're happy to trust you with amazon well people people also run it on their own machines of course right so we have you can download and compile it people have their own clusters or or run it on you know if they have if they have their own super computer time there's lots of people that do that actually the the that that was everyone until very recently until until lord evans uh started up this thing with sympathise actually when when did you start that up i can't remember uh sippitus actually started in july of 2015 and it really started because we felt that we had been working on meat for almost a decade and we were really more than a decade more than a decade and in fact even before meet uh as he mentioned was working on mpb and so we had these set of really powerful open source solvers and we really felt that we wanted to kind of take them to the next level and to give enterprises and companies who are doing real product design access to the most cutting edge solvers that didn't have these licensing restrictions as you know licensing commercial simulation packages is very expensive and for some companies the the budget for accessing simulation software is a non-trivial fraction of their overall cost and so the idea was what if they could leverage the very best in open source software and where we could support it with consulting and technical support that they could have assurance and being able to deploy these uh tools uh and to improve their productivity so when a customer connects to you know the instances or use whatever tools that you have there on on ec2 what exactly did they get did they get you know pre-canned simulations that they just supply the input or can they write their own applications and call your routines however they wish how how exactly does this work right now what we provide is a are the latest versions of meep and mpb pre-installed on ubuntu virtual machines and in fact that's a free offering and so our customers can deploy this amazon machine image it's a virtual machine with these tools pre-installed sized for their particular applications and for them they customize it so they add on their own tools or they add on third-party servers to create some custom tool chain and sometimes we we provide them with technical support for that but at this point in time the tools are free and they deploy it for their specific needs and we just provide them with some technical support yeah so just in terms of inputs basically once you have the programs you can run it on any geometry and any kind of device you want uh it's uh scriptable actually there's a funny story i mean it's scriptable initially using scheme and recently we added on a python interface and you can also call it from c plus plus as well so you can just write uh you know right basically write programs in any of those languages that that uh control the simulation and input any geometry want and allow it to change as a function of time and extract any information you want from it so it's quite quite flexible so i have to ask the obvious question is is it scheme because of mit no so the story the story here is uh um you know before meep came this other software which still exists uh it's just more specialized called mpb and uh so i was developing that in the back in the late 90s actually the same time as fftw and at that time uh most of the simulation codes that we had that were like these large fortran codes that you have an input usually and people who are fortran users uh will recognize this every time you run a new simulation you have to recompile the code because the parameters of the simulation where where put is like code parameters and or it had and or it had some um inscrutable text file full of numbers and these in the usual fortran formulas are usually space sensitive uh that you had usually had to write a script to write the input file for this this code so i wanted to have uh my program be scriptable and uh at the time you know python in like 1997 was still kind of a you know not completely on people's radar screen uh for company first computational science and the uh the the gnu standard for scripting language add-on was guile was this is a scheme uh implementation and and at the time that was one of the ones that was one of the only languages that was really designed and documented uh to be something that you would add on to a c program and and use it to control that that c program uh you know python said you know if if i started a few years later i probably would i probably would have used python um or lua or something like that but those languages were really on the radar screen at that time i also of course new scheme because because i'm at mit and you know i took scheme as an undergraduate and actually knew scheme when i was in high school as well so i was familiar and comfortable with the language so i ended up using guile and scheme to script mpb and then a few years later in 2003 when we started working on meep it was natural to use the same this the same scripting tools i developed for scheme for for controlling mpbs sorry for controlling uh meep as well uh so that's that was the interface for many many years was this scheme interface okay so let me ask you about python because python is all the hotness these days um how well does meep play with other python numerical packages like numpy and like so so python so so the python interface is very fairly recent actually there was a another python interface that was uh done by another group uh um at get university a few years ago uh that but we wanted something that was a little bit more closely integrated with uh with the core of meep and so this this was uh recently started i guess uh in the last year it was done and then chris hogan who's here as well was the the lead on developing that so yeah it uses numpy it you know so for example you can be running a meep simulation and uh at any point in time you can say give me the fields uh as a numpy array and then i want to pl or get a slice of the field as a numpy array and then i can plot it with matplotlib and so you can use all of the python tools it uses it uses numpy uses h5 pi the the hdf5 python interface it uses it uses uh the hooks in with the mpi uh library so you so that that so you can use python's mpi tools to talk to meeps mpi uh stuff so it's it's hooked in pretty well so give us a little background here how did meep get started because if i read your web pages correctly uh this has been this is a fairly mature project right yeah it's been around since 2003 was when it first got started and it was actually started by david roundy as people may have heard of him because he wrote something called the darks version control system uh which was a you know a it came before git was one of the early distributed version control systems it still exists and he's currently a professor at uh at oregon state uh so he started this along with uh um uh a couple of colleagues of mine mikhail benesco uh and peter bermel who's now from at purdue and i got involved uh very shortly as well uh so at the time you know basically in order to do research in electromagnetism you have to have you know access to the codes and every group had its own you know musty old fortran code for doing this kind of simulation and we were no exception we expected two different fortran codes uh so david uh started developing this actually was initially called dactyl uh for two footed because it was only 2d and only cylindrical coordinates and he started developing something that would handle cylindrical coordinates and be really uh scriptable as a c plus library and then it ended up being so so useful that he and and we started adding on full full you know support for other kinds of geometries and it became our main uh our main time domain simulation code after that [Music] so you mentioned that the uh the code is open source and that you were trying to bring you know these scalable you know cutting edge academic software to the commercial space and make it accessible and that's part of the business around consulting and making accessible especially for small companies utilizing the power of cloud while also supporting people's local systems uh but the the licensing i noticed you know fftw it it says that you know you need to contact mit if you want to get a commercial license for this so can you clarify like what license is meep itself under and is it under a similar type of arrangement so meep is also gpl you know the difference with fpw is that fdw is really only usable as a library so if you want to use it in your in matlab or some other commercial code uh you have to uh and you don't want to open source your entire commercial commercial code then you have to buy a non-gpl license from mit so meep is also you know plus plus and uh and python and and so forth but uh um so far people really haven't been integrating into like large commercial packages you you control it with little scripts that you never distribute so the license doesn't really prevent you from using that in a commercial setting as is under the gpl because you know the gpl basically has no effect on you if you don't redistribute the code so if you just write a one-off script to use meep for your device it has no effect on you so that's that's not so so so the commercial entities don't need a special license to use mpb it says to use meep or mpb uh so it so art events business model is not selling licenses uh it's you know more selling consulting and other kinds of things i you know you can speak to more than i one of the things i'm really focusing on here as i mentioned is to make uh the simulations accessible to a much broader audience and part of that involves offering the access to the tools through amazon particularly for companies who don't have their own local clusters but another aspect of what we're working on right now is to make the tool easier to use one of the big challenges with engineering simulation tools is that they require typically a phd level training to have confidence that the simulations are being set up correctly and when the bar is set so high it makes it very difficult for small companies particularly startups to really gain access to these kind of cutting edge technologies and so the focus that sympathies is really to try to make the tools much more accessible by automating the key functionality that requires uh accurate stimulation so for example choosing the right resolution in a meep simulation is non-trivial uh it depends very much on the materials that are involved and the structures that are being used and so choosing the right resolution has an enormous impact on the accuracy of the result and so we're working on ways to essentially be able to automate choosing the right resolution for a given application in order to ensure confidence and typically these involve a lot of trial and error and manual hand kind of tuning which we're developing tools to automate the other aspect is actually running these simulations in the cloud so for example choosing a cluster that's sized for the application in order to ensure a very high throughput is also non-trivial and so to really leverage the the cloud you want to choose a cluster configuration that's sized just for that application and that's also a non-trivial kind of problem to deal with particularly for new users who don't have experience and so again we're developing tools that can be used to leverage the cloud to deploy neat simulations in a way that's not possible today and quite frankly it's not possible with the other commercial solvers either so what is it about me i mean earlier uh in our conversation you mentioned that there's a lot of other software packages out there and clearly you're working to make it easy to use make it available in a cloud-based environment have consulting services and things like that but what about the meep software itself what makes it unique why you know other than licensing issues and whatnot why would i use meep instead of some of the other packages one of the key challenges with finite difference time domain solvers in general is the representation of materials on the actual grid the volume the volumetric grid and uh in very early research that we did when we were developing meat we realized that the choice of the representation of these material geometries uh is very much dependent on things like subpixel smoothing um yeah so so basically these algorithms work by dividing space onto a grid and so the question is what do you do with the material that's at a boundary uh you know you have a pixel basically you're discretizing the geometry into pixels and what do you do with a pixel that crosses a boundary between two different materials and how do you deal with that accurately and so one of the things that it had early on was we developed a unique algorithm for for doing a special kind of averaging uh that makes it much more accurate uh in handling boundaries so so it has some unique accuracy features enabled in its ability to handle uh you know discontinuous boundaries with high accuracy for example why this is important is that this allows you to reduce the sun considerably so typically in order to ensure very high accuracy you really need to crank up the resolution but with this subpixel averaging method that we developed this allows you to reduce the size of the simulations while still ensuring accuracy and this is particularly important for things like shape optimization where you're varying the size or shape of your material geometry and you're doing tens maybe hundreds of calculations and you want each simulation to be as small and as compact as possible because you're exploring a large parameter space for example and so this subpixel averaging really enables the application of me to these problems that previously required lots of computational resources yeah and another unique thing is just the scriptability the ability to to write you know call it as a sequence plus library or call from python hook into numpy you know the level of integration that that's possible there uh is is i think relatively unusual where a lot of the focus uh especially in commercial codes is typically making nice gui interfaces which are also nice uh for their own you know for certain audiences um but especially in a research setting or in a design setting where you need to run lots and lots of different variations on a given design it's really powerful to be able to program it i think we keep hearing one of your developers there in the background as well yeah my my one of my well my dog is in the background so she comes with me in the to the office a few days a week and she's a little restless at uh you know not being uh not being able to play with me all right well let me ask you a question that i ask uh all development projects on the podcast here is uh what version control system do you use and why so initially we used darks and it was because uh david roundy started uh started meep and he wrote the docs version control system and i use that for many but uh and i actually like darks a lot i think it has a much better interface in many ways than good um but at some point the advantage of uh being able to use github especially so overwhelming that we switched over so i transferred all the history from darks over to get there's nice nice tools to let you convert repositories one way or the other actually that was a little tricky because we were using dark so early on from such an early version that the darks to get migration tools didn't actually handle the early commits so i had to patch it a little bit in order for that to work so nowadays it's all git it's on github we use the github you know version tracking where you have to use the the travis ci all the usual uh uh all the usual tools that people use these days okay so since uh meep is open source i assume you have a bit of a community behind it how do you guys accept uh github pull requests you know how does someone get involved in the meat project yeah so we do accept pull requests uh i mean you the way you get involved is the way you normally get involved with something on github uh you know you submit a poll request you usually i would suggest uh first filing if you if you plan or planning a feature first filing an issue get some feedback on whether on the design of that feature how it fit in with other plans and once there's the green light then go ahead and and submit a pull request with the code implementation and uh most of the large patches at this point have been done by people who work directly with me or and or artifact um but uh you know there's been smaller contributions and and there's now that we're on github which is actually a relatively recent thing in the last couple of years uh there's starting to be a wider community of people that are that are actually contributing directly patches and things uh whereas before you know people could submit patches of course before with even without github but it was you know much rarer i think for people to get involved that way it's it's much easier now that there are all these nice online tools for tracking contributions and where can people find uh contact information for your new company uh you can check out sympathis.com symphonies is actually a reference to simulations being an impetus for new discoveries and technologies s-i-m-p-e-t-u-s dot com okay well thank you very much guys thanks for your time thank you thank you thanks guys [Music] you