Constant autoloading in ruby on rails, by Xavier Noria

[Applause] okay thank you so this talk is about constant autoloading and ruby on rails so if if ruby on rails were like normal ruby applications you know that if you want to use application controller here and you want to use the post model here you should require you know the files where they are defined and uh this is inconvenient because if you require then uh you know require only loads file one time so you you should need to restart the server something like that and if you instead of require use load then you are going to to execute the file each request you know redefining constants and doing all kind of weird stuff and in the end it's kind of you know it's there's some redundancy here right so you know that ruby on rails uh allows you not to do these things and indeed it's the the idiomatic way to to write not to put requires so this thing it's you know it's automatically load and this thing is automatically for you if needed so the the talk is about how this works and it has three sections first section it's a constant refresher because constant autoloading is like a technical topic that eats everything about constant so we we should have constants you know fresh in our minds which are the rules called the recovery work then we are going to to see how constant auto loading actually um is done and after that the third section is about the the request flow how how do we get the constants out a lot and then how we get the classes out to load you know because in development mode if you change the file then the changes automatically without restarting the server afresh in your application and that kind of fast so how does that work constants you all know constants you are ruby primers so you know they are kind of like a global thing that that is just like a variable but it's supposedly not um going to change actually you know the ruby allows you to change the constants but that's like a negative thing so you know what constants are and this thing is also a constant assignment like we in the slide we said before and uh some people do not recognize this as a constant assignment because you have the class keyword this kind of stuff so but indeed that's like that's just syntactic sugar for a constant assignment that's creating a class object because you know in ruby everything is objects it's like there's a small set of actions and everything it's you know derived from those actions so in ruby we have modules and class objects and we have constants and they like kind of work together but they are like very very decoupled things indeed so when you use the class keyword you want ruby it's constructing a class object and it's storing that class object in the c constant it's the same thing in this slide we are starting one in the x in the x constant so here what happens is that there's a class new thing going on behind the scenes and being stored in the c constant so basically what happens is this behind the scenes that's that's for real so that's a normal class definition class c inherits from d and includes a module let's put that example so what this thing is doing not technically equivalent but equivalent for the purposes of this talk is this thing which is class new class new gives you a class object which is anonymous it's just creating a class object like you could have a new string right inherits from d and includes m and then the the the result of this expression is a class object which has no name and after that if you assign that to a constant and you see clearly here's a constant assignment like x equals one right it's a constant assignment so if you then assign to c then you are getting in c an object because c is not this is not syntax this is a constant like it could be a variable so this is a constant when you evaluate this expression it uses a class object that responds to the to the name method and that gives you c y gives you c if we if we define it an anonymous class because that's the way that the way the interpreter works the interpreter when he is executing a constant assignment it has code that manually says if the thing that you are going to assign to a constant it's a class or a module and the class or module are anonymous i am going to set the name that point and that name is set in stone and it's not going to change anymore but that's happening when you say when you use the class module keyword that's what happens behind the scenes so these things that we all take for granted they are constants they are not classes so properly speaking hash is not a class like this thing is not a class this thing is a constant that happens to a store the built-in thing that it's a class for uh you know that models hash tables you know so when we say the hash class the string class and we write it like this we are doing a little abuse of language because what we really mean here is that the class object that it's stored in the in the hash constant and the same with the user model and that kind of thing so um they are regular objects and you can you can pass them around store them in other constants variables pass them as arguments to methods you can do all kinds of things so ruby has no syntax for class names it's all constants and constants are starting modules constantly it's a very big topic i have like a long long talk only talking about constant so this is a refresher where i selected like the the the minimal topics that uh we need to cover in order to you know to be able to follow the follow the the next sections so this is a snippet from venus where the module class is implemented and you see the the module class has a constant table so constants belong to modules in a in a very literal way you know there's an actual constant table which is like a hash table you can think about it like a symbol table you know that stores constants and associates the constants to their value you know so constants belong to modules and top level constants belong to object so when you are doing this thing what you are doing actually is it's creating an entry in that table so the table that module m has now it's going to have an appear which is x mapped to one right it's that's the way it works and uh let's let's let's say it again this thing is a constant assignment so we have module xml class sax parser so this thing is a constant assignment and this is equivalent to doing this thing so class new assigned to such person you see this node it's equal i mean it looks the same as this thing you know apparently they are kind of different things but they are not so this thing we write it this way it's equivalent to this other thing so what's this doing the same thing this is doing this is storing a sax parser constant in the constant table of the object that it's stored in the xml constant so if we spell everything what's happening here that's the whole story and you know there's a constant api so we have cons get cons set remove cons you can ask for the constants of so that whole api it's um uh manipulating that table so cons set it's you know like creating uh you know and entering that in the symbol table cons get is like a hash fetch you know so the constant api is about those tables and those tables are per module so every module in all your application has its own table and you can manipulate that individual table with this api i wanted to cover now the constant resolution algorithms because that's crucial to a constant autoloading how constant autoloading is resolved so we have actually three algorithms in ruby for constant name resolution first one is this so uh x equal one is storing the x the x constant somewhere where in the table of m that's clear right so m is the module in a scope let's let's say that way and this x is stored in that constant table here let's say it again this is a constant assignment class uses controller so where is user controller being stored that's very important so the the algorithm here is the same for the x uh with one tweak that it's in in the implementation of the class keyword so i'm sorry right so the thing here is the the interpreter says do i have a constant called users controller in the admin module if i have that constant that should yield a class object i will check it and if everything looks right we are going to reopen that class object right so that it's looked up in the constant table of module at me if i don't have it in that constant table we are going to define a new class and the constant that's defined as a side effect of that class keyword execution is going to be stored in the constant table of modular mean that's very important because if this algorithm was similar to the to the following algorithms we are going to explain but you already know uh and and for instance if if the algorithm look it up this constant house also you know in the in the global name space we are because then we we would depend on the loading order of things you know so this user's controller even if we if we have a top level usage controller there's not going to be any problem because even if there's a user controller in the top level here define it at that point of execution is not going to be found so it's it's a strict it looks looks at in the constant table of that module and if it doesn't exist in that table you are going to define a new class then second um resolution algorithm is this one when you have a a constant path there's no well-defined terminology for this team so fully qualified name constant path several you know jargon going on in the community but let's do it let's say this is a constant path so when you when you have this thing if this is this um okay so what happens here is that x is looked up in the constant table of m and if it's not found then you go up the ancestors of m looking in the constant tables right that's what happens here and the third thing is the generic one which is the i guess statistical is the one that most people use which is that you have an x here without any qualification and that's the generic thing and that goes this way i call it the 10 o'clock rule meaning that we are going to do to go to the 10 and then we are going to go up to 12. that means we are going to check x in c does it belong to the constant table of c no let's suppose that the the answer is no we are going upwards to the to the uh outer scope this is technically called the nesting so from c if it's not in c we are going to check the constant table of n it's the constant table so if n has ancestors they are going to be ignored so we are going to check the constant table if it's not in n then we are going to check m if m doesn't happen to have the constant either then we are going to check the ancestors of c and there's a little trick here because if c instead of a class was a module then there's the object object does not belong to the ancestors of molecule of molecules then there's a manual thing in the interpreter written that says if we are if this you know this uh most nested time space is a module then we are going to check object by hand in the case of classes no because in the case of classes until 1 9 you have always object in the ancestral chain so actually in one line we have basic object and finding things happening with basic object because if you subclass directly basic object object does not belong to the ancestors of of your class and then you can if you cannot even use the object constant there or the string constant like you know like in a relative name why because object and string are not global classes or anything like that they are regular ruby constants and they follow the resolution algorithms and since object is not you know in the change of things we are going to check they are not found so you have to do a column column thing here to you to convert that relative name to an absolute path and say hey we are going to look this up in object all right so that's the constant refresher and we are now going to cover constant auto loading consonant loading remember uh we don't need to require this thing it's it's it it all happens you know on the fly so here's the thing there's a in the constant api there's a thing called cons missing cons missing is a hook that it's called if you are in trying to access a constant in some within some particular module and that constant constant is not there and well so the resolution algorithm that i mentioned before if they fail to find the constant there's there's another loop and calls cons missing on the most nested thing right so that's the complete algorithm you you you call constant consuming consuming has this information has you sorry uh okay all right uh here so we have the name of the module that it's game you know that that it's uh who's consuming is being triggered you have the name this is this is a class you know the class method that that says the name of yourself the class and module and for instance in that case the name would return the string at mean column column user controller this is a string right and then we have the cons the cons name the cons name that you know that triggered this thing and well in nine one nine is pass it as a symbol but that's not important you have the name somehow so that's that's the two pieces of information you have about what happened so active support has this uh set of directories uh which is uh which is called autoload paths in range three one i think and before it was load path but it was renamed and auto loadpass by default has all the subdirectories of adb so app models app controllers uh avp helpers and if you throw any subdirectory there it's picked up automatically presenters or whatever you would like to have there observers whatever so auto lock ha a path has this indeed i said set but it's an array because the order is important so let's say it's an array and what happens is this so the situation is that uh we are here we are here at modular mean class uses controller and then uh let's let the examples is how we auto load the user constant that's the example we are going to we are going to to see so um which this is the information and this is what happens uh first we we look over all all these directories all the autolog parts and append to the directory this this uh you know this uh suffix so if i append at mean slash user's controller slash user will be is there a file there let's suppose the answer is no so we are going to backtrack and the track uh tests whether there is a directory without rb which is something i am going to cover uh later on so let's forget this one and then there's there's some assumptions and trade-offs here that are very important that i am going to explain that they are not assumptions written in the code they are implicit assumptions that happen at that point that allow us you know to to to continue looking for the outlook path then you you look for admin user no let's suppose we are going to find a global user like which is the normal you know the normal situation right but you active support tries admin user b if it doesn't exist back track and tries also to find a directory i'm going to explain it why then there's another set of assumptions here that i implicit and finally if everything is normal there's a user will be top level somewhere probably in app models and it's found it's load right so it's load it returns the the the class whose name is user and your action whatever is executed so that's what happens and we are trying uh these things here's the thing with the directories and let's suppose we have here let's let's suppose we have some workers and we have uh organized our workers in a work directory so for instance you have a worker and then inside worker event register and perhaps you have you have other workers for some key or something okay so in a constant path constants are single names so uh all right so here in a constant path we have two constants so this isn't worker even register is not is not processed like like a single unit they are two constants and the thing is that uh you look it up the the worker constant and then inside that module you're going to find event register so we we need first to find worker and worker is autoload one out when autolog comes back with worker load then const missing is still great for event register and the same algorithm runs for event register so there are two autolots here so the first part a lot is worker and if if you think in order to have the the worker name space defined you do not need to to write a worker dot rb right you organize your workers in a work directory but you don't need to to write a worker every file in order to define the worker module if you have it it will be executed it will be fine found and executed but you don't you don't need that and so yeah so if worker rv is not is not a phone then the direct the work directory is look it up and if it's phone what happens is that active support defines a module for you so it does this in this case of a worker would we assume that it's top level thing so object is going to have defined module new and this is assigned to worker and and this is a constant assignment therefore this module new which at this point of the of of the execution is anonymous it's going to end up after this car this call it's finished is going to be to end up having a worker name so it's apparently everything it's as it was defined in the you know in the file system when this happens uh we we keep track to of a couple of things one is uh the fully qualified names of constants you have autoload why do you do you keep track of that because we want to be able in development mode to and undo this thing and you know and so that if you change the file and user will be you are going to to have this refreshed so we are going to see later how do we work do we use this information but um uh so this this there's some watchers that monitor new constants coming in after the execution of the everything and they you know they store that information for later use and also they saw the file names the file names are used to detect uh circular secular you know constants uh autoloading going on all right and kernel load and kernel require are decorated by active support why because if you love the user constant executed executing the user will be filed and the user rb file happens to require no cohidi while you are watching the new constants that are coming you should be able to say hey nokogiri is a new constant but that was was not outlawed right so uh there's a you know there's a stack of watchers that are pushed by active support while these things happen that are able to say yes these constants are relevant we are going to store these ones this car then there's a stack there's a stack of watches that are you know popping and pushing and that the taker of these things so you can autoload user you can require nokogiti there and active support is able to distinguish between the two of them and same thing for for load okay this is the way it works and this is not the constant resolution algorithm why because you cannot write you can you cannot emulate constant resolution algorithms with cons missing the first the first problem is nesting you don't you do not have nesting information and nest the nesting which is the the you know which reflects the scope the name spaces you have nested at the point where the constant was missing uh um is not passed to the cons missing uh hook in the commission hook you know who am i and i know which is the constant that is missing but there's no more information so in this example we have we have three three different nestings the first one is the regular one right and and so in at this point the next thing is m n m which is you know the name spaces we have but then if you do model um a module sorry m n the next thing is just mn there's no m so active support is going to find uh in m and user v if it doesn't find it it tries m user v and if it doesn't find it tries objective emerald but in this case it shouldn't try m user v because it's not in the nesting i mean if ruby was the one that was resolving that constant m wouldn't be checked and this one is even more able because indeed if if you give me a list of modules arbitrary i can give you a piece of code that has that list of modules as nesting arbitrary it's arbitrary interesting you cannot assume anything about nesting in this case you have module av and inside that module will be module mn which is nesting at this point m n and a b and active support has no idea of this you know it's not passed to the transmission thing so you you cannot you cannot know it so not only you shouldn't be checking m but in addition to not checking m you should check av if you were emulating the algorithm but you cannot you you don't need you don't i mean in all these examples if you put here an unknown constant the same module is triggered for con's missing but the rest of the thing is missing so you don't know and that's that's uh even worse it could be that your model in you know in the generic thing if you are writing active support you are you know you have to deal with the generic thing uh with all situations that you can have in ruby files one of the situations that the module the cons missing thing is triggered on could be anonymous and there's a there's a the there's a weird thing here the the eval family of methods push uh push okay so next thing uh i didn't cover this in the in the refresher but nesting only changes with module with the module keyword and with the class keyword and with the eval family of methods if they are passed strings in particular if you open any block any kind of lock the nesting is untouched in particular if you say class eval block the nesting is untaught the next thing is it is very very lexical thing so you if you look at the source code you you see which modules are there define it that's nasty at that point it's in the source code you can see that and but there's an exception if you do module eval or instance level or classical and you pass a string as an argument then ruby one evaluates that the string pushes the other module in particular it could be an anonymous module so this is very very it's case but could happen so in this situation you don't have no idea of what's anything you don't even have a name for the module you don't you do not even have a name so you don't have to you don't have the nesting but even there's no name so you cannot assume anything so uh these are trade-offs done by active support trade-off the name of the module that gets the the hook call it reflects the nesting that's an assumption it's known that it could be false but you do that trade-off because you cannot do anything else you you can you cannot be smarter than this and so this is the assumption that you are basically in this situation and for anonymous modules you cannot do anything so the what active supports does it assumes that the nesting is object that you are you know your lexical scope it's the top level scope so you have to do something that's what it does in anonymous modules but fortunately thanks thanks to the fact that uh blocks do not push nesting the only age case that where this can happen it's with a ball family that receives strings as arguments the other thing that you don't know it's which is the algorithm that failed to find the constant so if the algorithm is the one for relative constant names like this one uh this this thing is going to find one which is a top level constant why because it checks in m in m is not defined let's assume that ancestors there's no ancestors for them then it checks object so you find x right so here this works but if you instead of doing m x this way you do it this way the constant is not is not found why because it checks in the table is not in the table finish and you know what this thing and this thing is going to trigger the same call you know in m that x was not found and you know your name which is m but you don't know the algorithm that triggered this thing so the trade-off here is something that it's you know it's a partial solution because there's no solution for this so if active support receives a constant and checks the you know it goes up the nesting or at the assumed nesting if it finds that custom defined in some of the pattern modules then it says hey this is going to be uh this is going to be this case because otherwise the hook wouldn't have triggered in the first place right but why is a partial solution because because it depends of of the constants that are defined at that point of execution so it could be you know depending on the load order or depends on the code path uh you know execution you know but it's the best you can do so another thing that that you may you may have noticed is that ancestors are not fault so active support does this backtracking which is like following the nesting but it does not attempt to find the ancestors of the originating module and try to find directories for those ancestors and start the game right so the corollary of all this is which is a you know a very important key of this talk is that active support does not pretend to emulate the constant algorithm why because it can't so the thing is you have to to think about this in in the in a positive way which is active support provides you this feature which is if you use constants this way and you follow the conventions and and and you know and and write the files in the comm in the in the following the conventions then that that's the contract that's what's going to work if you expect that active support uh solves any you know resolution of uh constants in uh you know following the generic rules of ruby is not going to work but that's not the contract you know so uh there are these you know essential limitations and that's the contract active support um offers so let's go with uh all right yeah i think i can i can i can finish the presentation so uh request flow request flow uh depends on on cache classes cache classes you know and it's the flag that says whether you want your classes to be reload or not which in development by default it's true uh so cache no so it's false and you know in production mode you don't have to go reload and so for doing this in it was it was simpler before but in 3-2 there has been some optimization there's a thing called file update checker that that it they will given a series of directories and file names it's able to say whether from the previous execution something changes basically so it has an api that says hey your set of things has been updated from the previous car it implements that kind of thing and then we have some uh okay some hd configuration things that are not generally very used but they exist which is autoload one's path you you are able to say to active support hey autoload these things but only once so do not reload and some people use it for instance to out a lot from lip some people like to auto lock from lip but not but do not have lip reload and they put lip in out a lot one spot the only thing is that uh in a very very little use cases people want to actually say hey this constant i know it's not out a lot but please unload this constant for me because i am developing some gem or something and i am you know trying with an application and i want my gem to be reload so that you can put in explicitly unlock loadable constants uh this thing and then in active support uh sorry not result in in the application and real time there's this thing that says well if you want to reload things we are going to push a middleware and this middleware uh looks for these things the routes the locales and the application files and it works this way if the if the roots are changed the roots are out load again and the constants are gone so the way this happens the way reloading happens is that the constants that were loaded are removed from the constant tables of three modules so if root chains roots and classes are reload if locales change they are loaded and also are reloaded the classes and finally if ruby files a change or if schema rv of a structure sql change classes are going to be reload so that's the thing if if the if the file is the file uh files change uh the constants are going to be wiped from the constant table of player modules and for doing that you just use you know the the you you have uh stored the constants that you autoload you have the api to remove the constants you remove the constants by hand and since you remove the constants in the next request uh when you are going to use the user model the user constant again is unknown why because the constant is wiped the user model the user class could be stored somewhere else theoretically so you are not cl you're not really reloading the class the class could be alive somewhere else if you were stored which is a bad practice but it could happen theoretically what you remove is the user constant you if if the user constant if the user class is stored somewhere else what you cannot going to do is uh access to that class object using the user constant that's not you're not going to be able to do that but the user class is independent of the user constant could be a store somewhere else but in any case you've removed the use the user constant so when you're going to use the user constant again next in the next request conjunction is triggered again and that close group and that's the presentation all right all right do we have questions can you change the name of the constant or a module once it's assigned or is it like blocked by the interpreter yeah good question so again let's let's do that let's do the point constants are just just storage just storage and you have you happen to have a constant table in the module so since ruby allows you to change a constant you could you could you you get a warning but you can so in that sense storing a a class you know this thing like module m class c this this c assignment inside inside the constant table of m is no different than doing c equals one so you can change any of those but i'm asking about uh you have this uh the the class object for example user here's your class that you assign to the to the user constant right and you then uh like execute the algorithm for removing the the user class from the table so then when you when you for example start the user class under under a variable if you do the variable dot name does it have the name of the user and can you change the the name the name no there's no api i don't know whether if you have c code you could do some weird stuff but let's say that way there's no api to change that you cannot change the name that's controlled by interpreter all right thank you yes what would happen what would happen if you said like class dot new do like x equals self.name insi as you're defining the class yeah so you remember that i said they are not exactly equivalent so the when you do the assignment and the class and the definition with the class keyword they are equivalent as for the repositories guy but i said there's not there's some details one of these details is that that if you if you say class c and body definition when you are in the body definition the the the set the self at the top level of the body which is the class has the name define it because when when when the interpreter process process processing uh the class keyword at that point the name was set but that that that's a good observation because if you say class new block indeed at that point what this that's one of the little differences at that point the class has no name it's an animal so if you say self name uh uh at that point of the you know within the block effectively you get nil i think it's kneeling one nine this has changed the one in one of them was empty string and the other one was near any in any case within the block the class is anonymous and if you check the name it's anonymous it's when you have done the assignment that you get the name and the assignment in that example in this slide was like immediately done but you could let you could say class new store that in a variable and half hour pass then you assign that variable to that to some constant let's say c it's at that point you get the name right yes here hi uh very short question what presentations are you using sorry yeah okay unrelated question but what presentation software are you using here so that's a really nasty and dirtiest groovy script that i wrote for this conference so it's where do you find it all right all right what no i wrote it oh where do i find it so okay give it okay it's all right so [Laughter] i'm going to be ashamed [Applause] [Music] okay i publish it anonymously it looks very good okay thank you yeah hi in release to point x uh leap directory was in autoload path yes and in three it is not yes by default you can always add it back that's right and some people i add leap directory to outlook pass and some people use required dependency to allow to reload their classes can you explain what which way is better or how they differ yeah i mean um that was like a field of kind of a philosophical uh decision which was that look lip it's kind of like bender but it's closer to my application to my company or something like that and some people in core felt that lib did not belong to the lead to the directories you should autoload and if for for in production mode if you run passenger with with ruby enterprise edition or if you are in in thread safe mode uh since constant autoloading which is i mean this topic is big uh there's many things that i've left but some of the topics which are interesting is that constant autoloading is not thread safe so since since it's not thread safe ruby ah sorry rails needs to execute all your files in the autoload files in order to be able to to to have control when you so in production mode constant outlooking happens but it doesn't happen on demand so there's a in the boot process you go and execute everything constant of trolling still happens because you could execute user's controller dr v and have a country constant in the top level and that comes to country constant if country country rv has not been low it's still going to trigger some consequences so constantly closing still happens but uh that all that's all executed in uh in boot time before you start to do any threat or anything right so the the thing is there was a discussion internally should we execute everything in leap and the answer was it doesn't look safe you could have strange things in leap like templates or stuff that you don't you do not want to have executed automatically by the framework so that was like the decision to say okay let's let's keep leap out of this you know automatic thing and and you know and uh right so only go for abp no which is the best practice it depends it depends on your application and in the end at the end of the day something that that is your i mean some people like to have everything out a lot i do i put a leap in auto load and some people don't i don't know it depends on you so i my answer is there's no best practice so to do the thing that that you like okay uh one last question um so if you're a cool kid and you add like services and presenters i'm right there yeah okay services and presenters and stuff like that to your app so you put them under app um can you name space them so like you have a sign sign-up service does it have to be called sign-up service or you can call it like service column column sign-up okay so you if you have app services it acts uh as far as the things we've seen it exactly well that app controllers so you can name space uh so when when we said when we saw those examples when we we looked for partial paths those partial parts are checked against all the directories in the autoload paths in particular if you have abp services among those directories um so you you will get the namespace resolve so you can do both actually but what i mean your class and the app services can be called signup service or services column column sign up that's right all right thank you all right thanks how sorry all right okay thank you