In-Kernel End-System Multihoming with ILNP - Gregor Haywood - EuroBSDcon 2024

uh today I'm going to be talking about n system multihoming uh so I got to start off by explaining what that is uh and why it's useful uh and then I'm going to get into ilmp which is the specific solution that I've been looking at recently uh I'm going to give some details of how we've implemented this in FreeBSD and I'm got to finish up with some uh data from some over the Internet tests we did uh between St Andrews where I'm usually based and one of the ITF meetings this shows this solution working over the public internet but to start off with why do we need multi what is this problem that we're trying to solve this is a kind of Once Upon a Time picture of how we might think about the Internet working um and if we ignore mobile devices and wireless uh and uh a couple of server deployments this is still how we might think normal setups work so we have our endpoint uh it is connected to our Network at one point so it's a node that has one identity we think of it as being one unit that does a particular task for us and it has one location on our Network and so we give it one address and that address tells us where it is and also who it is if we want to communicate with someone else we need their addressed so that would tell us who this other UT is and where they are and we send data out it can find any route across our Network to get we don't care about the route we just use the addresses to find the the end point so in this example we could send data out to one to our ISB and it could then go straight to isbb and across here or to ispc and take a sort of indirect route and eventually get there still get across the problem starts to appear when we get to deployments like this so now n.2 has two points of connectivity so it still has one identity it's still one end point maybe it's one server doing one task for us but it's got multiple locations and there's various reasons people might want to do this so it could be for resilience we want to keep operating even if our ISP goes down it could be for frut we want to make use of multiple Links at once uh or it could be related to Mobility so maybe we're moving between different networks and we want to maintain connections across that so regardless we have multiple locations and it's now unclear what should happen to our address so if we're following the logic from before this should maybe be uh with it should maybe have an address of B2 or it should maybe have an address of C2 but it's unclear how we would combine those so I've used the kind of ip6 like notation if these were ip6 addresses then the B part and the C part would be rooting prefixes they occupy the same byes of the address and we can't just amalgamate them so the problem we have is that the semantics of an address are overloaded it says both identity and a location uh and so those values become sort of intwined we can't separate them out and manage them as nicely as we might want uh so this is where we get our multihoming problem if we have multiple locations we can't reconcile it into one address uh this is related to the problem of Mobility so if we're moving between different networks and our address is changing over time uh we might go from having one location to having one location still but it's a different location that we used to have so Mobility is a kind of special case of this we do the multihoming dynamic we substitute in different addresses I'm not going to look too much at multihoming today I'm just flagging it here because it's relevant at a couple of points there are a handful of ways we could try and solve this uh so we could use multiple addresses uh so if we're meant to have either address B2 or C2 maybe we can just use both of them uh and whenever we have a transport session it picks one of those address and uses that to create this connection uh this means every transport session is forced to use one ISP in one link and we can't change the address mids session without terminating and restarting and so we're stuck on one link so if our ISP fails and that's some we happen to be using for a particular transport session that session must terminate and restart um and this is not ideal especially because there's not really a problem at the network where our end points are still both connected to the same inter Network and data could still get from one to the other uh we just have this problem because we can't address things in a way that let us get round's link fail we also can't do link aggregation so we can't use both Links at once within one transport session we have to pick one and if we've got sort of two links that have lied capacity and we want sort of combine them we have no way of doing that uh and similarly we can't do Dynamic load bouncing so if uh for whatever reason all transport settings to our endpoint are going through ispp we can't force anything onto our other links we have all this capacity and we have no way of using it to solve this we could create a new kind of address so I've said Z Z2 here so this is out side of the range from either ISP uh so we can now just use one address and say we this uniquely identifies us and if anything is to be directed towards us it just has to be rooted to that address and then we convince our isps to root traffic for that address to us this lets us keep one transport session to one address which is nice and it lets us use both links so we can get this Dynamic load bouncing or aggregation or fail over the problem is we're now having to use a provider independent address so one that's not in the address range allocated to TW IP uh now typically providers will want to aggregate addresses so rather than saying uh I have this list of distinct addresses that I can uh forward traffic to it'll use a rate say I can forward traffic to everything between this address and this other address and this is good because it means you can make a very compact routing table uh it's just a check if something is in a particular range when we start introducing provider independent addresses they don't fall into this ring so we can't aggregate them in the same way uh and so now our routing table lookups have to check a range and also this explicit list of fider independent addresses uh so we fragment the address space and these lookups become a lot more expensive uh and so if every time we do a routing decision we have to do this costly LCA uh that's that's not good for the entirety of the internet even if it fixes our one problem here we also might have an extra Financial cost so now in addition to having to have two isps I get links to both of them I have the extra cost of getting this other address that I'm working so this is another approach that uses addresses uh in a way we kind of know how it should work but it's got its own set of problems we could go and try and fix this with applications so if you've looked at quick or multipath TCP this is what they do they say the approach where we use multiple addresses for one machine is just about usable right it it works uh we just need to use it a specific address for a specific link so maybe we can make applications manage that for us uh we need some kind of extended API here to pass that information around so that the application can know it has multiple addresses to use and of inject that information in and switch between things um but this potentially gets round this problem um assuming we can find application protocols that we can build the applications that do this we can Define this API and then we can deploy it to every single application that's ever been created um so things like quick work quite well if you're Google we've got a big team of developers and you just need to fix chrome uh and servers and then most of what you worry about Works uh if you've got a smaller group of developers uh then it's a much bigger ask to suddenly learn a new protocol stack and redesign your system to use that um this is also a kind of protocol nism and that we've not actually solved the problem we've just said maybe some else can solve this problem for us uh so if we could find a way to solve this problem at the network layer that might work better it lets us keep using UDP and TCP without having to rewrite our applications one last approach that I'm not taking here is to use middle boxes uh so there's a whole host of attempting Solutions like this that do some kind of middle box at the border of your network that manages the uh multipath uh protocol this means you don't need to do anything your endpoints they can still act like they're single homed and you don't need to do anything for isps they can still act like they're just fing traffic to a single address um we hide the complexity in our middle box um the problem with this is that hiding complexity introduces new problems so particularly with something like Mobility if I'm using your uh Network and it's behind a middle box then I might have a particular idea about what my connections to the internet looks like that means I can handle the step onto a different network if I'm moving onto a different Wi-Fi network or switching on to 5G I might handle that in a particular way because I think I know how my connection works but it turns out because of your middle box I actually don't um and so that then falls apart so we're introducing new complexity that we can't see or control uh and it might break future Innovations or lock us into mindsets that uh sort of prevent us with from bringing new Innovations so instead a solution that we might want to look for is something that is in curle so we can deploy a fix to our operating system through whatever mechanism we use to update the OS and then it will work for all applications we don't to worry about getting every developer on board we want to do something that doesn't involve creating a new API so again we don't need to change applications we can just deploy this and it will work kind of of the box if an application uses the up updated operating system everything will fall into place we want to do this on the end system rather in a middle box so that whatever complexity we introduce we can see and control and account for and anything else that we do and we want to reconcile these last two points we want to have multiple provider aggregable rooting prefixes uh and we want to have a single transport layer ID so we can have one transport session that uses multiple roting pref uh and this is where ilmp comes in this is the identifier locator Network protocol uh I've got some links at the end to the rfcs def find this in detail I'm just going to give a very quick overview of this so that we can then see how we implemented it I'm going to contrast this with IPv6 there are ways to do similar stuff with IP ipv4 my implementation is the ip6 based one that's what I'm going to focus on um ip6 addresses are formed of two parts so you got a rooting prefix which is the first 64 bits and an interface ID which is the second 64 bits uh and I I colored them here to say the rooting prefix is a network layer value it identifies uniquely a network uh and the interface ID is a kind of Link layer value it uniquely identifies your interface on the local network and when you combine these you get a value that uniquely identifies the network you're on and uniquely identifies your device on that Network so it's a unique address for your advice on the internet ilmp slightly changes this so we use the same syntax uh and we keep the same uh rooting prefix so it means the same thing it still identifies our Network we're just going to call it a locator or an l64 um and instead of using using an IID we're going to use a node identifier a n um this is in green a transport layer value and that we manage it at the transport layer and we propagate it down through the step rather than identifying an interface it's identifying a node so we then add it to every interface that we were using uh and it gets managed at another layer uh to get into a bit more of the semantics of uh these values um need to use the same syntax as iids they can be managed in the same way so you can integrate them with slack and labor Discovery protocol um and we can make them random in Emeral so there's work to do similar stuff of iids um with nids it becomes a little easier to manage because we can integrate that management with our transport sessions um it identifies a node so this means no matter which interface we're using this value will be the same even if we then move between networks or routing prefix might change but the nid will be constant so we can use this as the invariance in our transport sessions it's the indicat the end point uh L 64s on the other hand uh are excluded from transport state so although they have the same syntax and semantics as routing prefixes and ip6 um we we remove them from transport State we just ignore them we don't count them in check sums uh we don't use them in PCB lups they're not part of the end System state um this means that they are mutable so we can change them and subtitute in a new one uh and our transport session can still continue to operate we just need some way of gluing these together um which means The Binding between a nid and an l64 to populate the uh address fields of an ilmp packet is managed dynamically so we find whatever nid our transport session is using and we find whatever L 64s happen to be available to us and then we join them together we send P out we also need a little bit more signaling um this is more important when we get to um Mobility where we want to dynamically change the set of available locators um but we can use it for multihoming as well which are locator updates these are new icmp messages we've got the I Cod points for them um and these let us update uh data structures within the curl that we use for this Dynamic binding so this means that if the set of locators that I am using for my endpoints change I can send a message out to any endpoints I'm communicating with and say stop using these locators and use these new ones instead uh or even just you can keep using this locator but also add these other ones to the set that's available I can sort of announce my presence on these exal links and because we can do all this Within the kernel um we can do it without having to change the uh API so applications don't need to worry about anything so onto some of the implementation details uh so I did this work in FreeBSD I did it as part of a PSD this was my first exposure to free BSD uh which meant that the first time I ran a BSD system was one I compiled myself which was possibly not the most sensible way to approach this problem um but it was certainly a way of learning a lot uh the general idea of what we've tried to do is keep all the dynamic bindings in a new data structure we called it the ilcc this used to stand for identifier locator communication cache but that's a misleading name because it's not actually a cache so now it's just the ilcc doesn't stand for anything um whenever we send anything out with TCP or UDP they consult this data structure to get the latest Dynamic binding use that to create their packets and send them out um and then we can update the ilcc with locator updates to get information about remote nodes or with rout advertisements to get information about the uh locators available to us locally so because are split out into these different modules uh and then uh add them in bit by bit so to start off with a bit of a closer look at UDP uh this turned out to be very very easy um so the UDP code uh was easy enough to make sense of uh all you had to do was change the uh way PCB lookups work when you receive a packet uh and which will come back to in a moment and uh update the PCB with the latest Dynamic binding so if you tried to send a packet and then when the packet got generated for being sent out it would use the latest values from that PCP uh and so would use the latest uh locat the protocol control box are stored in a hash table now normally the hash is a hash of the address so you receive a packet you hash the address to come from uh that gives you a list of sort of candidate protocol control blocks and you search through that to find an exact match uh and you match the add the local and remote addresses and the local and remote ports because ilmp excludes the locator values from the end System state and because that they could have changed since we created the protocal control block um we just use nids instead of addresses um which means if we we want to do this lookup correctly uh we first have to do a hash of the nid instead of the address but can then use the same data structure uh and then when we search through it we just want to match the nids instead of the full addresses um so again it's a very small change um and we can put this in uh conditional compilation so it can just pile away if we don't want it in order to do this to know we should do this lookup instead of the IP 6 one we need a way of identifying packets uh and that's where the noner comes in so we add an extension header uh which is uh we we use the ip6 extension header framework you have specific ones defined forp uh they include a nons which helps with authenticating our locator updates um but the useful bit here is that if we find it if we receive a packet and it has this extens header then uh we know to use the alternative Lup um this was again surprisingly easy uh so the extension header code was all there in the PCB if could just uh initialize that with the right uh layout for the header and it would automatically get added into every packet we sent and this was surprising um because I tried to look through the kernel to find some other codes that use this functionality and couldn't so I don't know if there was other functionality and it got removed or if this was just uh written into the operating system because it's uh it's standardized and it just happens to never be used until now but I am grateful that it works because that that saves me a lot of difficulty uh and then the last bit we need to sort of glue all this together is some stuff for session initiation uh so uh there's a modification to get add it says if you're getting information about an ilmp node it has different DNS entries uh we just use ety hosts um this means the get add info call works out that it's looking at anmp node it populates the ilcc with information about the remote node and then when we try to connect to it um the connection will check if that entry exists and if it does it will create ilmp session instead of ip6 um we set the nids explicitly so we say this node has to have this nid we uh have a tool for that eventually it might make sense for that to get integr integrated into if config but there are still some details of exactly how those semantics would work that mean we've kept with using a separate tool for that for now local L 64s are added based on root advertisements so it's using iv6 this stuff has to be getting sent out you pick up these to work out what is available on the network you are on uh and which router is responsible for forwarding stuff from that prefix um there's a little hidden complexity here in that we also need to track which we we don't just need to track the locators that exist we to track which interface they're advertised on and what the next hop router is make sure that when we send packets out uh they go to the router responsible for forwarding stuff from that locator um if we don't do that there's a risk we try and forward traffic based on just the destination and it gets sent out by a rout we are connected to but with the locator for the other one and that's likely to get dropped as a kind of uh spoofed pack packets defense uh but we can handle all this in the LTC uh we just parse the root advertisements and add that in and then we specify the next hop when we go to Sender package when we try and start a new connection it defaults to trying to use ilmp it's all enabled and it will check that it has the information about the remote host that it has at least one nid that it can use uh and that it has at least one local l64 to use and if all of that is true it will default top otherwise it falls back to ac6 you don't actually need to set nids explicitly so I've got some experimental stuff for maintaining a pool of available ephemeral nids um my research is primarily focused around using this as a way of achieving privacy benefits uh and this was looking at generating uh per transport session addresses so the you create a unique nid you use it once for One Transport session and then afterwards you uh decate it discard it before anyone else can use it um this is easier to do with FMP and IPv6 because we can manage it at the transport session which me we know when it's in use and when we're done with it there have been efforts to do similar stuff by modifying interface IDs at the link layer but that's much trickier because you don't know if it's still in use and so it's much more costly to try and expire them um and then there's TCP so all of what I've just said works for UDP and most of it is also what you use for TCP um but TCP is much trickier than UDP and there's a couple of reasons for that so although in principle we're doing the same thing we're using all of these structures and the same way of updating the dynamic bindings and we just need to do the thing we did with UDP when we send packets and the thing we did with UDP when we receive packets um the code is much fidd so the UDP V4 and the UDP V6 functionality is in two different files so if you're modifying the UDP V6 stuff you don't need to worry about any of the V4 stuff that's not the case for TCP uh it's all in uh the same set of files so each function will handle both I um ipv4 and IPv6 packets um which makes it more complex what makes it even more complex than that is that it's conditionally compilable so you could compile that without the ipv4 stuff or the IPv6 stuff and I was trying to add another layer of conditionally compiled codes that would only apply if it had IPv6 support uh so there was just a lot of condition um pre-processor commands to try and make sense of um the other big headache was Sy caching so to prevent certain Doos attacks when you receive uh an initial sin message to start a TCP connection uh you don't create the full protocol control block you just create a small entry in a sin cache and if the TCP handshake is successful then you create the full protocol control block and this is to stop someone sending a bunch of sins to make you generate your big table of protocol control box and do all this work only to not actually have any real connections um the problem with this is that I'd already done the UDP approach and everything was engineered around the assumption that I had a protocol control block to work with and now suddenly I didn't so I had to go and generalize all my codes to work with some other structure that would refer to it to make it work for both the sincast entry and the PC the PCB um but we got there uh and this does now work uh one last comment to make about this is there's an open question around how we choose which locator to use uh so if we've got multiple paths the question is which one do we use if one goes down obviously we can fail over to the other we probably want to do kind of even usage between them in some sense um to try and load balance but it's not immediately clear how we do this and how we detect this um so that's an area for future research the current approach is just to send a packet out on alternating locators um this was again a privacy experiment so about disrupting real time uh traffic analysis um it introduces some unexpected problems which we will come back to in a couple of slides um this is also all oblivious to any kind of congestion control approach so congested control typically will see loss we'll take that as indication that there is something going wrong on the path between your endpoint and who you're speaking to and start sending less data to try and deal with that congestion um if you're using multiple paths then you need to do something different um because you no long you now have even if you just have two locators two paths if you have two locators and the other node has two locators you have four different paths Each of which will overlap a bit with some of the others um and so the the states you have to manage for path selection and congestion control becomes much more complicated um we've not looked at that in detail yet uh so far we've been trying to check that this is a um a viable approach and it works over the public internet uh this is the next step um there's also some interesting questions about what you do with retransmissions and acts so if you've lost data do you retransmit it on the same link that you initially sent it uh to sort of treat it like a retransmission and it's exactly the same thing happening or should you retransmit it on a different link um so that you're less likely to hit whatever caused you to lose it or is that compacity not worth worrying about and you send it out on L link is available similarly do you act data on the same link that you receive it or do you act it on a different link which then risks it being lost do you act data from across all links in one message um so there's a lot of stuff that we could explore here that's questions for the future we have a working prototype uh the codes up on GitHub there's a QR link the end in one of the later slides uh this is based on previous D14 and we're in talks around uh trying to get this merged eventually uh once we've sorted out the last couple of caveats it is now uh stable it doesn't uh so it we can we can use it for extended periods uh it's probably some of these control issues that might be an issues to wider adoption but like I said they're an area for ongoing work and they don't always matter so much got to finish up with some data from a demo so uh for this demo uh we sent some kit over to an iatf meeting in Vancouver and we kept the server running back in standers uh so we then did some tests of I we've got the I server uh it's then connected to the global internet and then at thef hackathon um we had an edge router that was again connected to the internet and our Edge router was connected to our client thisum and I we had four different links here so it's sort of at the last hop we've got this um multipath functionality um and for the the demos we started off just using one link then 20 seconds in would bring another link online then another and then another and then gradually turn them back off again to go back to just using the original link so this would show us sort of going from just using one link as normal to a sort of full four path multipath setup and then switching back down again the goal of these experiments was to check for connectivity so we weren't sure what would happen with extension headers as we sent them over the internet and also with check sums that didn't include rooting prefixes so the main thing we were measuring is can we send data from A to B uh this was not intended as a performance test so we don't have baselines for uh IPv6 to compar us with um that's something we'll want to do in future um but that that was not the purpose of this test which is why we've talk about that kind of data here first up UDP so we've got fruit plots here for each interface uh over the full duration of our iper run so as you see we start off sending everything on one link and then fure drops when we bring our second link online so we've got some kind of limit that's uh saying this is the maximum amount our application will send um but it will split across the available links and then when we bring the third link online we' now got a third here third here and a third here and then when the fourth one comes up we split to doing Quarters on each link and then as this goes down we go back to a third on everything as this goes down it goes to 50/50 and then back to doing everything on one link um so we get this very even uh spread of load across all our links um which is uh nice right we we are taking full advantage of what's available in that we are not uh sending everything out on one link and creating more congestion there um we will hopefully be creating less congestion because we're we're load bouncing uh and if we Agate all this up and look at the total throughputs it looks like this so we're not getting any drops when we have our hand over we can just keep using all uh one two three or four links and then three to one uh and it's not impacting the application at least in terms of frent um and then there's TCP uh so the alert amongst you will notice something weird happening at 90 seconds uh I when I point out this is a log axis you'll realize this is very very weird uh so we're fine all the way through everything coming up and then things start going down and we're fine for a bit and then suddenly we get a drop we get a drop in frots of an order of magnitude uh and take a while to gradually recover uh until we get to sort of this bit at the end with just one interface uh it looks even weirder when we get into the uh looking at all four interfaces um so we start off with this bit uh when we just got one link uh the performance we get on that link is the same as the agregate as You' expect then when we go to two links we start to get this sort of much spikier pattern uh even though we're still getting the same kind of performance in the overall aggregate performance um so sending packets out on Alternate links is not quite giving us this sort of smooth dip it does have bursts on some it doesn't have burst to the same degree as mptcp if you look the data from that it would use uh full congesting window on One path and then switch entirely to another we've got a bit of a more even pattern for the most part until we get these sudden drops uh this one's happening at 70 seconds here others happening uh earlier this does not map onto a drop in performance in aggregate so this is we stop using one link and switch to using another um and then uh this one comes on it drops off at 80 seconds and then 10 seconds later we start having problems we are still exploring all of this data um but we were not particularly surprised by this so uh what we believe is happening is that we're getting uh some loss in this hand um so the that link goes down maybe an act gets sent to it and is lost because that link is now down um uh so we detect some loss we retransmit some stuff uh and everything falls apart because now our retransmissions are getting spread across three interfaces and are arriving out of order and that the out of the out of order transmission is then getting interpreted with more loss uh as we get the sort of spiral of everything falling apart um like I said this is not surprising um we we are not doing anything at all with congestion control to account for the fact that we're using multiple paths um we've just tried this out to see what would happen um if we had multipath we congested control and we're selecting the locators appropriately to manage that then we'd expect this to be smoother and to not have this Dro in performance because we would have defined a way to manage it rather than just hoping for the best uh so the key takeaways from this are that we can do n system multihoming uh the UDP results show that we we can just switch between interfaces um we can send this across the global internet so there might be middle boxes on certain Enterprise networks that cause us issues but across the public infrastructure at least um this works and we can start using it and deploying it we can do Fair multipath where we utilize all the available links and spr traffy across them um but we've still got some work uh to now get to with multiware contest control and locator selection um but we cannot do that because we have this working solution uh there's a QR code there to the code release that also has some links to some uh papers uh and some data from a previous set of measurements we did we'll eventually release the data behind those figures we're just in the process of dissecting it and writing it up but I'm going to finish up there can take any questions are you planning on um making this in any way available or visible to the applications for example by picking a reserved IPv6 prefix and then you can sort of of like Ping a note ID or something uh so the work so far has avoided doing that so we we've very intentionally avoided to making anything visible to the application um the idea being that we want it to be Deployable as a kind of colel update if we did want to do that we'd probably do it through an expand expanded uh colel API so if if we did Define some new functionality we'd say s here's a modification to the way you start a connection is there any way to make this work if the network forces a specific host ID on you for example using stateful dhcpv6 um yes so um because nids work like iids if we did have something like DCP to manage them um you could allocate that out and use that as a nid um it would start to give you issues if you were multihoming across different networks that were both doing that um because you couldn't use the same nids across them then but um if you're if you had a server say that was single home or that you was you had dhp sort of on both networks but it was aware of what it was doing uh then it would be fine yeah because middle boxes are always so um mindful yes but a lot of IPv6 stuff is slack based anyway so a lot of the time we can avoid that yes I I'm just wondering those results that you saw with the TCP uh did you consider that it may have been due to the packets coming coming in out of order yes so we we think that we're getting this sort of out of order delivery uh and that uh especially once we get to a higher number of paths we think when you've got a couple of paths it's low enough that you don't get much when you go to higher path you get this out of order delivery it's going to get detected as sort of uh lost packets much more quickly and we'll get that Dro in performance we were surprised though because we were just using um four Links at the edge router we'd have expected that to be very minimal um so we need to look a bit more to see what's going on and maybe do some fall off experiments at the next ITF it would be interesting to just see uh what the percentage is on higher speeds then what that retransmission is yes and then uh just one last question uh did you consider doing the same experiments with something like ESP traffic uh no so one maybe more question think yeah two minutes left did you uh I wasn't sure if you had to do these updates on both end points so both endpoints need your changes or just one endpoint does uh so so far we've just done them with one end point um at least that's what we were doing in these experiments uh I have done some smaller tests with doing on both so in principle that's absolutely fine because they're both using the latest the endpoint State you can change either set of locators and the protocol at least is Happy um the slight risk is if we are communicating and we both change networks at the same time and do a hard handoff and we both send out updates to our old networks we could both sort of migrate out of our own connection um but as long as you're not both very highly mobile and if if you can sort of manage some kind of soft Handover then that should work so like one minute left oh so more questions or yeah 30 seconds left uh yeah so I'm wondering how big is the the actual diff and the change how how hard do you think would be to integrate it uh so we've got it's it's sort of fiddly because it happens in a bunch of places right we've got this uh so all the dynamic binding stuff is in a new file so there of one big file to add um and then there's a bunch of um um sort of stuff between hash defines for like a hook into that from the roof revers code and it hooks into that from sending receiving UDP uh add into that for um the TCP code I can't remember off the top of my head what kind of line number we're looking at but the the problem it's it's not a huge diff in that most of it's just most of the functionalities contained to this file but it does touch a lot of places okay thank you okay so that was the time really up signal uh so um thank you for your talk and engag a lot of questions and too many questions time wise uh and yes uh on behalf of organization are so it's a traditional a star for traditional writing on it e e