Columbia GSAPP: Horizontal MSCDP: Conlan Olson and Noah Toyonaga

This is the first lecture of the 2026 MSCDP conversations with practitioners. Thanks everyone for coming. So this lecture series is paired with the summer intensive that you are all in now for the MS computational design practices program which started last week. Being a short intense three semester program in which students are both learning a ton of new skills and developing research frameworks to apply them within. This lecture series typically serves as a kind of introduction of various points of reference for practices that students may engage with in the coming year. Speakers themselves that we're introducing you all to are diverse pra practitioners from across computational design practice offering a range of perspectives on design, technology, engineering, and architecture. This year we'll start with Ka Nelson and Noah Toyanaga. Hi Noah, nice to meet you. Um, and who I will introduce more formally in a moment. And then for each Tuesday throughout the end of July, we will have four more speakers. Luchia Rebelino next week, Tiffany Sang, and then Lorenzo Vagi to close off on July 28th. Today, Conland and Noah will present Pattern Magic, a pipeline that converts more than 60,000 historic 2D sewing p sewing patterns from the University of Rhode Island's commercial pattern archive into accurate 3D garment reconstructions. The project aims to publish an interactive repository and tool set that lets historians, artists, and educators explore and reuse both the original patterns and the additional models. Pattern magic is funded by a magic grant from the David and Helen Girly Brown Institute for Media Innovation, a collaboration between Columbia University and Stanford University. I personally became aware of the project when we had the CDP final review last semester at the Brow Institute for Media Innovation and I just kind of saw the room that you all had converted into a giant scanner more or less. Um to introduce each of the speakers briefly, Coleman Olsson is a PhD student in computer science at Columbia University advised by professors Tony Patasi and Rich Seml. They study social aspects of computation focusing on algorithmic fairness, privacy, and applications of computational methods to the humanities. Their background is in math, theoretical computer science, and education. Noya Toyanaga, who's joining us on Zoom, is a physi physicist and artist. They earned their PhD in physics from Harvard University in May 2025. They are drawn to the mysteries and delights of the everyday, which they explore and try to explain through theory, computation, and experiment driven by geometric intuition. Their doctoral work reflects upon the genius of nature, the fatal mechanism of viral infection, the wisdom of craft, the ficundity of origami, and the beauty of the quotidian, the complexity of wrinkles in a piece of cloth. Stanford Noah studies the geometry and mechanics of soft membranes decorated by seams, a geometric motif that underlies the patterns drafted by a tailor and the microscale mechanisms evolved by single-sellled organisms alike. I'll turn it over now. Thanks both. Um, okay. Uh, thank you for having us. Thanks, Adam, for for inviting. Um, it's really exciting to be able to talk to you all. Uh, was looking up, [snorts] um, some of the other things going on in this program, and a lot of it seems really interesting to Noah and I. So, um, yeah, we're happy to be here. Um, I'm Conlin. I'm a I'm a student at Colombia over in the engineering uh, side of campus. Um, Noah, do you want to just say hi? Just make sure your audio is working. >> Yes. Hello. Um, sorry I can't be there in person, but good to be with you. >> Um, and so Noah and I are working together on um a project we're calling pattern magic. Um, and so um, so this is a picture of some of the type of objects that we're working with. Uh, so raise your hand if have you seen a sewing pattern before? Some people. Yeah, they're they're beautiful. They're these interesting diagrams. Um, and so each pattern like this represents more or less a garment. Um, but as you can see, the representation of the garment on this slide here is really different from the representation of a garment. Uh, for example, like seeing it in person. And so our project is going to be working in this space of representing garments and getting better and better at moving from one representation to the other. Um, great. Um, today we're going to talk about um, we're going to sort of take this talk in two parts. So we're going to start out actually with a zoom out and talk more generally about the role of representation in design. Um I think this both provides some motivation and some sort of theoretical grounding for then what we're going to talk about in our project pattern magic in the second half of the talk. So there's going to be two halves. The first half is going to be less attached to garments. It's actually not going to be very attached to garments at all but hopefully it tees up the second half uh well. So um let's start talking about the role of representation in design. Um so what do I mean by um a representation? Um here we have on on this slide we have two very different types of media. Um on the left hand side we have text. Okay. So text as a type of media has many representations. You could have the physical bound book. You could have a website. You could have a stream of ASKI characters that generates the website. And so these are all different representations of the same text. Um, you might be thinking text, these representations are not that far apart from each other. We sort of are quite good at moving between textual representations these days. We have printing presses. We have hypertext markup. Um, many people can read and write. So text representations maybe don't seem too different to you. But on the other side of the slide, you'll see two representations that are really different. And so what this is is um it's a uh 3D model of a flu protein. This is a glyoprotein on the surface of a flu virus that facilitates entry into into a infected cell. Uh so this this sort of like 3D picture here is what this virus kind of looks like. It's sort of stylized, but it's sort of what it looks like. On top though is a representation of the exact same protein, the exact same glyoprotein. So on top what you see is this string of I mean there's a label of of what we're looking at, but then you see the string of letters. And so what this is is a representation very concise representation of the sequence of amino acids that makes this protein what it is. And so these two representations are of the exact same object, but they're very different. And one might be useful for one thing. You know, I can go look at the um the 3D solid and look at some receptor sites or I can look at the string of um the string of letters and notice like um you know point mutations in this flu like glyoprotein. Um moving between these representations in contrast to what we saw for text is very non-trivial. So the process of moving from an amino acid sequence to a 3D structure is the subject of a whole field of computational biology. Um and um so unlike in text where we can sort of skate between the representations fairly easily, you can probably consume any of them that you want, here they're very different and moving between them is very difficult. So um this this some illustration of like how different representations could be. And I'm going to now talk about a bit more um in depth this example from architecture. So we're at GSAP. So maybe you some of you have a background in architecture, which is more than I have. Um but I want to I want to talk about architecture because I think it shows succinctly why representations are just as important to design as the underlying object, whatever that means. Okay. So here we have two representations of the same object. Um this is a trump designed by this uh French Renaissance architect um Philbert Delor. Um on um on the right here is a perspective drawing. So it's supposed to you know look the way that it would look to a to a human eye um of what this trump looks like. This is actually not a very good perspective drawing. Um if you're if you're a good uh drawer, you can can analyze it. It has multiple vanishing points. It doesn't quite nail what this thing actually looks like, but it's a shot at representing this trump. Um, arguably that is not really what Delor designed. Delarm designed something that's better represented by the picture on the left. So that is this diagram that shows you it's I think very hard to parse and requires some like an more extended caption to really understand. So, I'll gloss it a little bit, but um here is all the rays radiating out from the corner and then uh this ramped arch uh represents what the contour of the outer edge of this trump looks like. And then these diagrams are construction lines that tells you how long to make each ray. Okay, so a much more technical diagram in some sense less intuitive, but in another sense more accurate to Delaware's design. Um, this is a modern drawing that is closer to what this thing would actually look like. So, arguably this is closer to what Delor designed in his head. And finally, this is another representation used by the stone cutters who actually cut the stone to make this form. So, a third representation that's useful for someone making this this design. Um, and uh before we leave architecture, uh one more example from uh designer Frank Gary. Um, here's two buildings by Frank Garry. The one on the left is the Wiseman Art Museum in Minneapolis, Minnesota. The one on the right, um, is the Experience Music Project, now called MOP, in Seattle. U, these are both like very complicated buildings. Uh, sort of like Gary's Hallmark are these. They're not boxes. They're something more complicated than a box. Um, and these two buildings were represented in pretty different ways in the construction of the buildings. So, the Wisemen um that's the building um over here that's been on the left was constructed and um represented using pretty traditional methods. So, these are drawings from the architects who built this building. Um and they're they're drafted. Um the they were given to builders who then read off these diagrams and like cut the steel facets to make this complicated facade. Um you'll notice a lot of coordinates, a lot of very specific uh numbers that I can read and understand and measure. Um you'll also notice that the lines are largely composed of things that you can draft. So line segments and arcs of circles. Um so though the though the form is very complicated, it does have to be something that you can draft because you have to be able to represent it in order to build it. Finally, there's this uh very nice. So, Gary wants this T-shaped intersection. Uh, and how does a builder actually build it? Well, they need to know how to cut the steel that they're going to roll into the pipe and weld together. So, again, a representation of that three-dimensional form uh is sort of essential to actually making it. In contrast, this is the Seattle building, the Experience Music Project. This was an early building where Gary used uh computerized modeling methods. So he was using uh tools built for aerospace industry but using them for buildings. And so this is a very different representation from the draft drawings that we saw earlier. Now he can make these forms that are not line segments or arcs of circles. Uh there's these complicated curves that are mathematically parameterized. Um and so this this allows him to make these shapes that are sort of borderline unbuildable with traditional techniques. Um so on one hand this is a story of increasing representational uh possibilities allowing Gary to build something cool. Um on the other hand being able to move from a representation like this to a physical building uh was a whole required a huge infrastructure um you know investment uh it is not always so I think back to the theme of like moving between representations is non-trivial moving from this representation to a physical implementation required rethinking what architects had done for a long time. Um, and so I think the sort of twist here that maybe you can take away is this this claim from an architect and educator for scary that the the present condition of the phenomenon is that the building is the representation of the drawings that preceded it. So my hope here after going through this is that um you sort of believe the idea that representing your design is just as important as having the idea in your head of what you want to build. representation enables design and translating from representation to design and between different representations is quite difficult. Um okay so now we've talked you know in sort of generality about representation um now we're going to talk about um pattern magic and so Noah is actually going to start with a slight redirect to origami. >> Yeah. Um, so I guess you know part of my PhD I studied origami. So it's very familiar to me. But also I I find at least from the conversations that I've had that um it's more likely someone has folded origami than they have sewn a garment. I don't know what that says about um where we're at in the world, but uh that that is what I find. Um and so this shape here, if you folded origami at all, um in fact represents a crane. And it does so through uh this is called a crease pattern. The um red and blue lines indicate different directions of folding. Um and if you only look at the left um you might not be able to guess what this becomes. On the other hand, uh if you take a look at the right, these are the instructions for how to make that crane. And uh you see sort of step by step how to produce each of the creases that ends up you know being represented on the left. Um but you get it in a more uh digestible manner. Um and this you know maybe you might say oh well the one on the left I could kind of squint my eyes and I've folded enough trains I know what it is. Um if you go to the next slide, Conlin, uh I would argue probably this one you could not. Um so this is a origami surface that's you know been optimized to form this doubly curved uh layer and in this case you have thousands of folds all of which need to be articulated at the same time to produce uh this three-dimensional um surface. And so this is the type of flavor for what we are thinking about doing um with clothing patterns. Um uh I'll turn it back to you Conlin. >> So in a clothing pattern similarly the goal is to represent a three-dimensional form as a two-dimensional sort of plan. Um and so here we see um this is a pattern for a zoot suit uh produced by Lakma. Um, zoot suits are uh really cool and interesting garments. They um sort of were originated actually from a British cut but then were brought into uh the US and used in uh you know popular in the jazz dancing scene especially among African-Americans and immigrant communities. Um and there's a lot to be said about the history and the social the like social importance of a zoot suit. And so um I am now showing you know I have a two dimensional slide and so I want to show you a zoot suit. I'll show you a two dimensional representation of a zoot suit, but you might complain that you don't actually know what a zoot suit looks like. Um, a very experienced sewer would actually be able to squint at this pattern and know what a zoot suit looks like, but we're not assuming that's very common. That in fact, like I'm not a good enough sewer to do that. It requires someone who has seen a lot of patterns. And so this representation is sort of powerful in some ways, but it's missing the point. Um, and so here is now a rendering of what this suit looks like. Um, and we've actually gained a lot of information. Uh, we've gained, first of all, they chose a fabric and a pattern. Uh, we can now see how the suit drapes. Uh, it's more intuitive for us. We can maybe, it's not moving, but we can maybe imagine how it moves when you're dancing. Um, and you get a much better picture of this of this object. So, um, in some sense, we've lost information. for example, I don't know what's on the back of the jacket here. Okay, like maybe that's not a huge deal though because we've gained this sort of visual richness. Um, and so these these two representations are just useful for different things. I can't really make a zutsu from this one, but I also can't really understand what it looks like from this one. Um, like it like was the case with the flu protein. Um, translating from this representation to this representation is not so easy. Um, and so that is sort of the subject of pattern magic. Our goal is to work with uh these these patterns and bring them to life as 3D representations. So here you see uh all on one side three different representations of one garment. Uh this is this um very delicate u object which um you see as in physical form. I mean a photograph of the physical form um on that side. you see in pattern form on this side and the middle you see a 3D rendering moving between these representations all all arrows from representation to another are tricky um so for example moving from a physical garment to a pattern is a very common task for for example a museum conservator okay that's not our goal our goal is to go from pattern to 3D rendering um and Noah's going to talk about uh why we're starting with patterns >> yeah So uh basically this story all begins um with this archive uh the commercial pattern archive or copa for short. Um and this is a collection within the University of Rhode Island which is in Kingston, Rhode Island if anybody um has been there. Very quiet university town. Um but they have this really remarkable um library that uh was started by Professor Joy Emory um who was an avid pattern collector. Um and patterns just to be clear were never considered objects sort of uh worthy of archiving at least as they like were initially being produced. and they're being produced from the uh 1850s onwards, which exactly coincides with when the home sewing machine was uh being produced. So, Singer made the first home sewing machine. Fun fact is that uh that sewing machine was the first object you could buy on credit. So, the predecessor to credit cards and sort of home like individual loans of all type was the sewing machine. That's how important this uh invention was to modern history. Um, but if you buy a sewing machine, you need to know what to sew. And so this entire industry sprung up of producing patterns for home seists, primarily women homemakers to be able to sew clothes for their families. And like many objects associated with women's work, this was not, you know, considered uh something that should be collected, dated, um, and kept in a museum. So, we've lost hundreds, thousands, tens of thousands um of patterns over the years, but thanks to some uh collectors such as Joy Emory, the professor, but Betty Williams is another famous collector whose collections have been consolidated with the commercial pattern archive. The fashion institute of technology also collects a number of patterns which also have been collected in the commercial pattern archive. Basically, all of these sort of disperate sources have been uh put together and housed um at this library in Kingston, Rhode Island. And now they have around 60,000 to over 60,000 uh clothing patterns, which just to put this in perspective is of the same order or maybe slightly larger than the number of items in the Met Costume Institute, which gets a lot more press. So, you've probably heard of the latter um if not the former. So, what does it actually look like? Um, we had the we've had a couple opportunities to go visit. Um, Colin, could you click to the next slide? And if you go there, um, the extremely kind librarians will take you to the back and show you just rows and rows of filing cabinets. Each of them, uh, associated with a specific pattern maker and date. Um, and inside of those are these little packets of folded up very delicate tissue paper. Um, and so here's one. Um, on the right you can see how small it is. It's about the the size of your hand. Um, and this is again because these were to be distributed um to individuals and so they needed to be very compact to to be mailed and sold. Can you go to the next slide? In addition to sort of these traditional I guess uh patterns that just show you how to make one garment um they collect a lot of ephemera associated with patterns. So in the upper right hand side of this slide you'll see a number of photographs and these are photos from uh publications used to advertise patterns. So, um, Vogue Fashion, um, the the magazine, uh, actually started as essentially an advertising, um, magazine where all of the photos in it were, yes, showing you beautiful clothing, but not so that you could go and buy those clothes, but so that you could go and get the pattern that Vogue Pattern Company produced. So that it was the first time I had learned that about um Vogue when when we went to visit and it was fascinating. They also collect uh all types of other supplements and magazines and basically anything associated uh with home sewing. They also have a digital version um of this repository if you could go to the next slide conlin that you can go to online right now um or after this talk uh and you can scroll through their website and they have these things all laid out um in a pretty legible uh format where you can see you know what type of item is it uh what is the company that produced it the pattern number and then they also typically have a scan of the package package. Um, again, they come in these little uh couple inches by a couple inches packages uh which often have an image uh of the the garment either like rendered or sketched. Um, but that that's it. They don't have scans of the actual patterns, which is something we'll get to in a second. Um, next slide. So, I just wanted to walk through what you have when you have one of these patterns in your hand because I I know some folks might not have seen one of these before. Um, but this is what the outer packaging of this uh men's shirt um looks like. And so, if you click again, Conlin um I just wanted to highlight a few features. They usually come in different sizes. So, it'll it'll tell you the size. There's a pattern number. This is uh an index used by the pattern maker. Um what's interesting is that they weren't very careful and so these repeat sometimes. So pattern 9994 uh there might be three or four instantiations separated by 20 years. Um and so you just kind of have to keep track of what they actually are and that's a whole archival problem that the library thinks about a lot. Um, another interesting feature is that even though, as you'll see, this contains just one set of diagrams and instructions, there's actually multiple garments the Saturn will produce. And so, you'll see here uh on the lower left hand side, in addition to the orange shirt, there is sort of three variations that you could make. And the instructions will show you how to do that. And then on the back, um, you know, I just wanted to highlight how information dense just the packaging is. Uh there's this little diagram that tells you, you know, you need to pick a fabric to sew this out of, but for this specific design, it needs to be a knit. It needs to be stretchy. And uh this indicator here tells you exactly how stretchy does your fabric need to be. If it if it can't stretch from the black line to the white line um or to to the end of the box, uh the shirt isn't going to fit very well. So all of these little details are put together in these incredible packages. Um, and we haven't even gone inside. So, let's go inside. Next slide. So, this is the actual pattern and I've roughly made this scale slide to scale. So, you can see package inside of it is this very folded up um set of papers and the folding up presents its own set of problems. Um, but yeah, we'll we'll get to that in a second, I guess. Um, this is just what it looks like. Uh, and again, you have this incredible information density. Uh, could you click again, Conwin? I won't go through all these in detail, but you have different types of notation indicating different sizes, how you're supposed to cut this thing. In fact, this uh pattern panel on the left, uh you need to cut on the fold so that you get two symmetric um sides. You have different names for panels, different notches that indicate where things are supposed to line up. Um just all of this information that there's specific notations associated with this pattern maker that change over time. um all on this that we have to understand. Next slide. Um and then finally uh there's a set of instructions much like those origami folding instructions uh that tell you the seist here you know if I have my fabric and I want to sew shirt number one cut out this subset of the panels and then sew them together in this way. And actually, it's it, you know, if you read these instructions, you do have to pay attention because it's like skip step three if you're not making the long sleeve shirt and so on and so forth. Um, but this is all the information hopefully that you need uh to assemble your garment. Okay. Uh, yeah, and again, I'm just highlighting a few things here. Uh, this is what just one set of the garment looks like. And yeah, there's just these amazing diagrams all handone on here. Um, I don't know. I I just find them so beautiful to look at. Okay. Uh, next slide. Uh, okay. The next slide. Um, so what is pattern magic? At the end of the day, what are we trying to do? What we're trying to do is go from this thing, I'm going to hand you this package of papers on the left and I want to produce for you uh this image on the right which is a 3D rendering of that shirt. Um full disclosure, I made this um myself somewhat manually. Um and this is just to show you like what is the like the full pipeline, what is the dream. This is very work in progress. And so we're going to go through um how we are are are working through the various steps associated with how do you go from one of these to the other. Can you go to the next slide? And so we're going to break this down into sort of four main steps. Um you start with the package. You hand me the package and first I need to scan what's inside of it. Then I need to parse sort of render into a computational pipeline all of the lines and diagrams and indicators we saw in the previous um example. I need to figure out what pieces go together. We call this edge identification but in in sort of reality this is a lot like understanding what is in those instructions which edges need to get sewn together in order to make the shirt. And then finally uh you know all of this is living in the computer. we need to simulate, we need to produce this picture um that you end up seeing on the right hand side. So, I'll let Conlin go through the first couple of these. >> So, like Noah said, the first step uh when we have our our package of papers um is to at least get them into the digital world. Uh so, we have two ways of scanning these things. These things are first of all to say they're hard to scan. Um so, they're really big. Um, [clears throat] so like a pattern. I mean, generally the pieces are going to be really lots of different sizes, but like the big pieces are like several feet by several feet. Um, they're on tissue paper, which wrinkles really easily and is fragile. Um, and some of them have been cut out by someone who sewed it. Some of them have not. Um, so they're sort of a mess um to scan, which has been um like a fun a fun like physical project to work on. Um, we have two different methods that we're using to scan. Um, we started with maybe the the most uh straightforward idea, which is just to mount a camera overhead and take a picture. Uh, we mount it under an acrylic slide to sort of try to get out some of the wrinkles. Um, work on lighting it as best we can and we get, um, you know, pretty good images out of this. Um, a lot of the things later on in this slideshow are from images taken by a camera. Um but a little bit better than that is uh we also have running this big uh roller scanner which lives at Stanford and Noah runs that and uh that one gets you um you might be able to tell a little bit nicer pictures. Um deals with the wrinkling much better because it's being rolled between rollers. Um you do as before have to uh have like a a sheet coating it. So instead of a big acrylic slide, it's this um flexible um backing and cover. Um and uh and it it is it's somewhat timeconuming to actually unpackage these patterns, unfold them, mount them on the plastic, run them through the scanner, and get out your image. Um so there there is a bit of a physical bottleneck just manipulating these objects. Um and yeah, Noah and some research assistants at Stanford are hard at work at that. Um and then once we have these uh these digital scans, now we have to try to understand them as pattern pieces. Um so this is uh so we're calling this step sort of contour extraction. We're trying to understand the information that's on this scan. And so we might start with a scan like this. I mean it's already been cleaned a little bit actually uh from the raw image. Uh we do some like background removal and normalization. Um, but this is uh just some very standard image processing techniques. Okay. And then we're going to try to understand there's like some stuff out here which is just the paper. There's a bit of the slide that you can just see. And so we're going to try to remove all of that on the outside and then also flood the inside so that we're not accidentally chopping a panel up into multiple pieces of fabric. Um, so that will give you this is all still at the pixel level. So this thing is represented as this raster image and we're just manipulating pixels to try to get uh this this cleaner representation of the panels here. Um this is all done using pretty standard off-the-shelf computer vision techniques in OpenCV. Um there are of course fancy deep models to do computer vision. Um generally though uh the out of the box things have been working pretty well and um I think yeah custom tuning your own computer vision model is like not so easy and I think uh it's actually not clear that that would be much better. So uh sticking with the standard toolkit I think has been uh successful so far. Um so now we have this cleaned up but still rasterized image. Um and we actually want to understand this uh vector representations. I don't really want to think of a panel as a bunch of pixels. I want to think of a panel as um a vectorzed contour that I'm going to, you know, cut out and sew if I were cutting out and sewing it. Um and so, uh we're going to try to identify the edges here. That's not too bad. Um that yields a picture like this where every dot, black, green, or red, is um part of the contour here. And the panel is very close to what you would get if you connect the dots here with straight lines. Okay, but we actually once again don't want to think of this thing as represented just by like a very detailed polyline. Um, it probably has some simpler underlying geometry uh where these are parameterized curves. And then also we want to be able to chop it at the corners. So this thing is not just like a circle with some sharp points. It's actually like different contours that will be sewn to different things. Um, and so we're going to do some cleaning of this contour. going to try to remove the points that are not necessary. So everything in black um we've actually removed. It's not we don't need to pay attention to it. It's going to be interpolated with a curve. Uh the the red and green points remain and then I'm looking for corners. So the green points are what I think are corners and then between corners I'm going to fit a curve to my um my contour and that results in a picture like this. So this picture is now in a representation that we can use. Uh so uh we have edges that can be identified with other edges like they could be sewn together and it's a simple representation of these parameterized curves. Uh you might notice it's pretty uh imperfect. So to the human eye, this cusp here really shouldn't be a corner. This should probably all be one long line, this red and green section. Um if you look at uh what happened is that it broke on this edge, this annotation on the edge, these diamonds. Uh and so I think cleaning up stuff like this is something that you know we're still tweaking um because it is uh difficult to parse these contours. Uh that being said um you know these things are all going to be uh sewn in fabric or a simulation of fabric and so everything is like a little bit squishy and it like you don't need like pixel level resolution. Um but yeah I think contour extraction is uh working but still uh some things to to improve. And then once we have our contours uh the last step which is uh quite difficult and interesting is we're calling edge identification which Noah's going to talk about. >> Um yeah uh so this is where the break to like the frontier is. Um and so we're open to to all thoughts about how to do this better. Um basically so uh the picture on the left um if any of you have used clo this like clothing simulation design software this is like a screenshot from that where uh all of the edges um so you have the panels that you've extracted in the previous step. All of these edges need to get sewn onto one another. Um, so like this strange blob on the left with green lines shooting out of it. That is uh actually just the um uh sleeve. Um it's a short sleeve. Um but that needs to get sewn onto the front panel and to the back panel and to itself. Um so there's actually quite a lot of stitching that needs to happen here. That's a very pretty but confusing picture. A cleaner way of representing that thing is with the matrix. Um so this is the object on the right where if you uh go along either the top or the left column um you see just you know f sub e1 f sub e2 each of these represents a single edge. So in the previous step we've extracted all of the edges and so for one panel maybe you have five edges that's like FE1 through FE5 and what we need to know is that is that panel attached to any other panels and so you end up producing um in technical terminology assigned adjacency matrix um so everything is either a zero not sewn to uh a given edge a one sewn to a given edge or a negative one sewn to that edge but in the opposite direction. So you have to remember um that if I have two pieces uh I can sew them sort of together in this way or together in that way and we need to keep track of what's going on. Um but yeah, producing this matrix has been um a subject of study like a little bit. There's essentially just been um two um folks in computer science who have have looked at problems like this from one perspective um and this is the perspective that I like as a as a geometer is you can just look at the shapes of things and say aha uh for most garments there are certain rules that I need to follow like the length of one edge of a seam or the the pattern piece should be roughly the same as the length on the other side. not always true. Of course, uh if you have a skirt with pleat, um the length of the skirt that you end up bunching up into the pleat is much much longer than the waistband. But in general, that's a good starting point. And so you can throw away um a lot of possible entries into this matrix simply by looking at geometric factors like curvature, length, um so on and so forth. Of course, there's also a lot of things that you know from context. Um, if I were sewing this, I would know that um, one sleeve typically does not get sewn onto the other sleeve. Um, so there's kind of like rules associated with common sense. And thankfully, these garments aren't hoke sure. They don't actually have tens of thousands of minute little pieces. Usually you're following some fairly regular formula because again these would have had to be parsible to a semi untrained like home seist um following a set of instructions. So usually your intuition for like what does a shirt look like and what parts go together um follows and so you can begin to encode that as well um into this. Uh there's more recent approaches that use just very big machine learning um methods and we're like interested in exploring that but it's a little bit harder to understand what's really going on under the hood and quite likely it's a lot of these these rules but rendered into a more abstract form. So again at this stage what we're trying to do is produce this matrix because that tells you what edges get attached to one another. Okay, next slide. So finally, okay, we have our patterns. We know their shapes. Um, and oh, could you go to the beginning of that video? Um, I can't go to the very very beginning. Is that possible? Okay. Uh, maybe I'll just say if you go to the previous slide, if you go to the previous slide, um, you will see all of those lines uh, set up like this. and I just throw that into the 3D simulator and I press play and the punch line has been revealed but it does not work at all. Um it kind of just fails and collapses um in this manner. And so this is just to highlight that even though 3D simulation has come so far from uh you know its its uh initial phases um like we can simulate clothing incredibly well and this is the reason that like video game graphics can can run in real time with like flowy different text styles happening in them. um they actually require a still a fair amount of like hand tuning or um nuance and that's because simulation of thin sheets there's a lot of um mechanical instabilities associated with wrinkling um that happen and if you just sort of throw everything all together you get incred like crazy collisions like numerical artifacts and so you have to be a little more careful about what you're doing and so if we go to the next slide Um, this is a simulation that uh on the left hand you'll see the the one we saw before and then the right hand I've moved the initial position of the pieces around and added a mannequin. And lo and behold, basically the the shirt that we wanted um comes out. And the only difference between these two, the physics is all the same. The only difference is now there's a body so that things can collide and not self intersect. Um and also the panel pieces are spatially arranged so that they consolidate um correctly. So um that initial placement is very important. So that's the end of the the four parts. Um we've simulated we see our garment. Um and so just to conclude um to remind you what is pattern magic. pattern magic as we've shown it to you is we start with this physical archival document on the left which is a clothing pattern and we produce uh this object on the right the the shirt um as a digital rendering which is now transmissible it can be used in different media um you can download it play with it so on and so forth um but actually pattern magic is slightly more than that or different from that um next slide And it's not just that shirt, it's doing it to tens of thousands of objects. And these this brings up like a question of scale. It was easy enough for me to sort of like go and finagle and you know in that simulation put things around in a way that worked. But all of the like sort of I think very meaty and interesting questions are associated with how do you make this work um at scale? Um, and this is something that to the best of our knowledge, no one has really tackled before. And I think it's because it's a very like messy, um, nuance question where everything becomes much more complex when you're like, oh, it's not just for one. It's not just for 10, it's for like hundreds and hundreds. And so, uh, as you've seen, there's some things we've thought about, there's some things that we're still exploring. Um, we would love to, yeah, uh, continue this conversation. If you have ideas or would like to collaborate, our emails are there. And um yeah, uh I think I speak for both of us in saying thank you for for listening and we'd love to take any questions. >> Great. Thanks both. That was super interesting and such a good example of how like when you go about a kind of research proposal, it becomes incredibly complex to do many things that you would never have imagined from like the outset, right? I also really appreciate the tie to kind of the agency of representation at the beginning because that's a huge kind of through line through a lot of this work through a lot of our program and kind of transdisciplinary interipinary practice in general. Um so let's open up for a Q&A now. I have two mics which we need just so Noah and the recording can catch her voice. Um so if you have a question please raise your hand. I'll walk it over to you first to start maybe. Um so as you two know I think one really interesting thing about this project for the students is that a lot of them are going to be embarking on kind of how to think about their contribution to kind of various different practices right like how to take a research proposal turn it into a process or a technical pipeline in your case right I'm kind of curious how the experience of taking this from a proposal for a magic grant to a kind of working or semi- workinging right in progress technical pipeline like how much did you know from the outset of what was going to be an issue Noah I know that you um previous experience or some of your other work is with origami 2D representations and then modeling it in 3D like is that translatable to a certain degree into this work or was it kind of a lot of learning about it as you were kind of going Noah, do you want to talk about I have something to say also, but Noah, why don't you start? >> Okay. Um, yeah, I can start very briefly. Um, which is just to say this kind of fourstep breakdown that we've been talking about. Um, you know, you need to scan it, you need to parse it, you need to assign edges, and you need to simulate that at that level of uh granularity. this thing has been in that package since a year ago or slightly over a year ago when we were initially um pitching this. The amount of like depth associated with each of those characters, what ended up being the real challenges in each of those I think is something that we are are sort of like learning in real time. Um and that's very fun and that's how research uh like always works as as you know I said and you referenced like I have worked on various um other uh like technical programs some associated with origami which have a similar flavor you have like a 2D pattern and you need to go to a 3D thing or really on the sort of research side of things it's like given a 3D form what is the pattern that you you could use to make that um and I think that those type of questions that you find in kind of more uh hard science like applied math or physics discipline are beautiful but they're also very clean. Um you sort of cut the the you shape your question in advance so that you can produce a very nice clean output. In this case we're working because we are working with an archive with like real physical historical materials. the corners. We have no control over the inputs are so messy. Um even things like how are the lines drawn, you know, if you look at a line drawn from a pattern uh in the 1990s, that's going to be very different from that same line maybe drawn by even the same pattern maker 100 years before. And so you have to be sort of like responding to the problem in real time. Um and I think that's really unique but also very challenging. Yeah, that's very similar to what I was going to say. I was just going to say get your hands on, you know, try something like get your hands on the material objects if you're working with material objects and talk to people who know more about the thing or a totally different perspective on the thing. I think visiting the archive was a really kickoff point. >> Yeah. >> Hi. Uh thanks for your thanks for the presentation. Um I had a question that was to do with uh just being curious about like how you situate the project. say is it like in relation to say fashion and the kind of history of fashion and that kind of production in terms of pattern cutting or is it more like you maybe see it as relating to like retail and I I was wondering like do you do you kind of um is are you able to engage with people in those industries in in developing the project or is it not at that maybe that point yet? >> Yeah, it's a good question. I think so we've um we've talked to various people. We've talked to you know the the people who hold this archive who are very interested in the idea of archiving this really important piece of material culture. Um so if you're a historian they facilitate access for a historian who's studying a certain period of costume to go like look at these things. Um and I think our work uh we would love for an historian to be interested in seeing a rendering too. Like I think that would be really exciting to us. Um we've also spoken to uh garment conservators at the Met Museum uh where the challenge is um you know also sort of academic historical um but in maybe more on the with a flavor of like preserving um material culture and like making it more um accessible so that you don't you know if you can't make it to the Met to see a garment you could see it online. Um I think there's uh lots of other applications. I think early on and maybe still exciting is this idea of um you know uh realistic 3D models are nice for like video games or animations to like bring certain periods to life. Um I think uh yeah it's a lot it spreads a lot of like uh sort of humanities is maybe the target though. But Noah, do you have other things to say? >> No. Great response. Uh, you guys mentioned Clo 3D and I was just curious like did that software kind of serve maybe as an inspiration for the project or like what do you guys see 3D getting right and getting wrong? I know you guys mentioned the like so lines with the matrix earlier, but if there was kind of anything else stand out about the software, I guess like your guys' relationship to it. >> Yeah. Uh, good question. So, um, yes, I think CL3D was an important part of the motivation. um just that this exists the way that I think we usually frame it is like CL 3D meant that the last step of simulation to picture uh was much easier like we're not uh going to write a new uh soft material simulator that's you know whole field of study and physics and computer vision um and so 3D like kind of takes the very last step for us um but I think maybe another response here is that we've learned that it's quite difficult to you have to you have to set close 3D up for success you have to tell But um you know not only the edge identifications but also the initial positions like Noah was saying. Um and so I think um yeah CL 3D does do the last step but it does like the very very last step and you do have to like take get pretty close for the for the material simulation to go well but then yeah it's like I can trust Clo 3D to like uh simulate like how how a garment bounces like we're not going to try to redo that. >> Thank you. >> Thanks. Um, I can't help but think like this could be a great like database um for historians and designers. I was wondering um are the files that come out of u this whole process appropriate to be linked to a database or um a website where we can access these 3D models and help inform our projects or uh like research maybe. So I was wondering what is like the file handling around uh this process looks like. >> Yeah, I think another part of our our initial goal was exactly that to really take it to like an online database that people can interact with download. I mean for the things that are out of copyright which is most of this or large portions of this archive. Um and and maybe even like download and remix and re-upload I think would be really exciting to us. Um uh yeah, out of clo you can get like OBJ files. You can probably get other types of files too. Um I think uh one important thing to us is that um you know we're not archavists. I think we're trying to help archavists but we are like really admiring what uh COPA has done already. And so um I think yeah I mean this is subject to like more discussions with the archavists there but we would be excited about building into their system and not necessarily making a whole new one because library science again uh you know like soft material simulation is like a whole another field and I think we're we're really interested in plugging into existing libraries and databases but yes interested in that for sure. >> Um hi thank you so much for this talk. My question is more in the context of Columbia and the Brown Institute and the Magic Grant. I was curious um when you were applying for that funding kind of how much I don't know how much work had you already done on this project and also how did you frame it in the context of getting the magic grant um and just anything else about kind of about that application process like how you um positioned it and kind of like what questions or like not exactly push back but like what response you got throughout that process. >> Noah, do you want to talk about this one? Um, sure. Yeah. I mean, the uh beauty of academia, right, is that you can study anything and the challenge of academia is that you have to convince someone else that it's worthy of studying. That's like what the game is all the way down. Um, and I think that I had wanted to work with this archive for many years um before this opportunity arose. And um it just kind of I I don't know if you're familiar with the the magic grant already, but it's associated with trying to bring projects that speak to some aspect of a a journalistic inquiry or media production uh with sort of technological technical um engineering innovation and like bringing uh those things together. And so this just kind of felt like oh that that very naturally slots into that. And so to answer the first part of your question, um we did not have any work done um when we were applying. We said, you know, these are our backgrounds. We're both interested in sewing. Um like I've worked a little bit with uh certain aspects of simulation and surfaces and interfacial mechanics. um Conlin has a background in computer science and but I think what was more important at that stage was like we have a very concrete thing that we're interested in doing um and you know we sort of put that all together um and thankfully we're very grateful that they were excited about the project as well. >> Yeah, happy to talk more too about it. >> Uh hello thank you so much for sharing your project. I have a question if you have tried looking into kind of the reverse software to yours where you know fashion students they model a piece of clothing and then the software generates patterns. I have never seen it but I know it kind of exists so I'm curious if you looked into it as an inspiration. Thank you. >> Yeah. Yeah. No, it's a good question. Um it does exist. It totally does exist. Um it's very cool. um I think comes out of or I'm familiar with like the computer vision world and there are there are things that yeah you you give it the 3D some 3D representation of a garment and the goal is to try to decompose it into panels. Um this um I think is yes technically related and inspirational but pretty different in the goal because um our project as an archival project the goal is to bring this archive to life as a it's maybe it's maybe not a project about like designing new garments um but that does exist I think it it and it was inspirational um but uh yeah sort of is somewhat orthogonal but I'm happy to share references to cuz that there there is a lot of work done on that. >> Thank you. >> Hi. Um thank you for sharing your work. Um I was just curious about your general um sort of the relationship as to how you got here. Um I know you like started with like microbiology and then um went on to studying like origami and now pattern magic. So I'm just curious as to these things are so different. and how did you define that relationship to ultimately um get to this project? >> Yeah, it's a it's a great question. Maybe we should both talk um because I think we've had different paths here. Um I think so I'm a computer scientist background in theoretical computer science very mathematically involved. Um I do mostly uh algorithmic fairness and some privacy so of responsible computing. Um, so this is is maybe not responsible computing, but it is sort of the intersection of Oh, I think it's responsible, but it's not like that's not the point. Um, yeah, the the the it's still kind of in the intersection of like society and computing. Um, and in terms of I think yeah, it's it's nice that you pulled out the the like molecule simulation too. Uh, because I think this was uh or I can actually yeah, if I originally was thinking about talking for a much longer time about rep the idea of representations in machine learning which is very very deep. I I think architecture may be more fun to look at and more familiar to people. Um, but this idea of representing an underlying object uh with sort of like a an oblique view on it is is really fundamental to machine learning. Um, and Noah and I I think maybe literally met um doing a sewing project together. And uh but to get to that point, Noah, I think has had a much different path. So I don't know if you want to talk. Um, yeah, I don't want to rehash uh the my bio, but um I don't know. I I I think uh it's very impossible to give measure or explanation to what you're interested in. Um, and I think I've just been very fortunate to have a number of mentors that gave me a lot of flexibility when I was in undergrad, when I was a graduate student to yeah, just kind of follow whatever I was interested in. Um, and one thing led to another and um, I don't know, I guess the the general theme is I like geometry. Um, I'm interested in the geometry of surfaces. And it turns out basically everything is a surface and in particular all of us walking around are walking around carrying these incredible soft geometric objects on our bodies all the time that are clothing. Um and so it's I think that's um how I got to this specifically. >> Thank you. >> Uh hi thank you for your presentation. uh have you explored using AI to extract data from the archive drawings or do you know any uh AI projects AIS that are able to understand complex drawing? Um okay so uh yes so the intersection of sort of by AI I'm going to like talk about like modern big models um these things are they do help some aspects of computer vision and image parsing um in particular um one thing that uh sort of modern deep models um are useful for are reading instructions um in text form and like making ideas that are just embedded in natural language more accessible to computation. Um also um identifying text on a background is something that's been fairly well explored. I think in terms of reading like for example doing contour extraction um I have no doubt that a deep model could do this really well. I think training it and tuning it to do that is a whole another problem and require a lot of you know supervised training. Um and I think we have found that largely standard models are plenty for this sort of thing. And I think training a deep model to go from sort of like beginning to end is both like less interpretable along the way which I think is part of the goal and also would require like really really heavy training. So I think we're not relying on uh a deep model to to knock out a lot of things for us but it is useful in some things like finding and understanding text. Um I will say also because you asked about this there is a line of work um that uses deep models to generate okay I have hidden slides about this so maybe I'll show one um there is a line of work that uses uh deep models to um for example create a garment um in any representation that you like from like a text description. Um this I think is first of all very cool. Um, I think it's pretty different from what we're doing. I actually think there's like a difference in paradigm. Um, our project is trying to use computation to be able to move between representations more easily. This project, I would say, uh, dress code is using computation to sort of replace the process of design altogether with prompting. Um, I think I'm interested in this representational problem of taking a design that someone made and being able to look at it from different angles and like some representations are nice for different things. Um, but yes, it the technical feasibility like certainly I think it's been shown and probably will continue to be shown that you can use a deep model to do clothing design front to back, but you don't get to see inside them. >> There anyone else or is that it for questions? else. All right. Well, thank you both so much for coming and Oh, there are. Sorry. Sorry. >> Thank you for sharing. So, I've heard a lot of um critiques on the fashion industry about their uh beauty standards or the body standards. I'm wondering like uh cuz from the archive I saw a lot of like very generic body shape. Yeah. and your model shows like a standard white man's shirt. So, I'm wondering, are you guys interested in applying the same procedures to like for example um big size clothing kind of stuff in like more circum circumstances? Yeah. >> Yeah. I think um so thanks for this question. I think what we're focused on is, you know, the archive that we have, which does contain, um, you know, models with different body shapes, um, pictures on the front with people who aren't white, um, in some number, but definitely there are like big gaps in this archive. And I think like to the extent that we're digitizing this archive, we are going to reproduce those gaps. I think it is important though to make a pipeline that doesn't fail if you give it something that it hasn't seen before which I think is one benefit of doing this the more like like uh constructivist way like we are actually like reading the panels and like constructing them and so if a clothing pattern um you know as there are some in the archive is uh cut for a different body shape um or it comes from like a different sort of like socioultural like lineage of clothing design we should be able to handle it too because we're just reading the pattern and reproducing it. I think um this might not be true if you sort of trained a big model and you just fed the archive into it and see what comes out because then you really are going to reproduce um you know the sort of patterns that are in the archive uh forever because it's baked into your model. Um I think hopefully with this we are we might fail to be representative but only to the extent that the archive fails to be representative. And I think um yeah also like I said before part of the the dream is to have like an online archive that people can interact with. And it really is the goal is accessibility and allowing you know people to modify designs re-upload and um you know broaden like whose designs are held in such an archive. Um, but yeah, it's a good question and I am I do kind of frequently have the experience of like looking through our piles of patterns and it's just like, oh wow, like a lot of skinny white people. But yes, anyone else? Okay. Are are you both okay on time? We have a few more questions. >> I'm fine on time. No, >> are you good, too? Okay. >> Yeah. Yeah. >> Uh, yeah. So, um it just came up to my mind, but um like are you guys also interested in those? I I know um not I really know, but like I I'm pretty sure there will be some kind of fashion archive that um has incomplete patterns or like only existing in images or like a finished garments, but they don't really have a pattern. And I'm wondering if you are in at some point have interest in expending like using these kind of technology to like to that point about it. >> Yeah, very cool question. Noah, do you want to speak to it or >> um Sorry, it cut off a little bit short. I can answer the first part and come and take the second part. Um, yeah, I think I I can speak to a very uh narrow case, which is even in the patterns that we have on hand right now, and I should say that the um archive has super generously uh loaned us duplicates. Um, so everything that we're showing you here, all the photographs, these are all patterns that are duplicated in the archive. So they have two copies of them. Um, and uh, so I guess they're a little less precious. Um, and many of those, uh, why did they have a duplicate is because they've grown through donations, um, of folks just cleaning their attics and finding, oh, here's a package of five 10 patterns um, and sending them off to University of Rhode Island. Uh but because people are just finding them in their attics, the quality control is like uh basically zero. Um and many of them have been chopped up, cut out, sewn. And that's wonderful. Like you can see the history of like these things being worked with before. You can say ah this person wanted to use you know sew this garment because they cut out a subset of the panels associated with that. On the other hand it means that a lot of things are missing. Um, and so for some of these, you know, you will just be missing one piece or sometimes the pattern will all be there, but it will come in a way that's totally disorganized. Um, and I think um, for the first pass, we're going to restrict ourselves to patterns that we have the full uh, kind of thing there in its cleanest possible form. But once you do that, uh, it becomes easier to backfill like, oh, you're missing a collar. Thankfully, we've worked with callers before and we can probably, uh, suggest or predict, you know, like what caller might be used here or like, you know, put in a placeholder and say this is not the real one, but like um, uh, here's an example of one. And I think that you know kind of being able to go in both directions um aspect of the problem which Conwin spoke about in the context of of these very like large like big models is something that you can begin to do um even with this very elemental um type of of work and that's something that I'm excited about but is very far down the line. Um Conlin Yeah. Uh and then the the I guess I will maybe just give a reference for there are people doing really cool work uh starting not from an archive of patterns but starting from an archive of garments. Um I would recommend checking out both this Lakma costume and textiles pattern project. Uh which is very cool. It's online. They release their patterns for free. Um, and so they take garments that are physically held in the Lo collection and then, you know, write some like historical context about the garment and then also publish the pattern that produces the garment. So, super cool. Um, pretty different skill set because they're working with an archival garment, which first of all, you have to be really careful and um, second of all, it takes like some it's very different type of like sewing specific knowledge to go from finished to pattern. Um, but very cool. Another person uh another project doing this is dearch archive which does a fairly similar thing. They start from an archival uh garment release a pattern and produce a 3D rendering. So uh lots of really cool stuff happening. I I would be excited about doing it but it is I would say out of scope for the first iteration of this project but it's very cool. Yeah. Uh, so yeah, the tactile part and like the way you represent like stretchiness was super interesting. Um, I was wondering if whether in your project or even in computation in general if there's been any efforts to also ar use computation to archive like other sensories like maybe like sound or um sound or um uh smell like like or even in this material is that something that's been explored in like using that >> in the context of garments or >> garments or I don't know using computation as an archival method I Yeah, I think I don't know. I don't have a great reference. I would say one thing that we learned from talking to the people at the costume archive at the Met is that they're doing like deep deep archival work on like few garments. So like someone will work with one garment for a long time and they deeply understand the fabric that the designer used. They maybe understand um you know how they would conserve it, how they'd repair any damage. Um, and yeah, presumably they would understand other aspects of the garment, you know, context about where it would have been worn. Um, if it like moves, if it's designed, you know, the zoot suit's like so fun. It's designed to move really in a really nice way when you're dancing. Um, stuff like that. Um, but, uh, yeah, I think I I don't know, but it's it's a good question. I don't know if Noah, you have a thing to add, but >> um, I guess I would say that I think it's a The representation of this object sort of in all of its material glory is like an incredibly complex task. um it's impossible even if you are holding the clothing itself because these things are so fragile and degrade which is why we don't have uh you know so many historical garments and they require an institute like the Met to keep them alive and um part of that um you know you mentioned sound like how does something sound that's a very beautiful uh question that touches on a lot of different like physical aspects of the fabric what is it made of what is the weave um or knit um and how is it touching itself and all of those like questions um I can see being amenable to some sort of theoretical model but we don't have very good ones right now actually textile simulation um I guess I was a little bit glib when I said textile simulation is solved or is easy because we have clo in fact that's a very bas basic render. It's a spring and ball model. It doesn't get any of the actual physics of the garment. Right? So, if you could turn on sound and listen to it, it would not sound like a garment at all. And that's because you need to think about like what is the actual physics of, you know, cloth rubbing against itself. I think to the best of my knowledge, that's not actually something um that anyone um I'm not familiar with with research on that. It may be being done. Um but it sounds like a beautiful question. Um, and like smell is even more nuanced. Now you've introduced chemistry uh to the story. Um, so go out there and do it. It's very complicated, but it sounds fun. >> Hi, thanks for presenting. Uh, I was just wondering having gone through like all these years of archives, how did the set of instructions change and how and I heard you say that you're very interested in sewing. So I was wondering throughout these years, how did the act of sewing as well change? Like was there an assumption in these set of instructions about how the home maker like how advanced the homemaker is at sewing? I guess that's a good question. I don't know the answer to that specific question. Uh maybe Noah you should talk if you do. I will say one very cool thing to look at is uh just incredible object. This is a clothing pattern. Um, these are generally quite old, made like before 1900 generally. Um, and what this is is it's an encoding of like four or five garments all on one sheet and all the patterns are just superimposed with different line strokes and you're supposed to just trace a different line. Um, and so this was maybe made at a time when um, you know, printing like the paper the being able to print on one sheet of paper was more valuable than printing on like one to one scale on a bunch of pieces of tissue paper. Um, so yes, things have changed. You asked good questions. I don't know, Noah, I don't know if you know how like the instructions have changed over time. >> No, no, I I don't have a a simple answer to that question. Um, it's something that I think as we digitize and have more on hand to do it would again be great to look into that. >> Great. Thanks. Is that is that everyone? I think it's probably we're at time anyways. Great. Thanks both so much. That was great. >> Thank you so much.