Data as a Creative Constraint - Eric Socolofsky

hi um so we're kind of in the doldrums of the afternoon we just had lunch two full talks ago so you'd like to stand up and stretch for a moment please do I'll just prattle for a little bit while you shake it all out um okay sit down uh so I've been wanting to come to open visc comp for quite a while this is the first time that I've been here and I finally figured out how to do it which is to have Irene pay for it so thank you for having me um and I was talking to Jim yesterday and I understand that that from the gym that the art talk is kind of the art talk is kind of an uncommon thing here so that either means the bar is very low or it's very high and I'm not exactly sure but we'll find out um so I wanted to talk about this concept of using data as a creative constraint to an artistic or creative process um and uh I think that as as creative people here in the room we're all familiar with the concept of constraints in a creative process or an artistic process Charles Zs famously said that design depends largely on constraints constraints give us a wall to lean against they give us they they they limit the number of variables that we have to balance and give us a sturdy framework upon which we can experiment um so to bring data as a constraint into your process does not mean that the data will dictate the form or the output but it kind of offers us a raw material that we can um use uh to chisel down into a new artistic expression so that's this concept of bringing data into your art um and that was the sort of founding concept that I was going to base this talk on but I realized that the way that I do work does some of that and then it also actually goes the other Direction which is to bring the artistic perspective to your visualization um cute kid huh uh so uh data visualization fundamentally is communication um communication is a fluid medium takes on the characteristics of the communicator of the the content that you're working with and and often of the audience itself so mclin's concept of the medium being the message I think is very uh is very true for data visualization as well the techniques that you use to visualize your data have an outsized impact on the perception of the data so if you bring artistic thinking or presentation into your your data visualization practice then you're you're offering new avenues for communication about your data so we talked about bringing data into our art and I think that that generally results in what we refer to as data art um which is uh soort first and foremost it's it's art right the intention is to generate a an artistic work um and then the data are used to augment the art in some way or to constrain the art and we can go back the other direction which is to bring an artistic process back into our data visualization work um and this uh I think results in what marit Stoner most recently referred to as Artful data Vis in applied context or bespoke visualization um and these Concepts this way of moving back and forth between data and art are is a way that you can think about how you produce your work but it's also a research technique so if you expose yourself to both processes sort of in in this cycle I think it will uh add richness to the work that you present even if the work that you present is not necessarily visually complex um or you know if it's more sort of succinct and um and legible is is your focus then this can still I think apply in any way of thinking about your work so in terms of Art in general um I think the the area of art that is the most applicable to data visualization in my perspective is generative art um generative art means a lot of things to a lot of people I'm not going to try to Define it exactly but I'll try to sort of describe it um it also has a lot of aliases so it's known as algorithmic art creative coding is a newer term procedural art and even just computer art whatever that means I guess it means generative art um and uh there are a number of characteristics of generative art that I think help to describe it so the first is that of course it's generative right you generate something through a process and that process usually uh employs a set of rules a system within that which the rules sort of bound the transformation of your input into some sort of output um and in generative art uh Works generally or not generally actually sometimes you have emergent Behavior so when you feed something into this set of rules you don't necessarily know what's going to come out the other end it's not a requirement of generative art to have emergent Behavior but I found that some of the most compelling works of generative art do exhibit emergent Behavior think in the concept in the context of data visualization emergence is not necessarily something we want because if we don't know how our end result is going to look then how do we know how we're going to convey and communicate to people so um there pros and cons depending on how you're looking at it um sorry I have another bullet point slide and then one more after this we'll get through it uh there's a number of techniques that I think are applicable from generative art to data visualization can think of algorithms Randomness and probability particle systems recursion and repetition motion and color so I'm going to dive into each of these at varying levels of depth and look at how they're used in generative art and then also how they can be applied to visualization we'll start with algorithms um so algorithms can be used to illustrate behaviors of complex systems you can create an algorithm that sort of defines the way uh elements uh sort of appear in your work um you can use them to generate textures to indicate some quality of the data or even to act as a backdrop to the data in which case the data become apparent as sort of negative space against this algorithmic texture um and you can use algorithms to generate empathy if you use algorithms to add character or personality to the elements of in your your visualization or your artwork then you have the opportunity to introduce an epithetic link between the data and your audience so this piece in the background here and this one are both solowit pieces uh solowit is a generative artist that I think a lot of us are familiar with his work um primarily in the 20th century was uh concerned with drawing things on walls and Galleries and the way that he would go about this is to write a set of rules and present this written set of rules to an artist or a group of artists who would then execute these rules on the wall um often with pencil or uh some other form of marking um so the rule set for for this piece is lines while drawing 69 lines not long not straight not touching drawn at random using four colors uniformly dispersed with maximum density covering the entire surface of the wall so this is one of many examples of how this set of rules could have played out in a gallery context um the point is that it's all about this uh sort of process that emerges or this this piece of of of art that emerges from the rues set he's created casus is a oh no uh internet sorry I oh internet no internet do we have a wired connection I'm sorry I totally forgot about that detail bear with me oh have you guys seen this can I do it it doesn't work it in bed when you don't have internet and you load up a page uh that shows little dinosaur because um in Chrome uh you can if you start messing with your arrow keys and the dinosaur comes to life and you can start playing this little game where the dinosaur runs around and hops around so you can't do it in the Ed player so I can't Entertain You while we get the internet working what's that I don't want to do that okay let's try refreshing maybe oh no I'm sorry totally forgot about this detail and I can't see where system preferences is is this it YES Network where's Network connected sorry well I should have to use Wi-Fi right I should be able to should just go right it looks like I'm there let's try one more time oh go back all right we got it thank you okay Network thank you sorry um where's my M I need it okay so this is a piece by Casey Rus um he also follows a very similar pattern to solo wit before I get into that I'll say that he might be familiar to some of you is the co-creator of processing with Ben Fry at fathom here in Boston um and Casey's practice is very similar to Sola wits and that he will also set up a set of rules and then instead of having artists execute his rules he has software ex execute his rules um and he'll often even display his rule set alongside his his works on the wall in the gallery uh so the rule set here sorry is a rectangular surface filled with instances of element three which is another rule set each with a different size and color draw a tiny transparent circle at the midpoint of each element increase a circle's size and opacity while its element is touching another element and decrease it while it's not so we can see similar to solid wit that this pattern emerges this this artwork emerges based on the set of rules um here's an example that is more directly tied to data visualization this is a piece by nervous system also in Boston um in which they have an algorithm that generates what they call a highly controllable and isotropic macroscopic foam structure sorry Amanda um it's both lightweight and strong it's a cellular structure and what they do is they adapt the density of that foam they adapt the density of the foam per the foot strike of the runners uh they have the sample individual runnner foot strikes so we'll see here in a moment the areas of highest force from the foot strike are the areas of dense of the highest density in that foam that generated with an algorithm so the presence of the data actually manipulates the algorithmic texture here um this is a piece that I did for staman for uh news visualization showing a news organization that shows um the stories uh that are getting the most views at the center of a cluster that's determined by all of the other uh stories within that section so this algorithm is a simple clustering algorithm that lumps together everything that's in the same section of the paper um but by using an algorithm applied to a particle system I can then sort of remove that algorithm and allow the particle system to fall back into a more uh standard layout um Randomness and probability is another technique that uh is used to great effect in generative art um I think less frequently in uh data visualization but we can use it um we can use it to indicate natural variation that's inherent in the data for which the quantity is undefined or perhaps not as useful to understanding the patterns within the data um we can use it to represent buzziness in data so data are observations fundamentally and they're not always precise like analog instruments for example have a lot of inherent fuzziness uh and then uh or noise um using probability to represent probability itself is something that we often have to deal with as data visualizers this is a very early example of a generative artwork uh in which Randomness is very critical it's called musicales Vel apologies to any native German speakers um and basically what you do is you roll the dice and you hear things if the video play please whoa all right no play [Applause] um doesn't sound particularly random because it's Mozart uh and it's a bunch of individual pieces that are stitched together in the order DET determined by your die rolls uh jumping ahead a bit in time this is an moving from sound to visuals this is the a simple random walk algorithm which is used quite often in generative art it's a way of using Randomness to move through space and essentially the algorithm says move forward or backward or left or right a random amount every every uh step or every cycle uh looking at how that can be applied in an artistic concept context we have this piece by Guido Coro called random walk triangles in which there's a number of individual colored triangles that just randomly move around the screen and leave a little trail behind them so this one is maybe not um yeah anyway um and then uh Plankton populations is a piece that I worked on at the Exploratorium and which we show the populations of different uh types of Plankton in the world's oceans um as colors on the map like a biome map and you move this lens around on top of the table and when you look into the lens you see individual Plankton swimming around and the likelihood of individual plon appearing in that lens is determined by the density of the population of that type of Plankton at that point in the ocean so essentially what I'm doing is probab probabilistically generating these little critters under the lens depending on the overall population at that point in the ocean based on a simulation that models Plum uh and then um sorry come on next hi Jen um this one Amanda already showed so I don't really need to say too much about it but I think it's a a great example I personally do I know not everybody does of um portraying the uncertainty inherent in this case in polling data uh one of the nice points that I read that Gregor a mentioned is that it's not completely random it's bounded by the 25th and 75th percentile of the simulated outcomes so shows us uncertainty without being totally crazy and wild about it and if you want to know more about how you might use Randomness in your own work especially if you're interested in generative art or data art Anders Hoff has a series of writeups on uh incon convergent Donnet sorry I should have put that on here um that show you how you can sort of Shepherd and bound Randomness and sort of shape and into different forms so particle systems I think are something that we're all very familiar with here um so I'll run through it fairly quickly uh we can use particle systems to represent components of a system um representing the behavior of a system via the interaction between particles uh we can also show relationships between elements so for example a force directed Network graph in which the distance is inversely proportional to the strength of the relationship between the nodes and uh to show data elements so data elements don't have to be dots on a screen they can show complexity on their own they could be uh glyphs they could be Rose charts curves whatever something that has sort of inherent representation within it and then has sort of more of a a a bit of meta information about the entire system this is an early example of a particle system from 1986 by Craig Reynolds um called boids I think a lot of you are probably familiar with this one the simple rule set here is that each particle attempts to stay as close as well approximately the same distance from all of its adjacent particles so what ends up happening then is that this particle system naturally exhibits flocking Behavior just from that simple rule set and this is a more modern version of the same algorithm by Robert hajin flight 404 uh in which he has tens of thousands of particles doing the same thing and it looks very much like a starling murmuration um this piece by mimo Octan and quola is a data art piece that visualizes the movement of Olympic athletes from 2012 Olympics and what they've done is run uh video analysis on footage of the the Olympic athletes and then use that to determine sort of salient points of their bodies as they move through space and then apply that to particle systems um particle systems are triangles lines bars they use different forms as as the piece goes on and I would love to sit here and watch the whole thing but I can't um this piece by Zach Watson at staman Facebook flowers shows the spread of George S post on Facebook as it moves through the entire uh social network of Facebook well not the entire network um and each one of these pedals is an individual share so we see these bursts these tendrils that's sort of a viral moment where it gets shared sequentially through a lot of people um I know that we all know this piece by Fernando vas and Martin wattenberg in which they set up a flow field well I don't even need to show it because you all know it right uh they set up a flow field from Noah wind data and then introduce particles into it when I put together a talk full of streaming videos I thought there's no way that this is going to happen right sorry yeah hint. fmwi it would be great to not sell them short here though I'm going to move on to the next one my apology to Fernando and Mar Martin great so my workloads and theirs doesn't um so this is a piece that I did at the Exploratorium that was sort of inspired by their work hint. fmwi you should go check it out if you haven't seen it um instead of using individual dots for particles I'm using low Alpha white squares and instead of using only wind data it's also using a model from Noah that uh that forecasts liquid water content in the air so what we call in in the Bay Area fog um this is essentially a fog forecast uh that is then projected onto a topographical model at the Exploratorium and you can see the fog actually sort of follow the the patterns that we're that we're familiar with where it rolls over the mountains this is 280 running right here for anyone in the reservoir familiar with uh the Bay Area and it kind of goes between the the peaks of the mountains so this is where things get a little strange um recursion and repetition I think are are tough to uh find a use case for in um data visualization but I think that exists um trying to Rack my brain to figure out any context in which it would happen and the one that I found is is uh contexts or domains in which recursion and repetition are are really critical so machine learning genetics and evolution um are kind of the only places that I found recursion repetition to be useful for data visualization but if you have other ideas I'd love to see see that um this is one of my favorites from back in the hey days of Flash from yugon Nakamura is yo.com which is no longer online unfortunately um but this is a simple algorithm in which a clay ball splits apart into two two clay balls and then that splits apart into two clay balls and whenever there are four clay balls adjacent to one another they crumple back together into another clay ball and the space is bounded at a grid of 16 x 16 which means that when we fill up the screen we end up with this punchline we end up with this Singularity this this moment of recursion in which we start over completely this just goes on ADD INF an item um this is a piece by Alvin lucier from I can't remember the date 1970 maybe um which is an audio piece that might play there you go in which so that any semblance of my speech with perhaps the except ction of rhythm is destroyed what you will hear then are the natural resonant frequencies of the room so he actually speaks into this microphone and then records that recording back in the same sorry plays the recording back in the same room which is then recorded by the microphone and then plays that back into the microphone in the same room does that over and over again until you end up with the resonant frequencies of the room [Music] itself this is a fascinating piece conceptually and really difficult to sit and watch a performance of um so a more modern example of this is Patrick liel's version uh by doing the same thing with YouTube he records himself talking into his camera uploads it to YouTube it gets compressed he pulls it back down and then repeats the process over and over again and you end up with scary right that was about a thousand iterations right there um so this is an example from the illustration world uh John Franzen called each line one breath in which he takes a single breath and as he lets it out he draws a single line on the left edge of the paper he draws another line with the next breath as close as possible to the first line and he repeats that over and over and over again and what happens is that through this process of recursion or repetition recursion it's kind of Muddy um errors propagate through the entire work from left to right and you end up with this texture that feels very sort of like Fabric or cloth I was in a really boring meeting recently so I took the liberty of doing one myself it's pretty fun but I hope none of you are doing that right now um and then here's an example and data Biz try to bring it back this is a great piece by uh Tony Chu and and Stephanie y also known as r2d three um called visual introduction to machine learning um in which they demonstrate how a decision tree is calculated by finding all the split points through recursion uh and then they feed the data down through the tree so another technique is motion um motion is great for bestowing personality this is this is what animators do for a living right they take static models and they bring them to Life by adding motion uh they show they can repes motion can be used to represent relationships between things in your system they can be used to it can be used to illustrate change um it can be used to draw attention both to individual elements and depending on your context the piece itself that was important for me when I was working at the explor and there's just a lot of distractions just to create another distraction um this piece by Design IO it's Theo Watson and Emily goel there over in Cambridge uh is an early augmented reality experiment uh early 2009 um in which you hold up something in front of the camera maybe 2011 and uh the camera sees this this marker and sort of gives you this experience of moving up and down through space and uh elements in the screen move around and they and they sort of come to life through this motion that you can't see right now um and this this experience of being immersed in this world that has all of this character and this uh really sort of I don't know quality of wonder that comes from from the illustrations but especially from the motion of the illustrations and another work by uh with balloons in it with John Harris and sep convar as I want you to want me this is a piece that they created by mining uh data from or profiles from online dating sites in which they represent each person on the online dating site is an individual balloon and each balloon sort of has its own characteristics that they determine through natural natural language processing um and especially this movement called Matchmaker in which they algorithmically pair up two profiles I think uses motion to great effect for these balloons are just sort of intertwined and they're sort of expressing their love for one another this is a piece that I oh next this is a piece that I did with Zan Armstrong at staman um called the outlas of emotions in which we took Dr Paul data about the five Universal human emotions which you might be familiar with from the movie Inside Out by Pixar and we represent them the intensity of these emotions as arag graphs so the way that aaph kind of came to life is meant to indicate the anger the emotion of anger so it kind of flares up and this one is showing fear which kind of flicks out its Talons the next one is disgust which has this kind of heaving quality to it and sadness takes a deep breath and a sigh and then we have enjoyment which is a bullant and bouncy and happy so we tried to actually represent the emotion themselves using motion and of course the canonical example of how we can use motion to indicate a a state a change in state is Hans rosling's 200 countries 200 years 4 minutes the enormous disparities today we have seen 200 years of remarkable progress that huge historical gap between the west and the rest is now Clos in which we see the world's countries lifting themselves out of poverty increasing their life expans and their uh per capita net income um so color you obviously all know why color is important to data visualization so I'm not going to attempt to drill into that too deeply um but it's worth mentioning it's good for encoding for highlighting for legibility and then also of course for visual appeal so rather than talking about how we can use color I want to talk a little bit about how generative artists Source color um of course we we can use color Brewer but there are other techniques as well um this is a by that was not meant as stab color Brew is awesome you should all use it um Eric nosky was a flash artist back in the 2000s primarily um and what he would do is he would take photos and then uh as he he would use those sort of as a backdrop for his uh digital brush stroke so as the brush hits the screen it samples from an area around the brush from that Source photo and then his brush Strokes represent that they sort of repaint the the photo um Jared Tarbell did something similar with this box fitting image in which he used a box fitting algorithm to fill out space and then color it with a dominant color underneath each box and then Mario klingman also known as quasim Mondo took that one step further into the realm of data visualization which he did a pie packing uh visualization where he shows the distribution of the dominant colors underneath each circle um so those are about sort of sampling color from a point um this is an example from Vegas and wattenberg I get to show one of their pie that's good uh called flicker flow in which they take a bunch of flicker photos from Boston Commons and look and Sample them over the course of a year year so the year moves around in this uh Circle um and the thickness of the line in the Stream graph represents the the uh frequency of that color in the photos that they found from flicker so in the upper Le hand corner we see summertime down towards the bottom we see winter time I hear it snows a lot here um so this is a way to look at color sampled sort of across time instead of from a point in space and Brendan Daw did something similar with the cinema Redux this is Alfred Hitchcock's vertigo broken down into individ individual frames so every I believe it's one frame per second um reduced to an 8x6 pixel image and then a matrix made of that so what you had end up with is a reduction down to sort of the primary colors of each frame you can see the stripe about two-thirds of the way down where the protagonist goes through this total uh henic trip and goes crazy and if you've seen the movie then you know what I'm talking about and it's as colorful as it looks here and then uh Kevin Ferguson has a piece called Western Roundup in which he watched a bunch of westerns and averaged all of the frames together so this is sort of both across time and then across space where he's sourcing his colors um great so I showed you a bunch of pretty pictures and I know this is open viscom not IO so there's probably more Skeptics in the audience than I might find at IO um so I just want to say that while this is a celebration of complexity and Artful techniques I'm not advocating for this in all cases you need to use these techniques responsibly if you're aiming for clarity if you're aiming for the ability people to sort of understand the point immediately when they look at your visualizations then a lot of these techniques are probably not going to work for you so just be conscientious about how you apply your techniques depending on the context the audience and the data set um and this is a conversation that's been bouncing around the visualization Community lately is how we make these decisions depending on what role we have uh depending on what kind of products we're we're making um this is uh a diagram that Elijah Meeks recently put out which shows sort of on the product side the analyst data scientist Engineers over on the left side doing more precise numeric clean work and then folks over on the right hand side data journalist and especially uh consultants and artists doing more designed and engaging and complex work and what he's arguing for actually is not that this is the way it is and should be but that we can both sort of move towards the center and learn from each other um so and this is especially relevant to me personally because I just transitioned from ston design to Uber it's NCO team uh so I moved from the right side over to the left side uh but I want to land somewhere in the middle um and this conversation is something that we tried to capture in a medium publication visualizing the field and we would love to hear from all of you if you have thoughts about it then please uh tweet me and I'd love to tell you more the next few sides I have are another side project and I was I was happy to hear that there are other people here in this room that are considering this at well um lately I've been feeling like I'm not very well represented by my government um and I've been trying to think about what I can do about that and why that is and uh I think that this is a large part of it um this is about the third time maybe the fourth that Jerry mandering come up in this conf in this conference which is awesome um and this I think is a fundamental reason why I don't feel well represented our our system of representation is broken so uh some folks have come together to address the problem we have a D3 team uh there some people from staman current and former there people from cardo and from mapson and some other folks and uh we've been starting to do some experiments this is one by Mike merski where he used the efficiency Gap metric to generate a bunch of different wha IFS for Wisconsin so this is six different ways in which the district map could be drawn fairly according to the efficiency Gap metric for Wisconsin um and it all started when Waldo jqu from US Open data contacted staman to ask us to do a piece in which we bring the redistricting process out of the Smoky back rooms and out into the public it essentially uses GitHub as a backend for displaying redistricting proposals um allowing uh the display of of revisions to that and then for people to comment and uh potentially even to suggest their own Alternatives if they're so inclined so another call to action if you're interested in trying to fix Jerry mandering then tweet me and I'll get you on the slack and we can go from there thank you