Norman Bier - #OpenEd19 Keynote

I'm probably not going to do justice to his accomplishments but here we go Norman is a director of the open learning initiative and the executive director of the Simon initiative at Carnegie Mellon he has spent his career at the intersection of learning and technology working to expand access and improve the quality of education his experience in higher education spends the two-year and four-year sector of public and private higher education systems both domestic and international and also at commercial institutions but prior to joining all a lie he was a director and training of training and development at AI Carnegie Inc a CMU subsidiary that shattered to deliver software development education throughout the international partner institution he has taught Computer Sciences courses as an adjunct faculty at community colleges of Allegheny County and serve as a founding committee member of the Cooke Honors College at Indiana University of Pennsylvania he currently serves as a member of the board for the Shady Lane School and the next generation learning challenges funded Kaleidoscope open course initiative please help me welcome Norman via for our next keynote address that was fantastic so we have a lot of abuse for the suit from my friends in the audience some of you might have participated in a poll we're voting on what I was going to wear for open ed and if you're looking at the poll it was a tie between a Penguin's jersey and a political t-shirt I got a little feedback on Twitter and the Wardrobe poll was cancelled so I went with the suit right so first thank you all for being here thank you for the kind introduction before we get into the real talk I've got a couple of things I wanted to talk about it is an honor to be keynoting the open education conference it's a privilege to be standing here and have this time with you but I've been reflecting on just how remarkable it is for me personally to be standing on this stage to be talking to this audience I am very fortunate to lead the open learning initiative at CMU when I joined the open learning initiative about a decade ago the part of the initiative that I was most interested in was the learning piece at that point in time oli was building some of the most exciting and most robust courseware in the world and it was thrilling to be able to join an organization that was deeply focused on student learning on improving it and on really helping us understand how human beings learn the open part wasn't so interesting to me I tried to take a little bit of a look at what open education meant back in 2010 as best I could tell it seemed like there was a half of a crowd who was really anxious to bring down the publishers there was another half of the crowd that had an almost mystical belief in the power of an open license to to enact learning but I couldn't really see what this work had to do with the deep scientific work that I was interested in at oli open education was not for me it's kind of remarkable to be standing here on this stage but I was a part of the open community at least in theory and so I got dispatched with my colleague John rende early to open ed 2011 where we were going to give a talk on learning analytics like a lot of the country and that fall I was very concerned that was happening in the economy very concerned about what was going on with the Occupy movement was giving me a lot of hope and I walked into one of my first open ed keynotes having spent the morning reading the news and the news was that occupy Oakland was being cleared out people were being tear gassed protesters being shot I was a little worried about this and so I was a little bit concerned when for my first open education keynote I saw a guy jump out of a tent go through an entire occupy open edge stick move on to take a few shots at the earlier keynoter it was actually the only person I knew at this conference and who I knew to be a pretty nice guy and then move on to be you know relatively dismissive of learning analytics rights and thing that I had showing up to talk about I don't have a lot of memories from the rest of that open ed conference I will confess but I remember two common themes that were emerging from that keynote things that I've now been hearing about for the past 10 years that you know we're doing open education wrong and in this case I was doing open Efron's analytics were a problem but also what's been this ongoing tension between how we understand resources how we integrate those resources into our instructional practice what should we be doing with technology and how do we sustain all of this work I've been fighting with all this stuff now for a decade the other thing that I really remember from that conference was the Twitter back channel which was mean you know snarky and snide sometimes almost vicious and I have to tell you I'm not a nice person like if I say that the Twitter channel was mean if that's what got my attention something's going on there open education not for me so it's remarkable to be standing on this stage so I went home put my head down went to get back to the work of trying to improve student learning and at this point the thing that we were focused on was a project called the Community College open learning initiative where we're going to work through and build out for new gatekeeper courses we were collaborating with hundreds of faculty from community colleges around the country and a funny thing happened I have finding that as we were trying to develop these new courses put together this new course where anytime we found material that had a Creative Commons license on it was actually a lot easier to integrate into our course and the folks that were creating these materials were a lot more interested in talking with us and working with us on improving these things finding better ways to make use of them and it was interesting as well because I spent way too much time during this grant working with our general counsel's office trying to get approvals and right sub Awards and it turns out the Creative Commons license is almost magic once you get your GC to understand it it ends up removing friction it ends up greasing the wheels and so what I was finding in this work was that open in fact was a prime driver for that thing that I really cared about which was improving student learning so anybody ever had a bad breakup 2013 was a difficult year in Oh allies history in fact my support staff when the kid while admire anybody ever used to well I a couple of you if you've ever received a email back from the OL I helpdesk you've received it from Ken Wahl and Meyer who will refer to this period in no allies history only as that difficult summer so during that difficult summer we saw a really interesting breakout in the organization where some of my colleagues went out to try to take the methods that we were developing an oli and see if they could be scaled in a more thoughtful way via commercial platform our founding director my boss Kanda still headed off to the west coast to Stanford to build out a sister oli organization and there was a real question first on what I was going to do because I spent the summer like Hamlet hemming and hawing do I stay do I go and having decided to stay there was a real question on whether there was a continuing need for the open learning initiative whether there was support for it whether it was a thing that the community cared about and I'll say that these were particularly difficult days for me really a low point personally and professionally and it was in that period that members of the open education community really reached out and lifted me up told me that this work was important and it's been from that period a commitment of mine to see that this open peace can play a key role in how we advance and improve learning this community has become an exceptionally important thing to me this conference has become honestly my favorite conference of the year look forward to it each year and so it is remarkable to be standing on this stage with you keynoting the open education conference because it's become so important to me I am a sucker for public acknowledgments and I don't get the chance to make them that often and so when I talk about the people that played such a key role back in 2013 and for the past 10 years I was hoping that you could join me in thanking them it's a long list not all of them are still here in the open education community but for those of you that have played this key role in my work key role in my own life and this key role in open education back at CMU thank you yeah all right like a lot of you I've spent a lot of time over the past few weeks thinking about community I'm for Pittsburgh lifelong Pittsburgher we call ourselves users not for us the gentle y'all of the south or the use of the East Coast Pittsburgh is a city that is defined by its geography we're a city of three rivers and Hills and this means that we are city connected by bridges connected by tunnels Pittsburgh is a city of neighborhoods and I've been thinking about those neighborhoods a lot lately how these different cultures and different distinct areas rub up against one another how they're able to engage and maintain their own distinct identities while still coming together in some kind of cohesive whole the view that you see there is actually the view from my office the first nine years of my career at CMU I had a windowless office they just now gave me some space with some natural light and when I look out at that view I see some exciting things so that tower off in the left your left that's the University of Pittsburgh that is the Cathedral of Learning and I love that language I think its language we don't use very often today this notion that we're going to combine the secular and the sacred in that way the center you st. Paul's Cathedral Central Catholic High School see a practice football field where I sometimes lean out to give the Viking some advice as there training which they don't appreciate way in the back you see the road of Shalom Synagogue one of the largest synagogues in Pittsburgh and on the right you see WQED Studios which is the home of Mister Rogers neighborhood loved that view and I love again this notion that on the one side we have a Cathedral of Learning on the other side we have some of the most interesting children's educational programming being promoted secularly by a Methodist minister churches and synagogues in the middle I like what we're doing with this combination of the sacred and the secular in Pittsburgh you're gonna hear me talk a little bit more about Pittsburgh as I move on second thing I've been obsessing over that's going to show up many of you have read walk away a couple of you I am jealous for anyone whose hand is not up right now and that you get the chance to explore this book on your own Cory Doctorow is a man that knows something about open who's something been communities knows something about the weir is that we interact and try to build things and this is a really interesting exciting novel and it really has felt applicable especially over the past few weeks as he explores these questions of what happens when we straddle this weird space between economies of abundance and economies of scarcity but he also explores in a deep way kusas coordination problem this question of how can we come together as a larger group to do things that are beyond the grasp of any one individual how do we engage in superhuman work within walk away we also see an ongoing theme in which dr. al is misquoting a Scottish writer who in turn was misquoting a Canadian poet over and over extolling his characters to work as if you live in the early days of a better nation I've been thinking about that as I thought about open ed and ask how many of you are here for your first open education conference or have been in the space less than a year in fact come on stand up stand up seriously Bob there are 450 of you here for a point of reference back in 2011 there were only three hundred people attending the open ed conference the ability that you have as a group to impact and change and guide the future of open education is tremendous you have number US isn't that fantastic I hope that as you were standing you looked around a little and identified some other first-timers at this conference and I hope that you'll seek each other out in the break I hope that you'll share your stories of why you're here I hope that you'll talk a little bit about what you hope open it will be in 2030 and I hope that you'll take a little bit of time just a little to stay off of the Twitter back-channel if this is your first open ed you don't have to fight and begin thinking seriously about where you want to take this community so that was the prologue now we're into the talk when the program committee asked me to speak it was with the request that I talked a little bit about some of the work that's happening at CMU around technology around data and the ways that we are able to use data and technology to improve learning to advance our understanding of how human beings learn I have a usual stick if you've ever heard me speak you've probably heard my talk about oli and I like to spend a lot of time in that talk diving in talking about the data talking about we do but I've had an increasing sense that the usual shtick was not getting it done in part because I've been looking at community kind having community conversations around questions of what role should this data play and I'm hearing an increasing distrust of it I'm hearing an increasing set of conversations arguing that we really need to get back to simply trusting our intuition and teaching and that should be good enough so I've been trying to think about how I can talk about data how I talk about technology with all of you and ended up in prepping for an earlier iteration for this talk talking with my friend Michael Feldstein Michael and Phil Hill gave a fabulous keynote in 2015 almost prescient my don't filled you in here let's hear it for Michael and Phil you guys should go back and rehear their a female alright so I'm talking Michael Feldstein and I'm saying you know Michael what am I gonna do and go in I want to talk about data Norman you can't go in there talk about learning analytics and learning factors analysis and linear regressions and p-values and blah blah blah need to go in and tell them a story okay Michael let's tell a story but this can't just be a story about me though I do love to talk about myself this is gonna be a story about Sophia it's also a story about Morgan and Morgan who love to sit beside each other in class and answer when the others name is Co the story about Robyn it's a story about Madeline it's a story about G one story about aya these are a few of the students that I've had the privilege to teach at Carnegie Mellon introduction to computer science but in talking about their story and in talking about the role that open played in their education and in turn the role that they're now going to be playing in the future of open I do need to talk about this guy look at that handsome fellow he has no idea what's coming for him so 1993 I am headed off to a public university I'm a first-generation college student Pell Grant recipients so I'm part of that line down there on the bottom but otherwise pretty unremarkable kid right I am headed off to a decent public university called IUP as a lower class white American male with all of the privilege that entails and you know some of the lack of social awareness that you'd expect for that eighteen-year-old dual major English literature and then because I wanted to do something practical philosophy where are my English degrees in the crowd where are my philosophers I think the English Lit guys have it reasonably confident in my leftist politics as you can tell by the Lorax t-shirt right and so I was fairly confident when I walked into my first English Lit class taught by a man named Ken Wilson and Ken two or three days into class I don't honestly remember what we were reading I don't remember the larger discussion I remember Ken Wilson asking how many of you in class would call yourselves feminists hands went up mine didn't okay curiosity how many of you would call yourselves tenderness okay Ken's next question how many of you believe that women deserve equal rights how many of you believe that we should be getting the same pay for the same job hands up so my hand went up the second time and Ken just drilled me why was I possibly raising my hand the second time and not the first time he planed me in front of the class I remember it being humiliating little bit embarrassing but I also remembered sending me home and letting me think that you know how we identify ourselves the causes that we choose matter I have never been ashamed to call myself a feminist since and with that perspective I headed off to Carnegie Mellon and went to grad school and then was very fortunate to be working with an organization called AI Carnegie as you heard earlier I Carnegie was a spin-out from CMU focused on delivering CMU designed software development education to a large crowd we ended up working with a lot of schools internationally their faculty work to use our materials almost a prototypical loi without the open part and I was fortunate in that work to be working for a man named Alan Fisher Alan had been an associate dean back at Carnegie Mellon and in his work in the mid 80s working with Jane Margolis sort of looked around and said you know there's something strange in the computer science department almost 90% of our undergraduate CS majors are men and the 10% of women that we have coming into the class aren't lasting we're losing them each year why is that Alan and Jane spent here digging into this and what found was that there's an exceptionally large and complicated set of reasons that we're not attracting enough women into computing and that when they're there we're not keeping them some of this is curricular some of these materials that we use some of these are preparation and a lot of it is cultural I am proud for a lot of reasons to work at CMU but one of the reasons that I'm most proud of it one of the things that I love to point out is that after discovering this and after trying out some different things in experimenting over a twenty year period and Alan's work our computer science department was able to move from 7% women to parity we have 50/50 class of women and men in computer science I can't take credit for that so Alan leave CMU to start this organization called AI Carnegie and he's taking me and Jane are working on this book unlocking Clubhouse we're gonna try to apply some of those same methods to the courseware that I Carnegie was building and so when we jumped in and reviewed our third course in the sequence and looked up learning activities turned out that we had a whole sequence of learning activities that was biased against women that are we were losing Leo women we're not doing as well and they were dropping out of our program and Alan looked at it and said well I see this obviously like the kind of examples I'm using this is obviously going to cause problems so we changed the activities sent them back out into the field check the data and fix the problem this was revelatory to me it was it was revelatory that we were able to engage in this kind of iterative improvement because it completely changed how I thought about course materials and instruction it was revelatory to be able to work with these folks and see them taking these things they cared so deeply about and putting them down on the ground level making them work was also revelatory because as a guy with two humanities degrees I'd never seen statistics used in such an interesting and useful way really exciting so fast forward about 10 years and the open learning initiative received some funding to build out principles of computing fifteen one ten is a course at Carnegie Mellon it's our intro course my dirty secret is that I love to teach intro and so we began working to build out this introduction to computing course of course you can take now inside of oli and it was exciting we were trying to put into this course all of the best things that we knew about learning we were able to bring together a diverse team of faculty we were able to bring in instructional designers and learning engineers we were able to bring in learning scientists even brought in an anthropologist to study the process designers software developers even involving a few PhD students were able to engage in some research when we work to build this course we really worked hard I was very fortunate then to be asked to teach the course this was interesting because it was my first time teaching with the oli software let me tell you it's always scary to eat your own dog food learned a lot from that experience but it was also a really exciting time for me to go back into the class and talk a little bit about these questions of culture these questions of who we invite into computing and so this is a thing that I really spend a lot of time talking to my students about you walk out of beers class you're gonna learn how to program but you're also gonna know who Grace Hopper was it was great supper just found the bug right everyone's hello grace hoppers found the computer bug everyone knows that maybe if you do a quick Google search find the glam shot Grace Hopper my students don't get to know Grace Hopper they get to know this Grace Hopper they get to know Rear Admiral Grace Hopper twice recalled to active service promoted from the rank of Commodore amazing grace hopper for whom one of the only u.s. warships named for a woman exists first compiler race Hopper was a giant she didn't find a computer bug and my students learn about the multitude of women that have been contributing to computing some of the contributions that were made in some of our world's darkest hours the contributions have been made in some of our greatest technological triumphs and my students will learn about that weird drop that happened in the mid 80s the role that culture plays in excluding people from our work so my students will know that now you know it too so I don't tell you all of this to virtue signal not bragging although I'm proud of this work I'm telling you this as my bona fides this is work that I care deeply about this is work that I'm very enthusiastic about and so when we started to dig into the data when we started looking at the tale of the tape though my students were having parody on grades I was retaining them as well as we would like and when I'm digging into student feedback I'm not seeing the real difference isn't the kind of feedback that I'm getting but when we look at the activities it turns out that some of those learning activities the women in my class still were not performing in there as well as the men were so in some way I was failing these students despite the fact that we put in all this work despite the fact that my intuition had given us the best set of courses that I thought we could have this could be devastating right I mean in some sense it might be thought of to be embarrassing for me to come up here and admit to you this is failure but what's exciting about this what's most Imagi cool about it is that we don't have to stop at that what we're gonna do is dive back in and we're going to fix these activities we're going to cast a wider net to find better ways to get student voices into these activities and we're going to send them out in the field and we're going to test them and we're going to see how they work again and that's incredibly exciting to me but what's more exciting is that the act of doing this research working with one of our open ed fellow Steven Moore Steven are you out here hiding in the back so together Steve and I are us to take this analysis and use it for this individual course but to build it out into a larger set of analytic tools so that we could take a look at all kinds of courseware and really try to key in on areas where we're showing bias against and for different kinds of demographics and to me that's an incredibly exciting and magical part of this work because the problem for most of us of under-representation in computer science that's a huge problem it's a huge challenge not necessarily something I can do a about as an individual but this learning activity that I've written has some problems for women I can solve that and I think that if we're able to show this kind of information to a broader population everyone else will recognize their own ability to solve that - why am i able to do this well I'm able to do this because oli is obsessive in its collection of data forced and foremost and because I have data that crosses many sections and many kinds of institutions we're able to say useful and interesting things so one way to say this why I'm able to do this is because I've got the numbers but a different version of this goes back to allies origin story and oli has one of my favourite origin stories and one of the best so some of you may remember back in 2001 heady days of the.com boom lots and lots of excitement over the world wide web in ways that it was going to change the world and has changed the world ways that aren't all good - program officers Kathy casually Mike Smith made a major bet in finding ways that this technology could be used to expand access MIT is open courseware project was intended to take any kinds of course materials that faculty would share lecture videos quizzes exams notes put them up on the web and the amount of excitement that this generated at the time is hard to overstate Tom Friedman articles in the New York Times really big exciting things and this is interesting in part because this is before a period that we would clearly I be able to identify as open education existing right we have tremendous work happening with the open universities in Europe we have some conversations happening around open software but oh we are is a thing the idea didn't exist yet so you almost see it being invented so hot off of all this excitement Mike and Kathy head down to Pittsburg gather up some of our learning scientists and say hey do you see all that great stuff that MIT is doing yeah and we see what MIT is doing wouldn't you like to do the same thing no no we wouldn't not interesting to us and so they laughed Myka Cathy happened to be visiting us on September 11th and so by the time they got to the airport no planes flying anywhere I've heard since that there was some shared hurried discussion of trying to drive back to California Cathy vetoed and so they were stuck there to come back and talk to us and we had a fairly unique opportunity to spend four days with two program officers who asked okay what is interesting to you and the answer that they got was look we care about access we have an access agenda ourselves and we think that it's important but what we really care about is effectiveness what we'd really like to do is explore the ways that we can take advantage of these technological affordances to demonstrably enact learning and to find better ways to understand how human learning takes place okay that does sound interesting and so with that investment the open learning initiative is born and we're born with this mission to go out and design scientifically based online course we're taking the best of what we know from the learning sciences putting it into our course is but recognizing that there are big gaps that there are questions that we need to try to answer and so we spend a lot of time trying to answer those questions what's interesting is that in this moment you see a tension that has been sticking with us for almost two decades and it's that tension between access and effectiveness but in this moment you also see a different tension and that we start to belong together different communities different neighborhoods and we start to put them under a single banner so sometimes showing up with different kinds of agendas back to oli one of the things that's special about oli is that it presents an integrated view of learning so we've got integrated courseware and our design process is a thing that my friend Dale pike called learning design is hypothesis this notion that when we talk about a scientific approach to course design what we're saying is that deep down we believe we're making a hypothesis that this set of learning activities will produce a certain learning outcome will help students achieve a certain learning state when I phrase it this way kind of sounds like I'm experimenting on my students how many of you are uncomfortable with the idea of experimenting on your students yeah it sounds a little spooky doesn't it except for the fact that every time you walk into a classroom you're experimenting on your students you're just not being very explicit in your hypotheses friends you're not always collecting the data that you need to but we're all experimenting on our students this approach ends up being deeply rooted in a larger history of learning science and cognitive psychology at CMU and so when we talk about how oli works part of what we're working to do is try to understand what's happening as these knowledge states change and to do that we need to acknowledge that we can't see learning take place it's happening inside of our brains and so we're stuck building models of what we think is happening and if we're going to do something useful with those models we need to get them out of those students heads we need to be able to use these models to connect what's happening in learning science with the kinds of new instructional practices that we're trying to design so this means there we go this means leveraging that science and the design of these experiences it means using these models as we design new kinds of innovations and it means instrumenting these experiences now often you hear me talking about instrumentation and we assume that this must mean a technology it must mean courseware but we can definitely instrument our face-to-face experiences as well and what you have in this instance which is what half this diagram is an exciting feedback loop we're able to take these data as they come in and use them to continuously improve these practices to continuously refine our understanding of how these kinds of resources and innovations work but what's also exciting is that as we push this back into the learning Sciences my colleagues are able to take this in advance our understanding of human learning there's an incredible virtuous cycle that's achievable here at CMU we talk about this as a learning engineering approach which is a phrase that's going to get me some snarky remarks on Twitter I can live with that and we talk about it as engineering in part because we're an engineering school but also in part because the work of an engineer in many ways is to build more robust systems that are failure tolerant I must acknowledge that failures can take place and the kind of systems that we want to build should be able to try to work around those so when we talk about oli we're talking about a system of learning activities that's capturing different kinds of learner data using these interactions to give feedback to students to give really targeted hints but we're also able to use these to give new kinds of feedback to our educators things that they can use to change their classroom instruction we use these data to iteratively improve our courses and so this gives us the ability to look at different kinds of tools that we can put out into practice this is an example one of them the learning dashboard that sits inside of oli what you're looking at is a set of estimates on how well students are achieving specific learning outcomes sitting underneath all of this we have a complicated learning model but from an instructor's point of view very practical I can jump in to see where my students doing well in the green where are the areas that they're struggling and from there I can dig in to try to understand what are the skills that are giving them trouble what are the questions they're getting wrong what kinds of misconceptions are they exhibiting and then I can walk into class and change the kinds of instructional activities that I engage in now I just said that as though that's easy I'm gonna waltz into class with some fresh instructional activities right the reality is that that's the hard part that we need to be studying more and better understanding how we can integrate these tools into our instructional practice what does that mean and we're not there yet I am fortunate that oli is original framing was as a research initiative because in the end it can be really comfortable telling you there's a lot of stuff we don't know we're still working on it we have other tools that were able to use we're able to dive in to understand just how well situated our individual courses - giving us the kind of information that we need as that information comes back we're able to start to understand the underlying learning models and make some improvements this is a tool that's part of the CMU pool kit it's called the learning curve analysis any of you seen learning curves before a couple of you all right so quickly the idea behind a learning that when you look at an aggregation of student attempts to solve a specific kind of problem something that's really focused on an individual knowledge component which you would expect is that the first time they try these problems students should need a lot of help maybe they're asking for help maybe they're getting the questions wrong but with each additional opportunity to solve these problems you should see the amount of help that they need to go down so it's some very basic level what we're saying is that if you've got a sequence of problems and you can see that over time students are able to solve these you've probably got a pretty good learning model and in fact the one that you see up on the screen is a perfect learning curve it's a thing of beauty I'd like to tell you that every oli course generates these beautiful learning curves straight off the bat but the reality is that we often see learning curves that look like this first one means that we're sort of wasting students time they already know these materials the next two are interesting because they suggest that what we thought was a distinct skill probably has some extra stuff mixed in and we'd probably want to tease that out and see how we're supporting our learners the last one I actually still haven't figured out yet and been looking at it for years these are some of the ways that we're able to dive in and understand what's happening with learning and how can we improve our courseware but that ability to make these changes and to improve it ends up becoming so much simpler in an open space and so this is another piece of work that we're engaging in how do we actually start to build out better analytic tools that can be used by a broader audience how do we take these data and make them actionable for a larger population of faculty and instructional designers you'll know or hopefully you'll note that when I just talked about analytic systems I'm explicitly talking about human and the loop systems I'm talking about descriptive systems in which we are trying to support educators are trying to support students I'm not talking about more predictive systems these statistics are descriptive in nature and I think that there's often a concern when we begin talking about this work that there's a danger of it becoming dehumanizing that we're really trying to take faculty out of this loop that we're trying to and robot tutors to our to our students and I don't believe that that's the kind of work I don't believe that these are the kinds of systems that we should be pursuing they're not the kind of one we're just not in a space where these kinds of systems can be reliable but two they sort of miss the larger and important social aspects learning these pieces that we know about human connection and so finding ways to build these types of analytic systems that not just maintain that human in the loop but really are able to enhance the omein Enterprise of Education is important to us to achieve that it's been require a much larger and more diverse audience contributing to this work though and so when we think about what the future is of learning materials I think that part of that future must include these types of courseware systems that are able to give direct feedback to students they're able to provide data back for iterative improvement they were able to drive our larger understanding of how human beings learn and I often hear an awful lot of hesitancy about these systems right some of this goes back a very long time we and education are naturally conservative the new technologies have been scaring us for a while from the Phaedrus no one catches their socrates is promoting this newfangled writing thing which is going to wreck the ways students learn so changes to our instructional practice scare us but we also are justifiably concerned about new technologies because we've had decades of venture capitalists and edge upon ORS showing up on our doorstep and insisting that they're going to disrupt learning they're going to give us magic robot tutors in the sky right and so it's again it's understandable that they were hesitant about these kinds of systems but I would argue that we ignore these types of systems at our peril because we have pages and pages of evidence that these kinds of systems can improve student learning can deepen it can make it faster can make it more robust and if we ignore these types of systems we will end up seeding the design and development of these experiences two folks outside of the Academy either this means that we will be condemning our students to resources that aren't quite as good or it means that we're going to be forcing them to pay for things that are no longer open so the systems are going to be out there but I think that there's an additional complication in this and that when we talk about designing these systems we are explicitly talking about designing instruction and I would art that the work of designing learning experiences the work of designing instruction is explicitly the work of not-for-profit higher education this is a core part of who we are it's a core part of a work and I think that we ignore it we outsource it you know leave our hands free at our peril it's dangerous and so I think that we really do need to take a better effort to claim this space to engage more deeply with these types of systems but this means claiming them as open systems making sure that we can bring together open content open algorithms if we do that though we end up being able to ask some really interesting questions and solve some really interesting problems because it turns out that building these kinds of systems or to be more specific improving student learning is one of those superhuman tasks it ends up being beyond the work of any single individual how many of you design learning experiences good you find it challenging work you should write and one of the reasons is that with every instructional decision that you make you have an awful lot of decisions that you need to make how do we get started with any individual intervention do we start with the basics should we start with a more challenging understanding what's best I don't actually know the answer is it depends and depending on how you answer you then need answer some more questions focused practice distributed maybe something in the middle and we end up being able to work our way down a decision tree that becomes more and more complicated and this only covers a few of these branches so with individual instructional decision you have a tremendous amount of decisions to make how can we possibly know what is best any guesses on how many decisions you have how many options are in this space it has hurt a lot a lot fair so some of my colleagues back at CMU have actually calculated this fantastic paper in science instructional complexity and the science to constrain it over 200 trillion options how can we possibly make progress as individuals against this kind of complexity the answer is we can't engaging with this kind of complexity more deeply understanding how learning works requires a lot of things it's going to require us to make small thoughtful changes to course materials it's going to require us to share the results of using these materials out in the world I would argue that what it really requires is openness it requires open materials it requires open practices it requires the kinds of transparency that we expect of one another and that we've seen generally from the community I think that making progress in the learning sciences and improving learning ends up by definition being an open challenge and I think that this is another piece that we need to think about what a larger and more coordinated system can look like but when we think about those systems at scale we end up really doing superhuman work we end up really being able to better understand instructional context and this to me is incredibly exciting I love the idea that exploring learning even for a single human being is something that more than a single human being is needed for that this requires superhuman effort this is exciting work getting there then solving this coordination puzzle requires some humans but we can probably be helped by software it requires us to have some consensus on how we want to proceed on what things are important it's going to require us to share it's going to require collaboration and we do need to acknowledge that if we're taking advantage of these kinds of software systems the folks that are concerned have reason to be we know that there are dangers we see over and over in the news that as we start to implement algorithms and analytics biases always seem to be creeping in whether it's racial bias in health care whether it's on go against socioeconomic status and education we see this stuff creeping into our devices I don't know how many of you saw the story recently about the soap dispenser that only worked for white people and my friend Duda talks a lot how many of you know you - she's not here which is sad I was hoping she'd be here so this is actually her story she talks a lot about friend who uses a wheelchair and instead of using the wheelchair to head forward instead uses the wheelchair to go backwards a person is able to get a little more speed that way we're doing a lot of testing in Toronto of autonomous vehicles and the algorithms that can drive autonomous vehicles what do you think the autonomous car sees when they see a wheelchair which direction is that car expected to travel forward right and so in a lot of tests her friend is getting hit by the autonomous vehicles biases are going to creep into our software biases are going to creep in to our tools this isn't intentional nobody's trying to build a racist soap dispenser I have a lot of friends working on autonomous vehicles none of them set out to build robot murder taxis right but we need to acknowledge that as human beings these biases do creep in I mentioned earlier that part of my training as a software engineer was to recognize that some very basic level human beings make mistakes to argue that it's the nature of humanity and so we need to build systems that are able to account for those kinds of mistakes one way to do that is to refuse to use black box systems right we cannot use in our educational practice algorithms and approaches that we don't understand in part we shouldn't do that because we really can't trust them there's a parlor game at CMU among PhD students of trying to figure out if they can reverse-engineer closed algorithms for learning most of the time it's not very hard and there's a problem with these types of closed systems because that's not science I think that as an open community if we're going to take on the work of using and building these algorithms we have a real advantage that our culture and practice of transparency can open up these algorithms in a way that lets us acknowledge that they're not going to be perfect but also lets us borrow from the old Linux joke that with enough eyeballs all bugs are shallow I would argue that with enough eyeballs all of our biases are going to be shallow that what we need to do is attract a larger and more diverse community who are able to interrogate these algorithms and understand what's happening we can't afford to reject these systems we need to embrace them but and we need to embrace them on our terms and we need to recognize that our terms can include things like thoughtful and ethical data use I'm not going to spend too much time on this because I'm already running long but for those of you that are interested in this question of how can we collect data and use it in researcher to drive these algorithms in ways that are ethical there been a lot of folks spending a lot of time on this and I'd recommend you check out this work that happened at Asilomar pretty exciting and which is work that's still in progress if you've ever heard a talk from oli you've seen this quote herb Simon was a Nobel laureate at CMU who left the work that he was doing in economics to eventually end up really deeply focused on learning and education his sense was that if we're gonna fund 'mentally improve education we need to begin treating at his research it needs to be there and this research needs to be work that all of us undertake right with the same seriousness and the same respect that we undertake research in our core domains and he believed that this was work that had to happen in a larger community and so in some ways I hope that you'll hear this talk and you know I'll think about it as an invitation to join that larger community because what we need to make this work are larger shared ethical and open systems we need to build infrastructure that's going to support these kinds of efforts we need to be able to find ways to safely but also ethically share data we do know how to share content getting good at it but we need to be a little more thoughtful and a little more trusting and sharing the results of the use of that kind and on the whole we need to take on the work of open science so what are we doing at CMU to try to support this vision I'm really proud that earlier this summer we announced the release of the open Simon tool hit Carnegie Mellon has taken some of its best tools for learning science for instructional design for delivery and we've open-sourced them it was a lot of work to bring them together but we've put them out there in the field we're hoping that others will jump in and take advantage of this work and we're hoping that in making this a larger more open ecosystem others will begin to plug in their own tools and approaches this lets us do some really cool things when this whole vision of an open system using open tools works I'm incredibly excited to be collaborating with colleagues at the University of New Hampshire who are building out a set of open modules intended to simultaneously teach students about cognition while also giving them the tools to learn better metacognition is really exciting I'm excited to be seeing work happening at CMU around how we can teach core competencies collaboration better writing skills that our communication skills better conflict management skills and how these kinds of experiences can change the larger learning experience because our early research suggests that students that are exposed to this kind of training in this kind of Education end up reporting a much more positive college experience which is pretty exciting I'm incredibly proud of the work that some of my colleagues are doing Amy ogen judith oda chino in understanding what we do when we take these technology enhanced learning tools and start inserting them into different educational contexts this is a screenshot from one of my favorite keynotes which we don't have the time to dive into but if you head to my Twitter stream a link to it is pinned to the top when you have 40 minutes this is another way for you to spend your time and I'm incredibly excited to be working with colleagues at Santa Ana College crystal Jenkins Chris - are you here let's hear for Kristal where we are fundamentally asking how do we more deeply involve students in this work how do I bring those student voices into understanding where these models are failing how do I encourage a larger learner population to help us address areas that data has identified as deficient in our course we're really exciting stuff and so what's needed in this space are more tools and infrastructure but what's also needed are different kinds of social norms different kinds of commitments commitments to one another commitments to the work it also requires an awful lot of intellectual honesty a little bit of humility and I think a willingness to let our minds be convinced and changed by evidence and I think that that work together can help us go forth and live as though we're in the early days of a better nation so I hesitated to include this last part of my talk it was something that came to me months ago when I saw the date of this talk the past few weeks seemed to make it a little more wrought risky so I tend to brood on things as wandering the house and brooding I'm brooding and brooding and my 13 year old signs what is your problem dad what is wrong with you well I'm worried about this thing that seems really risky okay boomer then he came back he said why are you worried about something that's risky dad be brave and he walked up the stairs and a 13-year old I'm just gonna be brave started stuck doing some things so I want to talk about something that happened in Pittsburgh a year ago I mentioned to you that Pittsburgh is a city of neighborhoods the Squirrel Nut Oil neighborhood is a neighborhood that's at the center of our city geographically but also socially and spiritually Squirrel Hill has one of the largest Orthodox Jewish populations in the US and Squirrel Hill is a central spot inside of the Pittsburgh ecosystem I told you earlier that we call each other neighbors but Squirrel Hill is literally the inspiration for Mister Rogers neighborhood Squirrel Hill is an incredibly important place and a year ago within this lovely neighborhood we had one of the worst acts of anti-semitic violence to hit the United States some of you probably learned about this from the news I learned about it about an hour before it hit the news when my then twelve-year-old walked downstairs I said dad I just got a text from David David is the drummer in my son's band David says he thinks our concerts canceled today because there's SWAT guys outside telling them they have to stay inside the house and so we began getting alerts from Carnegie Mellon CMU is only a mile from the Tree of Life synagogue began making phone calls and I hadn't realized until that moment how many of my staff and colleagues live within five blocks of the Tree of Life I associate this with the open education space in part because we heard about the shooting went and tended a visual and the next day I jumped on a plane and spend some time with my colleagues at littleman and I ended up meeting on that and wait to call in a staff meeting over zoom which was deeply painful challenging but I also want to talk about this because as I told you I find it difficult to not acknowledge things publicly when I have the opportunity and roughly a year ago eleven of my neighbors were murdered so this is see so and David this is rose this is Richard this is Melvin this is Joyce who was a tremendous learning researcher this is Jerry this is Irving this is Daniel Sylvan and Bernice these were my neighbors and when we talk about what's happened to those people we can talk about the weird American gun thing which is too big of a problem for me to tackle and when you talk about humanity's weird anti-semitic thing but that's also too big of a problem for me to tackle and so what I do want to note is the weird and strange role that electronic communications played in inspiring the shooter to increasingly up his own rage these people that he saw as apparently being too welcoming of immigrants folks that he believed were taking his jobs and when you go back and you look at this message chain what you see are a lot of small steps that eventually lead to tragedy see a lot of misinformation and it leaves me asking whether we as human beings were actually ever intended to engage electronically increasingly convinced that any kind of electronic communication if it's the only way that we're going to engage is going to end up becoming problematic I think that we're weird primates that were really intended to see and touch and smell one another how many of you are familiar with the digital polarization project alright well now you've all heard of it I think that this is some of the most important work that is happening in the open education space I hope that you go out and take a look at it see how you can use some of these open materials to help your students better understand how they can find truth how they can engage with this crazy stuff that happens on the internet and how they can contribute back into this space how they can contribute their own truths as well and I think that as a community we would do well to maybe take this course ourselves to think a little bit about how we are engaging or what kind of information we are spreading to one another through the reason onic media last quick bit and with one more Tory doctor I'll quote one that I love because when we talk about the systems that we're displacing and have no doubt that an open education we are trying to display systems we can often lose sight of the larger picture that as we're getting caught up in our own fights about hierarchy our own fights about who's in charge we lose the fact that we are actually displacing much larger and angrier hierarchies and communications and so the past few weeks I think have been difficult and challenging and in some ways these are the stakes for the kinds of work that we're doing doesn't mean however that we need to accept this we're able to do a little better with these feuds in these blood sports I am deeply thankful that the program committee asked me to come and and and speak today it's really been a privilege and an honor but I want to really acknowledge the hard work that that program committee has done I was speaking last week with a member of the programming committee from last year who told me that it was one of the most thankless jobs that ever done that they basically spent time receiving criticism never any praise and that was last year's committee any of you that have been engaged for the past few weeks know that these folks have been in for an awful lot of criticism and yet what they have done is put together an incredible and cohesive program for all of you it's work that I am excited to be a part of they've reviewed over 400 presentations and so I hope that you can join me in thanking the program committee for this tremendous amount of work thank you for a Thank You Matthew Amy Tania Thank You Regina thank you John thank you Christina Thank You Kelsey thanks Brandon so have to harass Brandon thank you Lisa Thank You Mia and thank you David and with that go forth and see the great work that that program committee has done go forth and see the great work that all of you are doing don't do some good stuff together thank you [Applause]