[upbeat music]
VALERIE TAYLOR: So welcome to our panel on Leveling the Playing Field with MOOCs, I’m happy to have our distinguished panelists here. So we have Ben Eater, Lead Exercise Developer with the Khan Academy, we just heard the presentation. We also have Maggie Johnson. Okay, let me just… Thank you. So we have Maggie Johnson, Director of Education and University Relations at Google. Maggie manages all technical training and content development and information management programs for Google engineers and operations staff, as well as Google’s K through 12 educational programs in STEM and computer science. She also manages the university relations for Google, building strategic research partnerships with faculty and labs globally. Maggie is also very passionate about rebuilding K through 12 computer science education. Prior to Google, Maggie was a faculty member and Director of Undergraduate Studies in the Department of Computer Science at Stanford University. With respect to educational frameworks, Maggie brings the perspective of the Google Course Builder. There are five key features with the Google Course Builder, open source, in The Cloud, free, easily create online courses with great flexibility for activities, multiple lessons and video, and also great support for multiple assessment types. This is a list of some of the courses offered with the Google Course Builder, with an example from the UniMOOC on entrepreneurship that was offered at the University of Alicante. Our next panelist is Pooja Nath Sankar, who is the CEO and founder of Piazza. Before founding Piazza, Pooja was a developer at Kosmix, and at Oracle, and worked at Facebook on the newsfeed team. She received her undergraduate degree in computer science from IIT Kanpur in India, a master’s in Computer Science from the University of Maryland, College Park, and an MBA at Stanford Graduate School of Business. Pooja developed Piazza to be a remedy for students who are not given the intellectual space, freedom, or support to fulfill their educational potential and desire for learning. With respect to educational frameworks, Pooja will bring the perspective of Piazza, a framework to encourage student engagement in course discussions and questions. WitsOn is an example of the use of Piazza for online sharing among women in technology. The sharing was among undergraduate students pursuing STEM with women mentors from industry and academia for six weeks in October of last year. There were 632 mentors and over 15,000 students involved for which 90% were women. David Evans is an Associate Professor of Computer Science at the University of Virginia. David’s research seeks to create systems that can be trusted even in the presence of malicious attackers and that empower individuals to control how their data is used. He won the outstanding faculty award in 2009 from the State Council of Higher Education in Virginia. While on leave from the University of Virginia, Dave was Udacity’s Vice President of Education, and helped to develop a dozen computer science MOOCs. Dave has taught two large open courses for Udacity, including an introduction computer science course that has enrolled over 300,000 since the first offering in February 2012. And here we have some statistics about the cs101, thanks Dave. We can see that approximately 10% of those who sign up for the computer science course completed the course. The completion represents 38% of those who attempted at least the first homework problem. The demographics also indicate significant international participation, and 12.5% women. What we’re going to do for this session is that we’re going to have it be interactive, because we enjoy hearing from you, and as we promised from the last session, you’ll have an opportunity to ask questions of Ben Eater, as well as our other panelists, Maggie Johnson, Dave Evans, and also Pooja as well. So I’ll start off with a question, and then I’ll open the floor for a from you, and then we’ll go back and forth, keeping track of time. So now, starting with the statistics, with respect to diversity, which is our focus, and looking at MOOCs, and also other similar type of environments, as we focus on women and people with disabilities, as well as traditionally under-represented groups, being African-Americans, Hispanics, and Native Americans, then what we want to start off talking about is the demographic data. And we recognize that this data is important but we wanna ask the question about the challenges in obtaining this data, and also the opportunities for addressing these challenges. So with that, I’ll open up the questions and so maybe Dave, if you’ll like to start.
DAVID EVANS: Okay, so the data that’s here is from a small survey of students who finished the first run of the class, and in general, there are lots of questions about what data you should collect and one of the appealing properties of MOOCs is students can participate in them without sharing their identity if they don’t want to. So students can participate anonymously, and when there’s discussions in the forum, people don’t know where they’re from, who they are, male, female, all these things, unless someone decides to share that or picks a username that makes that clear, all those things are hidden. Now I think in terms of understanding more about what works, MOOCs are a great environment for doing experiments and there are students who are willing to volunteer demographic data, and they’re probably great wealth of data and experiments that could be done to try to learn about how different types of students are succeeding in MOOCs or not succeeding in MOOCs, you can do experiments to the level of individual lessons to find out whether this works or doesn’t work, and whether the type of student effects that. So I think there’s a big opportunity here, there has not been a lot of data collected in the demographic side yet, and some of that is questions about, you’ve gotta worry about different laws about how you collect demographic data, as well as whether that, you don’t want to require students to do anything, at least the philosophy at Udacity was really other than providing an email address, we don’t wanna force students to provide anything to start taking a class.
VALERIE TAYLOR: Okay, and maybe then we’ll go down to Pooja.
POOJA NATH SANKAR: Well I guess one of the biggest things to think about is the sentence or discussion that sparked me wanting to start Piazza was actually at Facebook, among women engineers who all stood up and said, when I went to MIT, or when I went to Stanford, or when I went to Harvard, I didn’t feel like I had a support group, and that was really shocking to me because I went to India where social discussions on boys and girls wasn’t very common. And to think that women who’d grown up in America, where they did start interacting with the opposite sex from high school, they also felt they didn’t have a support group. One thing we should just all be very cognizant about is maybe this problem could be exacerbated in a 100,000 student class or 10,000 student class where the women don’t speak up enough and they actually feel more in minority. And so thinking about what we could do I think is an area of interest for me, specifically.
BEN EATER: To your question on challenges in collecting demographic data, one of the things Khan Academy, our mission is a free, world-class education for anyone, anywhere and we wanna lower the bar to sign up or to engage with our content, and so all of our content is available without signing up on our site. And if you do sign up, we try to make that as frictionless as possible. We use third party authentication services from Google and Facebook, we don’t try to share a lot of data with them just because we wanna keep that as frictionless as possible. Relatively recently, we now will allow you to create an account with us directly, without using the third party authenticators. When we did that, we had to ask for birthdate ’cause we need to ensure that, for people that are under 13, we respect certain privacy things there for legal reasons, but we also decided to ask for gender, and it’s optional. And we actually until last week had never looked at that data. But I knew I was coming here and I figured I would be asked. [audience laughs] So I looked at that data last week, so you guys I guess had some impact. And it turns out that of the data we’ve collected, I’m not sure if there are biases, you guys would probably understand that better than me, but of the data we’ve collected, it looks like about 47% of our registered users are women. And I also know you guys are particularly interested in the computer science, and so I looked at our computer science platform, and I showed you the programs earlier, we’ve launched that less than a year ago, and there are over 600,000 programs have been created on the site. And it looks like about 40% of those programs were created by women. That’s basically data we’ve gotten, as far as demographics, our demographics skew fairly young, pretty much the mean is around, there’s a kind of nice normal distribution around 14 years old.
VALERIE TAYLOR: And Maggie?
MAGGIE JOHNSON: Sure, well with Google there’s really three sets of classes that we look at when we talk about Course Builder, So the first set are the what we call internal for external, and these are courses that we build internally to teach our users about products and services that Google offers. So we did power searching last summer, we’ve got a big MOOC that we’re gonna be doing on Maps and Google Earth, which just announced at Iowa last week, so those kinds of courses are ones that they actually represent Google product launches. So we’re very sensitive about the kind of data that we collect there, and very limited due to our privacy policy about the data that we can collect with any students taking those courses. So that’s one set. The second set are courses that are partners or people externally who are using Course Builder. The way it works is it all runs on App Engine, and if you’re a user, a Course Builder, then you actually own your own instance of App Engine and you run your course, you can brand it any way you like, and you own the relationship with your students and you own the data. So we know that many of our users, our external users on Course Builder have collected demographic data, but we can’t access that, either, because that’s their instance. So we can’t really get at it, either way, because of our privacy policy, or because of the way the terms of service for App Engine are set up. Now, one third area that we use Course Builder for is for internal engineering education. And we have about, I think we’re close to 20,000 engineers now, and a good percentage of them are women, and we use Course Builder internally to do engineering training. And in that sense, we’re able to do some of the experimentation to understand what are the best ways that we can support our women engineers in the training and the types of environments that are gonna be most conducive for them, and then we take that information and we pass it on to the Course Builder engineers so that could get built into the larger platform. So we use our own engineering community as kind of a lab for the things that we can build that will be supportive of, even if we don’t know who they are, we wanna try to make it supportive for any women that are taking the courses on Course Builder.
VALERIE TAYLOR: Okay, thank you. So now I do wanna open it up for a question from the audience, if we have one, otherwise… Oh, a pioneer. Thank you, yes.
AUDIENCE MEMBER: I’m confused about the economics with regard to these MOOCs. Some I know are coming from colleges and universities, and I don’t understand the way in which the economics of this works. Why is it to the benefit of a major college or university to put one of these online where, as far as I understand, the student is able to use it for free?
DAVID EVANS: So it’s very early in this. There are many different business models that are being explored. Part of where the opportunity is, is that the cost can be very low. At least the cost per student can be very low that you don’t necessarily need a big revenue stream to support it. Now, one of the opportunities is if you have more resources, you can do better courses and put more into it, and then do the kinds of things that Ben showed us, with those kinds of much more developed exercises and put more resources in the courses. The different models that are being explored, some are recruiting models where your students who are interested in getting a job can indicate that, and the MOOC provider might be getting a recruiting fee from companies that they connect students who do well in the courses to. There are also ongoing experience where students are paying tuition to get credit in the class. So I think this is, starting over the last few months that there are more courses that you can take for credit, and students who want to get credit will be paying for the privilege of having some additional part to the course, being able to take a proctored exam as part of that and get credit. So I think it’s not clear yet what the right business model is. I think there’s a huge opportunity in the sense that the cost can be very low, that you don’t need the bulk of students to pay anything, that you can have a free and open course that the vast majority of students taking it won’t have to pay anything for, and they’ll still get a lot of value and contribute to the community, if there’s a small fraction of students that are getting work value and paying something for that. But it’s very early and none of the providers as far as I know, are profitable yet. So the right business model, both for the universities and any companies involved in this, certainly remains to be seen.
MAGGIE JOHNSON: I think it’s really interesting, too, if you look at how this all started, that it was at Stanford, and they took three or four of their courses that they were basically flipping the classroom and using technology in order to provide a better experience in class for their students and because the technology was already a part of the in class and on campus experience, it was very easy to open it up to the world and just see what would happen. So they did that experiment and it just kind of took off and all kinds of things happened, and it happened in such a way that yes, some for-profit companies got involved, but as Dave said, they’re not quite sure what they’re gonna do yet, but it was something that kind of took off in a way that was surprising, so that’s part of it too.
DAVID EVANS: Add one more thing.
MAGGIE JOHNSON: Sure.
DAVID EVANS: We talk about MOOCs as this new thing that’s happened in the last year, but there’s been open education going back decades. One of the most successful projects of this was the MIT open OpenCourseWare, and MIT put courses online and they got tremendous value, in terms of applications from around the world of students who had already taken MIT courses and were excited to come to MIT and they increased their applicant pool and the quality of it, and viewed it as very valuable in terms of just spreading the message of the university. So I think there are ways universities can benefit from being associated with open courses, even if there isn’t a direct revenue stream coming from it.
BEN EATER: It is very early and hard to tell where this is going, and I don’t think any of us really knows where it’s going. One thing I do think is we are gonna see kind of down the road is that there’s gonna be this sort of decoupling of the education component and the credentialing. And you’re kind of seeing this with MOOCs where the MOOC is providing the course, and in some cases they are starting to provide a credential, and that perhaps is a monetization strategy for the for-profit folks, and even a sustainability strategy for nonprofits like Khan Academy, perhaps, is that this credentialing is something separate and I think as you see credentials start to emerge, like these micro credentials that say hey, you are a level three programmer, and you’re certified by this well-respected credential that is emerged from this process, whatever happens, then you can go anywhere and you can do that. You can study yourself, you can take a MOOC, you can go to a university, all of those things are gonna have different benefits, depending on how you do it. And then I think the MOOCs are going to, the for-profit MOOCs may actually be able to demonstrate value and say, hey we can prepare you for this credential that has a real meaning in the world, and they may actually start charging at that point, perhaps. It’s hard to say where this will all go.
MAGGIE JOHNSON: One last thing, Google is doing something that’s not academic. So all of our courses, the internal for external ones that we’re doing they’re short courses, it’s like teach somebody to be a better searcher, how to use Maps more effectively, and there’s obvious business value in teaching our users to use our products more effectively. So, there’s another thing too.
DAVID EVANS: I don’t think we’re worried about Google running out of money. [all laughing]
MAGGIE JOHNSON: Well, no, you know, but, it’s a business model, right.
VALERIE TAYLOR: And a follow on to that, so as we talk about the economic model, going back to the title, to with the leveling the playing field, as we talk about the economic model, if you can comment on your view in terms of, do you see that, especially being that it’s open, leveling the playing field in terms of more diversity, more inclusion, more women involved, more under-represented groups, also people with disabilities, if you could comment on that aspect as well. So, Ben?
BEN EATER: Yeah, I guess I’ll take it. So this is all sort of pure speculation and I’m not exactly an under-represented minority here, but I do feel like in some sense the story that I just told you all, in terms of how I got stuck at some point in my education, and I really don’t know where I got stuck, I don’t know why, I just know that there were these gaps that sort of formed in my knowledge that eventually led to me hitting a wall and led to me getting kind of frustrated, maybe I was sick one day, I don’t know what it was, but I can certainly imagine someone who perhaps has a teacher, is in an environment that is not empowering to them, they get kind of stepped on, they get left behind in some sense. And as I just talked about, once you get left behind, once those gaps start accumulating, it’s really hard to catch up. And so what I see in terms of these things that are online, these things that are self-paced and available when you are, can allow people who might otherwise get stuck or left behind to catch up or to go at their own pace and even race ahead.
VALERIE TAYLOR: And I guess, what about frameworks that also help with the engagement?
POOJA NATH SANKAR: I spoke to Professor Tucker Balch, who teaches a computational investing class at Georgia Tech, and he uses Piazza for the engagement component. So his class is hosted on Coursera, but all the discussion is done on Piazza, and just quoting him, his takeaway after teaching this class twice now is MOOCs biggest weakness is the lack of interaction between faculty and students, and students and students. Often, as I said, for women or minorities, the feeling of isolation can be exacerbated in a larger class where you think everyone else is excelling, but actually everyone is doing as poorly as you are. And for him, his take, again, because I really treat him a bit bigger expert than I am, ’cause I haven’t taught these classes, but he did say forums in general are an essential method for filling the gap and for him, he found Piazza’s an essential component for the online courses, reflected also in in feedback from students feeling connected to each other while they were doing his class. So I think if we think about promoting diversity in these classes, letting women be known that there are other women out there, and also the boys are struggling as well. [speaking off mic]
VALERIE TAYLOR: So now we can take a question from the audience, we’ll go back and forth.
AUDIENCE MEMBER: I have a question. So I’m curious about the recent announcement of Georgia Tech and its offering of, or it’s partnership with Udacity and the master’s program in computer science, and the potential, or the opportunity that that represents to not only, so it’s scaled workforce training, into scale, the technical workforce, but also the opportunities that that might represent for diversity and for inclusion and that, and I wonder if any of you have an opinion on that.
DAVID EVANS: You said it very well, better than I could say it. I think there is a real opportunity, and one of the things that I had hoped, in the early days it seemed like you didn’t need a degree. That was one of the opportunities of opening education was to say, you don’t need a university degree. There are other ways to demonstrate that you’re talented and capable. And it’s become more clear to me that yeah, there is still this currency in academia, the degrees really do matter. And so providing a low-cost degree with the open format will open up opportunities to students that couldn’t afford a high-cost degree where they had to be on campus for four years and pay the kinds of costs that it costs to do two or three years, to do a masters at an institution like Georgia Tech. There is also a plan to keep those courses completely open as well. So the students who are enrolled and paying in the degree will get some extra services, get some human-aided grading and assistance that you couldn’t afford to provide to every student, but most of the course content will be free and open for anyone.
VALERIAN TAYLOR: Did others, okay.
BEN EATER: I think this goes right to my point of the sort of decoupling of the education from the credentialing. You can go and you can take the course, you can get the value from it, but if you want that credential, then you can pay for it. And I think that makes perfect sense. I think it seems like a reasonable business model, and obviously as these things expand, we may even see that cost come down even more, and frankly I think that’s very exciting.
VALERIE TAYLOR: Okay. And I also wanna comment on, in terms of the question about Georgia Tech, that in the audience is Charles Isbell with Georgia Tech, who can probably answer very well in terms of, with Georgia Tech.
CHARLES ISBELL: So, two things. One, hi. [audience laughs] Two things, one, on the credentialing question, I think it’s actually important to understand that the way that we’re structuring it at Tech, is that only the people in the degree will actually be tested and assessed by Georgia Tech faculty. So even though the course itself will be open, just like very other MOOC is, or almost every other MOOC is, those people will have assignments, they will do many of the other things, but they will not take the final exams, and the midterms, and they will be graded. And so, I do think there’s a business model there where you can imagine sharing course content across multiple universities or multiple departments, where each one decides how to test them, and each one decides what an A is or what a B is. And I think that’s actually an important distinction worth making. On the question of diversity, our current belief, I mean, this is an empirical question, but our current belief is that it will increase in absolute numbers, all of the groups that we happen to care about, it is unclear whether it will increase in terms of percentages. Tucker Balch, in fact, in a internal argument among the faculty about exactly this issue for diversity, pointed out that although as a percentage, the number of women and African-Americans taking his course was smaller than in our typical classes, he had through his one single course, taught more women and more African-Americans than he had the entire 12 years he had been at Georgia Tech, all of his other courses combined. So I do think there’s a lot of potential and there’s a lot of possibility there, but it’s gonna require a lot of work by everyone involved to make certain that we bring everyone along.
VALERIE TAYLOR: Thank you.
MAGGIE JOHNSON: I love that point, because maybe it’s a glass half full kind of view, where there’s been a lot of writing about a lot of controversy about how many people drop out after they register, and there’s just this huge drop off of people. And when we look at our search courses, we have like over 40, 45,000 people that have actually finished it. And that is 45,000 more than we would ever reach any other way. So there’s something very positive about that, and I think it really is important that of that 45,000, it’s gonna be a larger set of women and under-represented minorities than you would have ever reached otherwise. So there’s something to that, even though the drop off rate, and there’s all these other kind of scary statistics, the numbers are just so big that it’s really interesting.
VALERIE TAYLOR: And that leads us to another question that I’ll ask. As we talk about the sheer numbers, and you can talk about the numbers versus the percentages, but if we look at in classroom versus the online and MOOCs, if you could talk about some of the differences and how maybe some of the differences may be more engaging or less engaging, with respect to diversity, so we can talk about numbers, the sheer numbers being much higher, you can talk about the percentages being lower, both of those, in terms of contrast. So if we could talk about those differences in terms of the two environments.
POOJA NATH SANKAR: Sure, I’d love to share. So I’ve had the fortune to sit in on many professors’ classes and see how they teach their class. As Eric Mazur of Harvard, for example, I’ve seen how he has small groups of people at tables, and they’re doing problems, and then they revisit problems after the answer comes out. Or Lorena Barba of Boston University does the flipped classroom. And it’s really fascinating to see how all these professors get very creative in the more traditional-style of classrooms, given the size, maybe it’s 50 people, 100 people, 20 people, they’re able to have very creative styles of teaching their particular subject that, in MOOCs, it’s just not possible if you’ve got a class of 10,000 online. But again, I think it’s important to Maggie’s point of like, what are you enabling? If it’s net positive, that’s a good thing. Of course it may not be able to match to what you can enable in a more traditional style classroom.
BEN EATER: You’re definitely enabling more people overall, which is great, but absolutely right. One of the problems I think, particularly with a lot of the MOOCs we’re seeing now, and I think Udacity’s probably done a better job then most of this, but they are very traditional in the sense that it’s a lecture. And even when you go into a physical classroom, thankfully we’re starting to see fewer and fewer lectures, as Pooja mentioned. And I think in this day and age, there’s absolutely no reason to collect everyone in a 300-person lecture hall and just lecture at them. It’s a very dehumanizing experience. [laughing] And it’s actually in some sense, more dehumanizing than sitting alone at a computer and actually having a conversation on Piazza, or on some of these online forums. ‘Cause there you’re actually interacting with people. In a lecture hall, you’re not interacting with anyone, you’re just probably falling asleep. I think there is some advantages to getting out of that lecture, and the MOOCs aren’t necessarily always doing that. What I think would be great to see is as these systems evolve, as there’s more technology that goes into them, these online courses can actually become, in a sense, more adaptive, and that’s one of the things we’re trying to do at Khan Academy is that when you get through part of this course, traditional class, and even a MOOC now, if you get partway through the course and you kind of hit a wall and you get stuck, the course moves on and there’s not. Maybe there’s some forms and some resources there, but if those aren’t doing it for you, you’re kind of, you have to move at that pace or you’re gonna get left behind. And I think one of the things technology can do is it can provide the ability to slow down and work at your own pace. And for folks that may get discouraged, if they get into something and then it a wall, having that ability to work at their own pace can be enabling to see that, oh, I actually can do this if I work at it.
DAVID EVANS: And one of the things that you get with scale, is also amazing contributions from students. In my small in person classes, students contribute in the class discussion, there’s certainly a lot of value that comes from students. Once you get to a big online class that’s open, there will be a handful of students that just make amazing contributions, that for any point that I make briefly in one video, they’ll write up a 10 page post in the forum explaining in lots of details and answering all the questions from hundreds of students in the class about it. And that’s something that you get with this open format, and another thing is, well, it’s there forever for every student. Now, the next time any other student comes to that video, they’ll see a link to here’s all the discussions about this, and you can read all the depth that goes far beyond what’s in the actual class, as well as help for, here 20 other students got stuck on this problem, and here’s the different ways they understood it afterwards. So there’s a lot of value that comes from the students in the class, and it builds over time. I think it leverages in two ways with scale, ’cause the scale only motivates students to contribute, ’cause they get the social validation of thousands of other students reading their contributions, rather than if it’s in a small in person class, a few students might notice.
POOJA NATH SANKAR: I’ll share an anecdote that actually builds exactly on this. So Dave Fetterman was an engineer at Facebook while I was a engineer there, and just a few months ago I got a Facebook message from him saying, “Pooja, Piazza’s awesome, congrats on building this.” So he was taking Tucker Balch’s class, and now imagine you’re doing a large, open online class, and you’ve got world-class engineers doing the class with you, and you can read their answers and contributions, and that in itself is so powerful.
DAVID EVANS: And in that sense, there’s far more diversity in open online classes then there is in any university class. ‘Cause there’s students taking my Intro CS class, and you heard from Michael earlier today, that have tens of years of experience as a top-level programmer helping answering questions from students just starting in their first CS class, and that doesn’t happen in university classes. And when there is that kind of gap in university classes, it’s often kind of intimidating, and overwhelming, it’s not helpful, but in the online classes, it often can be very helpful.
MAGGIE JOHNSON: I mean, it all speaks to the potential of this, because in its early stages, it’s tried to simulate a lot of lecture-based kind of things, a lot of traditional ways of teaching. But the adaptive learning possibility, the amount of data you get from even one MOOC is just enormous, and the kinds of things you can do with that data, the social element and having many many students, we’re just beginning to tap into the potential of what this could look like that could be completely different from the way teaching and learning is done now. And that’s the evolution that has to happen, and we’re not really quite there yet, but we have a traditional MOOC now, which everybody knows what it is, and we need to kind of break away from that and think of ways that we can start taking advantage of some of these capabilities.
VALERIE TAYLOR: So it’s very interesting too, as we talk about the difference between the in classroom and the MOOCs and the online that just the sheer number allows you, that you can have groups, so you can have that critical number. At one point with in classroom, you always said you need to have a critical number of women to feel comfortable in a classroom. And now, while the percentage may be low, the numbers are there such that you–
DAVID EVANS: There’s a Facebook group of women taking the class.
VALERIE TAYLOR: Right.
DAVID EVANS: Here’s the community of students in Cairo taking the class. That’s one of the things that you can get at these sub-communities that you can never get in a small class, ’cause there’s just not enough common people that have the same schedule and have the same interests to do that.
VALERIE TAYLOR: Right, so while you may have a percentage that’s 20%, that 20% may be two or three. Whereas in the MOOCs, you may have a percentage that’s one percent, but that may be hundreds, so it’s that sheer number. And I know there was a question.
AUDIENCE MEMBER: Yes, I was wondering to what extent your applications are accessible to people who are blind or deaf.
MAGGIE JOHNSON: Well, with Course Builder, we definitely have made it accessible, in terms of disabilities, blindness disabilities. I think that’s the main place that we focused right now. So Course Builder out of the box does provide those capabilities. One of the things that’s a challenge though is that because it is a platform that anyone can use, there’s a certain set of steps that one needs to take as a content developer to make sure that the content they create fits into certain templates that we provide, but we can’t enforce that because it’s an open source system, it’s a free system. So we have put all the kind of basic framework in place for that to happen, but we rely on the people, the content creators who are using Course Builder to finish it up, because they have to do the final set of things to make it work.
POOJA NATH SANKAR: We’ve actually been very fortunate, a few folks from UCSD San Diego, they offered to test and help ensure we were conformant. And so from the very early days, once we opened up Piazza beyond one or two schools, we were able to work with them and get them to test our site and make sure we had actually complied.
BEN EATER: At Khan Academy, I would say, is this probably an area where we could do a little bit more. We’ve got all of the videos subtitled and that sort of thing. But as far as the website, as far as I know, we haven’t done very extensive testing. We definitely try to make sure, that you don’t need color acuity, those sorts of things, but we haven’t done really extensive testing. That’s definitely an area, if there are folks here that could point us in the right direction, I’d be happy to talk with you.
DAVID EVANS: Similarly, when the first course were done with Udacity, we really didn’t understand this as well as we should have. We certainly thought about it, realized it was important. Doing captioning is not enough. You’ve got to understand more than that, and that’s something that has improved, but it is definitely something people are still working on.
MAGGIE JOHNSON: It’s a really significant engineering thing to take on.
DAVID EVANS: To make all the visual stuff, you can have a voiceover try to explain it, but that often is not enough if you really need to understand the spatial relationships in the video to be able answer a quiz.
VALERIE TAYLOR: But that’s a good area of opportunity with the online.
DAVID EVANS: And certainly in many ways, MOOCs are much more accessible to many types of students that couldn’t function well in a regular classroom. So I think there is a big opportunity there, but it has not been advanced to the point where it should be yet.
BEN EATER: And to that point, we’ve gotten tons of anecdotal evidence from kids with dyslexia or ADD, and all these things that say, Khan Academy is somehow getting through, and they’ve tried all sorts of other things, nothing else has worked. So we’re getting anecdotal evidence that that is working. So there are definitely segments where we’re, in some sense, better than what they had before. Definitely a long way to go.
VALERIE TAYLOR: Okay, and now there’s another question from you.
AUDIENCE MEMBER: I’m wondering, given the enormous numbers potentially that you are teaching to, to what extent are you able to monitor, engage in the conversations that occur in the different discussion forums to ensure that the content that’s being shared among students is still accurate, that there is fidelity to the content? That seems like it would be a huge job.
DAVID EVANS: That’s rarely a problem.
BEN EATER: Yeah.
DAVID EVANS: Technical correctness, if someone gives an incorrect answer, another student is gonna post a better one, and it’s gonna be very quickly resolved. I participate in the forums, I think sometimes I feel like I can add something, and it is good to have a sense that, and I think students appreciate the sense that there’s a real teacher there, occasionally, but it’s never because of there’s some technical point that I thought I needed to correct. ‘Cause those, pretty much other students are, there’s students in the class that know more than me about everything in the class, and then will be happy that they’re, correct other students or correct me if something’s wrong.
AUDIENCE MEMBER: Is it sort of expected as a part of the process, the instructor, his following along with all the discussions– [speaking off mic]
DAVID EVANS: I saw as my role at Udacity, I was involved in many MOOCs. Some instructors were very active and built a real great relationship with students. Others, the instructors were not as involved in the class. And I think it did have a big impact on the success of the class, whether it’s a causality or it’s more of a correlation. But I think students who felt more like putting effort into classes when they saw that the person teaching it was actually involved, and little things like having short recorded office hours, or being in the forums, or answering email, I think do go a long way to making it feel like it’s more of a personal experience than consuming content that someone created and isn’t involved in.
MAGGIE JOHNSON: We’ve actually measured this to some extent, in terms of having TA’s, which we’ll, for our courses on products and services we’ll always have TA’s involved in watching on the forums. And we’ve done some control groups where we’ve run courses multiple times such that we can determine that if we have that engagement, you will have a higher completion rate. So, it’s a positive thing to have some interaction with either the instructor or someone representing the instructor, it helps a lot.
POOJA NATH SANKAR: So one of the core design decisions in Piazza actually is the notion of a wiki answer. And so there’s a single wiki answer that all students keep editing and instructors could edit as well. And so the feedback I get from professors is students end up policing each other, and it’s like Wikipedia in that it keeps getting better and better, if there was any misinformation provided.
VALERIE TAYLOR: And related to that, can you comment on maintaining a respectful and engaging environment among the students such that students feel comfortable?
POOJA NATH SANKAR: It’s phenomenal, I peek in on some of the large open online classes on Piazza, phenomenal. The students will chip in saying, “Why are you giving professors a tough time? “They’re trying their best, they’re working hard. “I think we should be respectful here. “If you don’t wanna be here, “just log off, don’t participate.” And I found professors are known to stand up for themselves, and it’s just phenomenal getting that sense of, students watching out for the professors, that scale. It’s really heartwarming.
MAGGIE JOHNSON: We’ve had the same experience, it’s almost nonexistent. And if it is, you catch them and you get them out of there. But it’s very very small numbers.
DAVID EVANS: There’s very rarely, and it’s amazing how rare it is, but there are occasionally battles between students in the forums. And one thing about having your open class, is you do end up with 13 year old boys in the class, and they sometimes get into ego battles with each other. And that can get ugly, and we try not to moderate, or delete stuff, but there’ve been, in the entire history of Udacity, maybe three cases where someone had to be removed from the forums ’cause they were just antagonizing too many people and not contributing usefully. But it’s remarkably rare for how many 13 year old boys there are contributing.
BEN EATER: By and large, I would say most of the discussion, or all of the discussion really on Khan Academy is very productive. We do have the situation where our median age of our users is around 14 years old, so we get a fair amount of interesting posts. But basically the way we handle that is anyone can flag posts, and we have moderators that are volunteers that go through and clean those up very quickly. And if something gets a certain number of flags, it’s automatically removed. It is a job to police that and to remove some of the things that are just clearly off topic. But the discussion, the things that are posted that are actually discussion are actually pretty phenomenal.
VALERIE TAYLOR: Okay, I know there’s a question.
AUDIENCE MEMBER: Thanks. I agree, I think it’s really exciting that more and more people are choosing MOOCs as a way to supplement their education, when it comes to CS and tech. And my organization always encourages us to think big and that the future is always closer than we think it is. So keeping that in mind, how do we evaluate the folks who are graduating from these MOOCs now as they’re entering the work force? What are your thoughts around the way that we interview these candidates, and how do they compare to someone coming from a traditional CS background, like from a Stamford program versus coming from a MOOC? Just wanted to hear your thoughts.
DAVID EVANS: So I think there’s still currency in traditional degrees. But I think it should be the case that students, and it certainly is the case that many of the best computer scientists don’t have any degree. I think as a field, we’re very fortunate that computer science is probably more merit-based, there certainly are problems, but it is a field where someone can build something and demonstrate what they can do, separate from needing a traditional degree. I think it will, there is still things that students who go through that traditional university program, and I am still employed by a traditional university, so I think there is a lot of value that comes from that four-year traditional university experience for those who have the opportunity and can afford to do it. But for students who have enough self motivation and initiative, they should be able to build a portfolio that’s just as impressive as students who’ve gone through a traditional university degree.
BEN EATER: I agree with pretty much all of that. In the sense that I think we are gonna see this world where these independent certifications come up, and some of the MOOCs you’re already starting to see, they have certificates and then they’re partnering with universities to provide different things. And I think we are gonna see this world of micro credentials that pop up that say, hey this person is a great programmer, or this person is a great writer, or this person is whatever it is. And you’ll start to see people that come in, and employers are gonna have to start basically, being be the ones that drive this. Employers are gonna say, hey we value these credentials because they target what we want. I think traditionally, a university degree has been that signal that someone is someone that an employer should consider. But it’s been a weak signal. And I think these micro credentials could be a much stronger signal, because they’re specifically targeted at what the employer wants. But I think that’s half of it. And I think the other half is this portfolio of creative works. At Khan Academy, we’re starting to create that in the sense that we have our computer science platform and when you create programs, those show up on your profile. And so if you look at someone’s profile, you see, oh yeah, they know algebra, they know trigonometry and everything, but also here’s this portfolio of creative works that they created. here’s all these programs they made, Oh and also, here’s all their discussion history, and you can see all the discussion that they’ve taken place. And so you can look at that, and we’ve actually done this at Khan Academy, going on two people that we’ve actually hired based pretty much solely on their Khan Academy profile and their contribution to the Khan Academy community, because those are the things that we value.
MAGGIE JOHNSON: There are some challenges here though. [chuckles] ‘Cause if I think of, from Google’s perspective, the thousands and thousands of applications and resumes we receive, to modify the way we do our hiring and recruiting in order to take into account all these micro credentials, it’s a really big deal. Right now, we’ve got great filters in place, in terms of the academic credentials that you bring. And that allows us to really deal with the thousands and thousands of applications that we have to process. So there’s a transition that’s gonna have to happen over time, it’s not something that’s gonna happen real quickly, because it is a high touch thing you have to do. you have to deal with the person individually. And with the kinds of things we’re discussing here, you would have to assess them in really deep ways in the moment, during an interview. And that whole structure has to get set up in some way to do it.
BEN EATER: That’s what we’re doing now though. We are doing these really deep assessments in the interviews. Our interview process I think is not too different from Google’s. We get hundreds of applications a month, and we’re only a 40 person organization at Khan Academy. Our interview process, essentially, we put you through five hours of writing code and trying to see, and even that is like it’s not a great signal to us, it’s not a great indicator what someone can do in five hours, as these other, these portfolio of creative works, and these other things, which I agree is much harder to evaluate. But if there is a, if the industry can come up with a gold standard credential, that is much more rigorous than anything we’ve seen so far, maybe it involves an oral examination, practical examination, things like that, I think that does help and…
POOJA NATH SANKAR: We may also find unique opportunities come up, for example, when I was trying to find some help with front end design work, I started looking through forums and seeing who was participating. And you may find a four-person team somewhere in India trying to start a company, and they go onto this large open online class, see some students who are participating, they’re in India, they partner together, they start the company together, there could be really fascinating other benefits that are impossible today.
VALERIE TAYLOR: Okay, I know there was a question here.
AUDIENCE MEMBER: My experience is the number one challenge in teaching a college course is when you have a wide variation in the students’ background, ability, and goals for taking the course. And it seems like MOOCs are gonna face a much much bigger challenge because they have so many more students in the course. And I notice the statistics you have up on the board, we’re all very impressed with what’s been done so far, but what you’re showing over there is that by one measure only ten percent of the students finish your course, by another measure only 38%. And that’s even with the advantage of self-pacing. At a traditional university, that would be considered a disaster. I’d be fired if only 38% of my students were finishing the course because my dean would ask me what happened to those other 62%. So I wonder what your thoughts are about when MOOCs become the centerpiece, if they do, of a traditional education where you’re responsible not just to the 10% that finish, but to the 100% that start. What techniques do you foresee being used to really do mass, not just enrollment, but mass education?
BEN EATER: I should first mention that my first in person intro to CS course, the completion rate was about 40% as well, and I somehow didn’t get fired. [audience laughs] That’s, obviously, not what we aspire to in in person classes or MOOCs now. Some of this, if the barrier to get started is really low and there’s not penalty to leave, the completion rates are gonna be lower than they are for an in person class where students have to go to their advisor and are paying tuition are more tied into the class and are, in many ways, a captive audience and have to go through a lot of hoops to get out of class. So the completion rates for online classes where students had to pay tuition at the beginning are comparable to what they are for in person classes where people had to pay tuition at the beginning. So, in that sense, I’m not saying we shouldn’t be trying to make the completion rates better. But I think comparing them to in person classes where students have paid tuition and have to beg to their advisor to get out of the class, is a pretty different kind of scenario than making it very easy to start a class. Certainly, part of making the classes easy to start means lots of people start and realize quickly this is the wrong class for them. They should take something else. In terms of the stats you have there, about half the students never do anything. Maybe that means the enrollment numbers are very inflated from what they should be because you’re counting students who didn’t even watch a video. So in that sense, the completion numbers are including lots of students who didn’t do anything. But it’s also, definitely, something that we’re very early in this. And one of the things we do have is tons of data to see where students are leaving, trying to understand why from that is much harder. I think it’s still very early in this. So I’m not real concerned about the completion rates being low.
POOJA NATH SANKAR: You touched on a subtle point, which is if MOOCs replace, and I think that won’t happen, and maybe students will, their parents will make sure they go to a four year college. They pay a lot of tuition, and they actually complete the degree with no dropout rate. But this is, maybe, enabling 200 million people who can’t go to those four year colleges in America, say, and they can learn some courses. And I often am touched by women out there who now have kids, they are past their four year college degree, they didn’t choose to study CS, but now at the age of 30, they’re wondering if they can start learning how to code, because maybe they can pick up a job that’s more flexible for them to have kids, and be a mom, and also have a job. And we should be thinking of this as opening up the size of the universe, not taking from one to the other and having the same struggles, but let that be its own struggle, and let there be good completion rates in in class in traditional colleges, but then let this now enable hundreds of millions of people around the world who otherwise, maybe, didn’t get that opportunity when they were younger.
VALERIE TAYLOR: And related to the retention rate, so you realize it’s also the numbers are very different by orders of magnitude in terms of in classroom versus an open and online, but in terms of when you start to look at the percentage in terms of retention, do you think that can be a discouraging factor for women or minorities? Or do you find a difference in the retention with women or minorities or students with disability with online platforms?
BEN EATER: I think that’d be very interesting to study. We do have that data, we just haven’t looked at it. We have so much data, as do all these MOOCs, and it’s almost overwhelming. But it’s fascinating that we could study that, and we could look at that. You know, I think it kinda goes back to my point before of the self-paced nature of these things, and not all MOOCs are self-paced, the synchronous ones are not, but if you do have something that is self-paced, I think it can actually be encouraging in the sense that you can move at your own pace, you can progress through it. And even if you’re discouraged, I think, and actually going to the previous question about dropout rates, I think there are things we can do to increase engagement and keep you coming back. Gamification is one thing, or even just getting better feedback and show that, hey, you are making progress, this work that you are doing, you are getting better, or something is improving and provide you some positive feedback that what you’re doing is providing some value. I think that’s an important component of that.
VALERIE TAYLOR: And to look at what’s providing value for different groups. And to somehow tailor–
BEN EATER: I think that’s an open question, for us anyway, is to figure that out exactly. Again, that like we have this data, it would be possible for us and I think we should go back and look and see, okay, hey, is there a difference between these different groups? Where one group gets stuck versus the other group. Or if we change the way that this is messaged on this site, or we change the way feedback is given to one group versus another group, does that affect their engagement? And we’re doing those tests now, but we’re doing them in aggregate. There’s certainly no reason we couldn’t do them based on different demographics, and it would be fascinating to do that, I think.
VALERIE TAYLOR: I think, especially, the messaging.
BEN EATER: Yeah, that’s a great point.
VALERIE TAYLOR: And I know others may want to chime in there. About the retention.
DAVID EVANS: It’s something that should be studied and has not been studied. I think there are lots of opportunities to do experiments, as well as to change aspects of MOOCs, and find ways that work better for different types of students. That’s something that I think is still very early in, but it’s something that an organization like NCWIT should be pushing providers and universities to do those kinds of experiments and figure out if there are gaps in what MOOCs are doing now that can be better understood.
VALERIE TAYLOR: I think I know there were questions from the audience. We still have time.
AUDIENCE MEMBER: I think this is really interesting to hear from the for-profit/non-profit corporation perspective on the problem MOOCs. I understand that both are trying to make education more accessible to everyone. But it seems like for-profit corporations are more focused on increasing users in their own business products. So I was wondering if you guys could speak on the potential of both for-profit and non-profit corporations, working hand in hand in solving this problem of open online courseware?
MAGGIE JOHNSON: Okay. So there’s a couple different things we’re doing at Google. One is providing different kinds of educational opportunities for our users, and that is, obviously, core to our business, and it has a business value to it. There’s another aspect to what we’re doing with Course Builder that we feel is much more supportive of just open online education. Because one of the things about Course Builder is that it is opensource, it runs in The Cloud, it’s free. And the reason why Google did this, why we open-sourced this platform is because we really wanted to make it possible for everyone else to get involved with MOOCs and get involved with online education. There are many colleges and universities that are not participating with Coursera and Udacity. There are non-profits that wanna do MOOCs. There are K12 teachers that wanna do them. There’s people who wanna teach their neighborhood how to cook. Everybody should be able to do this. And one of the reasons why we have put Course Builder out there is, certainly to support everybody else, other than higher ed, which is where the for-profits are really focused, but also to really make it possible for anyone to be able to tech and learn. So it’s very much a part of why we wanted to do this.
BEN EATER: Kahn Academy, of course, is a not-for-profit, and our mission is a free world class education for anyone anywhere. For now, like a lot of the MOOCs, they are free. And so, in a sense, that is completely aligned with our mission. And so we’re very supportive of that. It remains to be seen what happens in the future. There are investors that have put money into these for-profit organizations, and for a couple years, I’m sure they have some patience. Four or five years down the road, they’re gonna wanna see some type of return. And it remains to be seen how that works. But for now, I think we’re very excited to work with anyone who is trying to get free education out there. That’s certainly our mission.
VALERIE TAYLOR: Okay, and I know there was a question.
AUDIENCE MEMBER: So there’s been some research that’s shown that when people have a bad experience in a course, if stereotype says they won’t be good at the topic, they’ll think that’s confirming the stereotype that they’re not good at the topic, or stereotypes do not say that, we’ll think instead, “I just didn’t work hard enough.” As I look at these numbers, I see 100,000 people that had a bad experience in computing. And that seems to be very troubling. Are we doing anything for those people who drop out to try to make them realize that they could have succeeded?
MAGGIE JOHNSON: Oh, one thing I think is really important to recognize with any of these numbers, like the 94K are people who registered for the course. We have the same huge numbers for our courses. So you’ve got this huge number of people that pressed the button to register and gave their email, end of story. Then they have to realize that, actually, this is a real course, and I’m gonna have to put aside this much time, and you’re gonna lose half of the people who registered because they didn’t realize how much time they’re gonna have to devote to it. So I just wanna caution about reading too much into how easy it is to register. There’s no barrier. That’s why the much more useful statistic is the group that actually came and did something. They did the first homework, they watched a video, they got involved. Because at that point, you can start to think about is the content turning them off? Is that why they are dropping out at that point. But that huge drop off from registration, we’re not concerned about that. It’s the people who start and don’t finish, that’s the group we’re much more concerned about.
BEN EATER: And in fact, there are a number of MOOCs where if you register for the MOOC, then you have access to the content after the MOOC ends. And if you don’t register for the MOOC, then you don’t have access to the content. So there have been a number of MOOCs that I have registered for because I’m like, oh, that looks interesting. I know I don’t have time to take it, but I might later want to go watch those videos, and the only way I’m gonna have access to those videos is if I register for it. And maybe they’re doing this to inflate their numbers, I don’t know, But that certainly is pushing those numbers up.
DAVID EVANS: The Udacity courses are always open to that content as always. And I think the question is a good one. I agree with what Maggie said, that the ones who don’t put any effort into it, you can say, well, that’s unfortunate, but it is the 60% or so that put enough effort into it that it seems clear they wanted to learn what the course was about, were willing to make some effort to it. Now I don’t think non finishing is necessarily a bad experience. So I’ve talked to many students that said, “Yeah, I got what I wanted out of “the first two units or the first three units, “and that’s enough to be able to start “writing some simple Python programs and “understand something about how a search engine works.” And they didn’t feel the need to learn the deeper computer science things that go in the later units. I would, as a computer scientist, like them to learn everything and stay with it. And even what it means to complete the course is sort of fuzzy, ’cause new content keeps getting added to it. So in that sense, there certainly is a large number that, I wouldn’t say necessarily having a horrible experience, ’cause if it’s a little bit bad, they can switch to watching cat videos. It’s not like if they’re enrolled in a course at the university where they have to stick through it and really having a horrible experience for three months, but are students that potentially could be attracted to computer science and aren’t being attracted by this particular course. So that’s something, certainly, I hope we can learn from. You can see anecdotes where there are students who post in the forums that, “Oh, I wanna quit because of this reason,” and often, you see great responses from other students convincing them stick with it a little more, and then they come around. But that’s a pretty small fraction of the ones. Most people who get frustrated are just gonna go do something else, and you’ll never really understand why they left.
POOJA NATH SANKAR: I think it’s really also important to understand the background of these people, and it goes into psychology. I just had a baby 10 months ago, and I was trying to lose all the weight. People I’m working out with, they’re like, “Yeah, you know, often, when you try to lose weight, “you try six times, and the sixth time it sticks “and you get on a diet that works,” or something. And the same idea, these people haven’t full time given up their job and they’re back at school. They’re also trying to juggle motherhood or trying to juggle a job. And maybe the sixth time when they go and sign up for this class, they’ll actually finish it. So understanding the background of these people is really important.
DAVID EVANS: Another thing I think this has the potential of, and it’s already happened, that there’s lots of alternate ways to get started. So if they find my course to tedious and dull with too much black and white, they can try the Kahn Academy course and get doing much exciting things right away. So there’s, certainly, what I hope will emerge is many different paths, and then you wanna find what’s the best way to direct students to the path that’s gonna work well for them. And I think we don’t know enough how to do that yet. But certainly from anecdotal discussions and if they reach out to the right people, they might get directed to, well, you’re frustrated in this class for this reason, maybe you should try a different approach.
VALERIE TAYLOR: Right. And related to that, let me just ask, how much data do you collect when students leave, of course? Because in a classroom you know when a student has dropped your course because of registration. You know the student’s grade at the time of dropping your course. So you have a lot of information available. And if you can comment on how much data do you collect or is available when you start to talk about students just not participating.
MAGGIE JOHNSON: For a couple of our search courses, we’ve surveyed that big group that never came, that never started, they registered, but they never started, and they didn’t really do the survey that well either. [audience laughs] Well, some of them did. But we did get enough information. When you’re talking about 150,000 people, if you get 20,000 of them to answer, that’s a pretty good sample. And by far the majority, it was, “I don’t have time.” That was 85 or so percent of the group that did not start. And we’ve done it a couple times now, and it’s same percentage, same reason. That was consistent.
POOJA NATH SANKAR: We did a survey with the women of WitsOn. And it’s really fascinating, by the way, the pattern in WitsOn is most people read contributions, so they didn’t necessarily ask a lot. They did name their fears. We had one post that said name your fears. Oh, it was phenomenal, all these women sharing all of their fears, and these other women mentors in industry or academia were kind of sharing their own perspective of how they had the same fears actually. But the exit survey, the two biggest takeaways, one was 47% felt more motivated to continue pursuing STEM, these are undergrad women in STEM. And 50% felt equal, and I think it was 1% or 2% that felt less motivated. The second interesting stat was 92% of the women would want to come back in a subsequent program around women in technology, and 8% were not interested. So that was also really encouraging for us.
VALERIE TAYLOR: And I don’t’ know if Ben, if you wanted to add.
BEN EATER: Yeah, so it’s a little bit different with Kahn Academy because we don’t have courses per se. Our activities are fairly discrete, you can come, you can engage, watch a video, you could watch a series of videos, you can come and engage on a couple exercise, and maybe earn proficiency in a group of exercises or just one exercise really. So we see a lot of different use cases. And some of them are just, “I have a test tomorrow, I need help.” They come, they engage, and then they leave, and that’s not a bad thing. I think we’re providing value there. We are looking at how can we get people to come back. And so we do collect a ton of data. We collect every video that someone watches, when they paused it, when they restarted it, and things like that. So it gives us a lot of information about, hey, everyone stopped watching this video at minute five. What happened? So that gives us some data there. We do the same thing for our exercises. We collect right answers, wrong answers. And so we do have this data that we can look at. We’re just starting to focus on getting users reengaged. We’ve recently started collecting emails from users. And so we’re looking at ways to email them and predicted, based on their activity on the site, other things on the site that they might be interested in, and emailing them and saying, “Hey, did you know we had this video on finance,” or whatever, “this other thing that might interest you?” And hoping the people come back and reengage through that as well.
VALERIE TAYLOR: Okay. And so, as we wrap up our panel, looking at disruptive education technologies, if you can each comment on what you think are some good things to be done to increase inclusion for women, minorities, and people with disabilities in MOOCs or online environments. And also if you can comment on, in particular, NCWIT, if you can comment on recommendations for NCWIT. So what we’ll do, we’ll start with Maggie and come forward.
MAGGIE JOHNSON: Let’s see. Well, I think that there’s a lot of experimenting that needs to be done. One of the things that’s been really clear with the discussion up here is that, we all have a lot of data, we all have the means for understanding what might be causing women and underrepresented minorities to either be engaged or not engaged in some of these kinds of learning experiences. So I think we have to look at that data and really figure out what it is that we can do that goes way beyond what we’re currently doing which is a traditional video exercise, whatever, that kind of linear sequencing. What are the other ways that we can bring more of a social aspect to it, how we can really enhance that part of it, and then also, one of the things that we’ve learned, just with our own internal engineering education is that our women engineers really want to work with study groups, whether it’s just women together or just to have a group that they can go through the content together with. That’s something that we really want to make it possible for, if you’re 150,000 person MOOC or a much smaller class, how can you find others that you can work with, whether it’s in the same room or across the globe, but make that virtual room possible. And once we are able to start moving in directions where we can really see who engages with that and who actually takes advantage of those kinds of features, we can just make it that much better and make it that much more conducive to supporting those kinds of collaborations. So that’s one thing I know that at Google we’re very much looking forward to and starting to look at what are the features that we can add to Course Builder or that can be added to MOOCs that are going to make this more conducive to different types of groups. And NCWIT, I think, is obviously a group that has a lot of this kind expertise. So being able to tap into that is gonna be something really useful.
BEN EATER: Yeah, and I think from Kahn academy’s point of view, I would echo a lot of what Maggie said, you know, we have a ton of data, and we really need to start looking at what that data can tell us, and find better ways to engage with users based on that data, see what differences exist and all of that. I think one of the things that we are doing that has been quite effective is kind of what I’ve been talking about all along, is this self-paced learning where folks can engage and move at their own pace. We’ve seen that to be hugely empowering in the sense that someone who may have been left behind in a traditional classroom or maybe had some bad experience or some discrimination, or some type of event happen where they were taken out, or discouraged, or something, and then they got caught behind and never caught up. I think the ability for them to reengage, move at their own pace, catch up, and get ahead is huge. And I think one of the things that you all can do, you’re all change leaders, you have organizations that you have some influence in. One of the problems that we have as Kahn Academy is, when I speak to a room like this, I asked everyone, how many of you are familiar with Kahn Academy. Almost every hand in the room went up. But if I go to inner city or somewhere where there’s underrepresented minorities and things like that, and its’ not some kind of Kahn Academy event, nobody knows who we are. We’re providing free tutoring, and this is what those folks need. What I would ask of you all is really just to go out and raise awareness that, hey, there’s this free tutoring thing that exists, and you can engage in it, and helping set up after school programs in the different organizations that you’re involved in would be huge. Because that physical environment where students can come together, work together, it is really key. ‘Cause when you get that peer validation, you can work at your own pace through some of the things on Kahn Academy, you can master certain concepts and now you are a pure tutor to your peers, because you mastered this thing, and that’s very empowering to to teach others. So you know, that’s a big way that you all can help us.
POOJA NATH SANKAR: I’d like to share, again, what inspired me to start Piazza was feeling isolated and I never wanted any other woman specifically studying a STEM science engineering math subject to ever feel isolated. And today I’ve noticed that our students, on average, spend three hours a night on the site, and this is boys and girls. And so I believe that, from my own empathy for a student who can feel very shy or scared, I’ve been able to build a product today that brings students together, classmates together, and it goes back to whatever movement however we think about getting education to the masses, how we can create a very comforting environment for them so that they feel confident and want to finish. I really believe women studying computer science who are able to code, can create a future for themselves that is very favorable to a balance between family and work. I’ve seen that in my own life. When I was pregnant in my first trimester, I would go home and sleep all the time, and the team was very understanding, and I want to be able to to create such opportunities for more women. So being very understanding that minorities, I think, need a bit more of a gentler hand in understanding that they, perhaps, feel shy in ways the men may not be able to understand.
DAVID EVANS: So I think open education has this tremendous promise and potential. And we’ve heard a lot, and Michael and Ben and others have talked about privilege. If you look at the students that are being reached by open education now, a very large fraction of them are coming from privileged. And you can see on this slide that’s been up for the last hour and a half or so, that little tiny blue sliver is Africa. 2.2% of the students who finish cs101 are from Africa, and about 17% of the world’s population is from Africa. So in terms of looking beyond the superficial diversity that we tend to look at in the US, and I understand NCWIT has a mission focused in the US looking at globally this opportunity for open resources, open education, to reach parts of the world that are underprivileged and don’t have the kinds of opportunities that almost all of us have in the US, that it’s not happening yet. But the opportunity is there, and hope that people will put pressure on both the for-profit companies, as well as the non-profits, as well as the universities, to be aware that this is the huge opportunity that open education has, and something’s wrong if five years from now our mix of students still looks like what’s on that slide. The more narrow thing I’ll mention that I think many people in this room can contribute to is the subtle messages about whether someone belongs or not that happen in NIT, and I think very few of the people teaching computer science are outright sexists. I hope that’s the case, and I think it is true. But there is also a pervasiveness of language, and examples, and ways of talking that often give women the impression that they don’t belong. And one of the things I tried to do when I was at Udacity, I was reviewing scripts for courses and was looking for things like using generic he, using ninja, using you guys, using the kinds of language that is commonly used in every classroom in computer science in the country, well, maybe not every classroom, but a large fraction of them, that gives many women, especially if they’re in a class where they already look around and see that 10%, 20% of the students are women, the sense that they don’t belong here. And that’s something where I think the openness and public nature of MOOCs mean that there is more of an opportunity to say, well, this is a place where we can fix that and pay close enough attention, because this is gonna be visible to more people, and I hope I succeeded in my courses in not doing that. There are probably places where I was not as aware of cultural references or things that sounded exclusive as I should’ve been, and that’s something where I think there is an opportunity and a value that people that are attuned to that can provide to people creating course. Once the courses are there, one of the values that a recorded class has is it can keep being changed and improved. So noticing things like this and putting pressure on people to fix them. So I think that’s a fairly small thing, but I think that people in this audience have the right attitude and can play a big role in changing that.
VALERIE TAYLOR: Okay. So with that, let me just get our NCWIT t-shirts to say thank you to our panelists. [audience applauds]