The Ed Techie

Art of Guerrilla Research workshop

On Monday I ran a workshop with Tony Hirst on the Art of Guerrilla Research. This was a vague idea I'd floated a while back, and Rhona Sharpe of ELESIG got in touch, asking if I could run one of their masterclass workshops on it. This was a good opportunity to think through the idea with others.

With tongue a bit in cheek I proposed a manifesto for Guerrilla Research which was: 

• It can be done by one or two researchers and does not require a team
• It relies on existing open data, information and tools
• It is fairly quick to realise
• It is disseminated via blogs and social media
• It doesn’t require permission

It is that last one that is most significant I think. This is stuff you can just do, it doesn't require funding, permission from IT services, access to privileged data, etc. It is thus more exploratory in nature, Tony used the phrase 'recreational research' (I hope Tony blogs his slidedeck soon). Another point I was trying to explore is one I've made before, that we have become enculturated into thinking about research in a particular way. What constitutes research is the 2 year funded project with a journal article at the end. Like much else to do with digital scholarship, it is not the case that this traditional approach is not valid, but rather that we now have a much more extended toolbox and set of possibilities. But culturally we still fall into a certain set of behaviours.

It was interesting that a lot of the people at the workshop classified themselves as being outside traditional academic roles, so couldn't engage in traditional research anyway as it wasn't part of their remit. For these people guerrilla research is all they've got.

The slidedeck is below:

The Art of Guerrilla Research from Martin Weller

 

[Update: here is the video of the session, it's a bit quiet]

 

Open research ethics - the puppy killer scenario

Yesterday I ran a workshop called "The Art of Guerrilla Research" for ELESIG, along with Tony Hirst. I'll blog it later but basically it was about what sort of research can you do without permission and funding, eg asking questions of open data (hence Tony describing the things he does).

One issue that was raised a few times was that of the ethics of it. The assumption has long been that anything openly available is fair game. So for instance there is a lot of research that uses travel blogs as its data source, and they don't require the permission of these people to analyse them or interpret them. In general, this is my stance too, but thinking through the types of things Tony does with data led me to come up with a scenario which would raise ethical issues. I offer it just as an example of how it isn't quite as clear cut as you may think regarding openly available data.

Let us imagine that there is a heinous crime we can all agree is very bad - puppy murders (I'm using a silly example so people won't get distracted by a specific crime, but you can replace puppy murders with a small or large crime/amoral act of your choice). Tony does a FOI request to find all the people convicted of puppy murders over the past decade. He then finds which of these have Facebook pages that are openly available. He creates an interest graph of their listed interests, and shows that puppy murderers tend to have a number of interests in common. He blogs this, just out of interest.

Someone else then comes along and finds all the people on Facebook who also have these interests, and publishes a list of 300 people who have 'puppy murderer' type interests. One of these, although entirely innocent of any puppy mistreatment, is attacked by a mob who accuse him of being a puppy murderer.

Now, this has used all openly available data, publicly and knowingly shared by the individuals. But by taking it and creating a new interpretation of that data, new knowledge has been generated which the original posters could not have foreseen. The new form of this knowledge then carries an ethical dimension. This is obviously an extreme example, but it illustrates the potential complexity of assuming all open data is fair game.

Design responses to MOOC completion rates

Well, my previous post on data for MOOC completion rates caused a bit of a kerfuffle on Twitter. It was interpreted by some as saying "ONLY completion rates matter". And also of not taking into account other factors such as what learners who don't complete get from a MOOC. That seems rather like criticising Alien for not being a rom-com to my mind - they're doing different things. This research was showing one aspect with the quantitative data available. It is part of a bigger picture which ethnographic studies, surveys and more data analysis will complete. It wasn't attempting to be the full stop on MOOC research.

Anyway, here is another graph that Katy created, showing attrition rates of active users (those that come into the course and do something, not just those who complete assessments) across disciplines:

That's a pretty consistent pattern. If we saw it nature we'd give it some name like "The MOOC attrition law". My interest is as a course designer, so given that the drop-off pattern seems fairly robust, what does it mean for design? (Doug will have issues about the power-lawness of this) 

I think there are two responses (but maybe you can think of more).

Design for retention

The first is to say that completion is a desired metric. There may be courses where you really do want as many people as possible to complete. Imagine you were running a remedial maths course for instance, then it won't help your learners much if they only cover a third of the subject matter they need for whatever purpose (Bridge2 Success got learners through maths so they could get onto an employment program, so completion was very important here).

In this case you need to address the 'problem' of drop-out, because it is a problem for you. There might be a number of ways you do this: by adding in more feedback, using badges to motivate people, creating support structures, supplementing with face to face study groups, breaking your longer course into shorter ones, etc. The point is that you design in features that aim to improve completion.

Design for selection

The second design approach is to say that completion isn't an important metric. Here you accept the MOOC attrition law and design the experience with that in mind. I have some sympathy with Stephen Downes when he says no-one finishes a newspaper but we don't talk about people 'dropping out' of a newspaper (I've heard him say this but can't find a link - anyone?). So even to talk about drop-out is to map the wrong metaphor to MOOCs. His analogy breaks down a bit however because not all readers drop out at page 7 of a newspaper, they dip into different sections. People tend to drop out of MOOCs by week 3. It's not as if they're coming in and doing a bit from week 7 and a bit from week 5, and then leaving. They're simply not getting to those later weeks. And even if you are of the 'completion doesn't matter' camp, I'm sure most course designers don't think the content in week 5 is half the value of that in week 1. 

So, in this design approach you might break away from the linear course model, to allow people to do the 'newspaper' type selection. A course might be structured around themes for instance, and each one around largely independent activities (I tried to design H817open a bit like this). So in this case completion really doesn't matter, learners take the bits they want. 

In both cases I would suggest that the completion rate data is useful for you. In the first case you know what type of completion rate to expect, and in the second one it drives you to be more innovative in design approach. And that's the point about the research - it helps inform decisions.

By the way - this is my fifth blog post in 5 days. Just in case Jim Groom berates me for not blogging often enough...

Completion data for MOOCs

As I mentioned in the previous post, I am doing some Gates funded research on MOOCs. My part was learning design analysis, while Katy Jordan has been looking at factors influencing completion rates. All this work is Katy's, I take no credit for it. She would blog it, but is about to have her first baby any day now, and strangely that has taken priority over blogging about MOOCs, so she said I could blog it on her behalf.

There will be a paper that details the full results and methodology, so I'm just giving some highlights here. Katy collected completion data from 221 different MOOCs. The range was limited to ones where we could get this data, so is biased by a couple of big data sets from Coursera and Open2Study. This last one in particular has good completion rates so may have biased the overall results somewhat. 

If we start with enrolment figures first then, the picture looks like this:

(N = 220; Median = 18941; Minimum = 95; Maximum = 226,652). Although the really big numbers get the headlines, there are a lot of courses in the 0-10000 range.

Completion rates across this sample are:

(N =  129; Median = 12.6%; Minimum = 0.7%; Maximum = 52.1%). This bimodal distribution may be a result of those Open2Study courses. Those guys appear to be doing something right in terms of completion.

Does enrollment numbers relate to completers? The answer is yes, there is a significant negative correlation, ie the more people who enrol then the lower the percentage who complete. This single graph might well put paid to the idea that we can educate the world with massive courses:

Yeah, I hear you say, but it's not completion that's important but how many people are active. I'm afraid that shows exactly the same picture, ie there is a significant negative correlation between number of active users and the percentage who complete:

Does the reputation of the university matter? The answer is: possibly. There is a positive correlation between the Times Higher Ed score of a university and enrolments:

But this could just be because more elite universities have joined the elite clubs of MOOC providers. As with all correlation we can't imply causality, it may not be the university reputation that leads directly to the higher enrolments but a related factor. Anyway, it turns out that reputation has no effect on completion rates:

Have enrolments increased or decreased over time I hear you ask. Well, there is a negative correlation over time with enrolments, so numbers on individual MOOCs have declined. This may be that with more MOOCs available people can find the one that suits them better, instead of everyone doing AI, but if you wanted to be all sensationalist (I'm looking at you, journalists) you could say it indicates that the MOOC bubble has burst:

Have completion rates changed over time? Yes, in the opposite direction, there is a positive correlation with completion and date. This could mean people are getting better at MOOCs, and if the trend continued those completion rates would be less of an issue. But I suspect it is related to the point above, in that there are more MOOCs around so people are taking ones more closely related to their interests, so more are completing:

(see http://www.katyjordan.com/timeline for more detail)

Does the course length have an impact on completion? You bet, a negative correlation with course length, so the longer your course, the lower the percentage who see it through. This is probably obvious, but it shows that if you get to 12+ weeks it's probably just some bloke in a shack in Arkansas left:

This raises the question, should we create MOOCs of 2-3 weeks in length (many of Open2Study courses are this long, which may account for the good completion rates)? And if we're creating 2 week MOOCs, how are those different from OERs? UPDATE - see Doug's comments below. He makes the point that obviously the longer a course goes on, the more people will drop out. So we can't say length causes drop out, although even in this respect, you might be better off having 2 x 3week MOOCs than 1 x 6 week one, simply because you catch more people (if completion is important that is).

And a tantalising one to leave you with: does gender influence completion rates? We can't tell as only a few data sets included breakdown by gender, 13 in this case. So we couldn't draw any conclusions from such a small sample, and Katy will probably tell me off for including this, but I do so as a plea to others to release data. From the sample we had, there is a possible positive influence on the percentage of female students and completion, ie women tend to complete more than men, who no doubt wander off to watch football during week 3:

We looked at a number of other factors including assessment type, MOOC type and platform, but the variation in sample sizes makes conclusions from these difficult to form. This data raises a number of interesting questions, which being correlations it can't answer on its own. For me these are:

1) Can we improve those completion rates from around 10%?

2) Are learners getting better at MOOCs?

3) Should we limit ourselves to short MOOCs?

4) What demographic data may influence completion rates?

5) The big one is, does completion even matter? (If it does, then MOOCs probably aren't the solution to your problem)

The iceland of Dallas

 

<Dallas deathstar in the snow - this may, or may not, be a metaphor>

I was at the MOOC research initiative conference in Dallas, Texas last week. As Jim and others have reported, we got caught in icemageddon, but that's a whole other (war) story. I'll be doing a few posts about the conference. It was a fantastic meeting, well done George Siemens, Amy Collier and Tanya Joosten for putting it together. I got to have some great conversations, and meet people I've know online for years. Which is by way of apology for my first post being a bit negative.

This one concerns one aspect of the conference that I am having difficulty articulating, so I'm going to try and work it through in this post. There was a data strand to the presentations, and I went to a few of these. There is some fascinating stuff being done, particularly when you have analytics on so many learners. But I also had a vague sense of unease about some of these.

They were often presented by super-smart, young computer-science researchers from privileged universities: Carnegie-Mellon, Stanford, Harvard. I don't have anything against super-smart, young or computer science people, or anyone from those universities (some of my best friends are young people :). But there was something a bit, well, cold, about it all (and not just because of the wind chill factor of -10). People were nodes, and they could be manipulated to move from peripheries to the center by tweaking certain elements. It was easy to forget you were talking about learners, and not sales of baked beans.

But I think we need this research, it's useful and can tell us a lot about what's going on. Candace Thrall was stuck at the airport so couldn't give her presentation, but a colleague gave it on her behalf. She mentioned that one of the transitions we were going through was from a theory-led one to an evidence based one. Prior to this Jim Groom was telling me about Mike Caulfield suggesting we were in post-theory now, where only the big-data mattered (this was from a book I think, if anyone knows which one, let me know). 

UPDATE: Mike has done a great post elaborating on the post-theory debate here. It explains what I was trying to get at in this post.

I felt I had a glimpse of that post-theory world, and I wasn't sure I liked it. We may have been too theory-heavy before, where the evidence was inconsequential, because hey, we have a nice theory. But the pendulum swing to lack of theory where we only care about the evidence seems to lose sight of the people in the system. So I guess my plea to the super-smart, young computer science researchers at ivy league institutions who are now getting into to education is - don't ignore the bearded old guy with a bunch of theories in his back pocket, we need those too. 

Like Dallas, education should be warm and welcoming, and the danger is that data fetishisation will make it like the post-apocalyptic Dallas I experienced.

The art of guerrilla research

<Image http://www.flickr.com/photos/idfonline/5981013497/>

In my presentations on digital scholarship I often make the claim that we have the opportunity to rethink the form that research takes. We are accustomed in academia to thinking of research as being of a certain 'size'. Usually this means it is funded research or something with a traditional output (research paper or book). But digital, networked technologies allow us different ways of approaching research. As I am forever saying, this is not to say they supplant the existing methods, or are superior to them, just that we have a richer mix of options now.

I've started calling the 'just do it' approach 'guerrilla research'. This term has been used in software design, for example Ross Unger and Todd Warfel argue persuasively for it, claiming that "Guerrilla research methods are faster, lower-cost methods that provide sufficient enough insights to make informed strategic decisions." I think there is a lot to be said for this approach to academic research also.

What might guerilla research look like? Well, as I've mentioned before, my PhD student Katy Jordan did some work on MOOC completion rates using open data that has been widely used. Or Tony Hirst likes to play with open data sets (although as he pines, it's not as easy as it was). Or a bunch of research that analyses travel blogs.

Guerrilla research is characterised by the following as I see it:

1. It can be done by one or two researchers and does not require a team
2. It relies on existing open data, information and tools
3. It is fairly quick to realise
4. It is often disseminated via blogs and social media

Guerrilla research needn't be in competition with formal, funded research. In fact it's a good way to get started on this. For instance, Katy's work has led to funding from the MOOC Research Initiative. If you want to demonstrate to a funder that your project is worth investing in, then being able to show some interesting preliminary findings is useful. And not just to say 'these are interesting' but to be able to demonstrate tangible interest because when you blogged it, the post had X many hits and was retweeted Y many times.

I feel we have a tendency to spend a lot of time trying to build up to a proper bid, when much of that work could be done in the open. And if you are a researcher struggling to get funding, then guerrilla research may offer a route to maintaining a research profile while you are waiting for that REFable grant to be realised.

What do all these numbers mean?

<image - http://www.flickr.com/photos/mmorgan8186/5946796450/>

Bloggers, or anyone who maintains an online profile, have an ambiguous relationship with visitor stats and data. On the one hand we like to dismiss them as meaningless, but then secretly feel chuffed when we can outscore someone. I've tried to promote them as one way of measuring impact, but with the caveat that context is important. For instance, if you're a blogger in a relatively obscure area, such as Barry Town football club, then your range is limited and unlikely to compare in absolute numbers with, say, a blog reviewing Apple products.

I recently passed 300,000 views on this blog, over about 700 posts - that's not as exciting as it might sound as I've been going since about 2006. My friend Liam says he gets about 200K a year on his mobile tech blog. Here are some more stats from tools I use:

• Slideshare - 220,000 views over 6 years on 59 presentations
• Blipfoto - 92,000 views over 420 entries
• Flickr - 49,000 views over 1,140 photos
• Citations on academic articles - 1,620 (according to Google)

The first thing to ask is how reliable is this data? Tony can probably answer that better than me, but I think the blog traffic doesn't take into people who subscribe via RSS, so in that case underestimates. At the same time it does include my own obsessive self-clicking, so that may even out. It may also include bot traffic which doesn't really mean anything at all.

But if we take the data as accurate for now, the more interesting question is what does it mean? I would like to make the case that 300,000 visits to my blog equals 300,000 careful, considered reading of my posts, but the data doesn't back that up. Google analytics tells me that my average visit duration is 57 seconds. I suppose that is just about the time it takes to read a blog post, but it probably doesn't compare well with time taken on reading academic articles. My bounce rate is high too, at around 85%, so people come in, read a post and then go elsewhere, but I think that's fairly typical for blogs, I'm not providing a newspaper where you want people to go from one feature to another.

Blipfoto probably has the highest return on audience per post, and does better than Flickr. This provides a nice comparison as I have linked my blipfoto account to twitter, so it automatically tweets when I post, but I haven't done this for Flickr (which I think of more as a repository). I get 219 views per post on blipfoto versus 42 on Flickr, so we could say that the effect of linking to twitter is a 5-fold increase in traffic (if that's what you want). But I would say that Blipfoto is the most personal of these tools and the least connected to my professional life, so I wouldn't argue for any professional impact here.

What I think we're struggling with here is the newness of all this. What is the value of a slideshare view versus a download of a paper? Is there a conversion rate? I've tried to keep a foot in both the traditional camp, through publications, and the new forms of dissemination, so these figures might offer a reasonable comparison between the two. Looking at my citations you could crudely say that one citation is worth about 200 blog views (or vice versa).

However, it's probably futile to try and measure these things. It's not as if citations are a particularly reliable metric anyway, so we shouldn't use those as a currency to convert everything else into. It will also vary widely depending on discipline, and individual. And more importantly, we already have plenty of systems that our career path requires us to game, so why invent more?

Make no mistake, turning altmetrics into performance measures linked to real things like promotion, job security, money would have negative effects. For example, there is a correlation between number of posts and views, so if I was purely chasing the numbers I should post more often, regardless of quality (what quality?), or I might think I could get better numbers from splitting this post into three separate ones. This already happens in traditional outputs, I remember someone advising me not to write books because for the same effort I could have 5 REFable publications. This return on investment view ignores the internal validity of the form. I also tend to think of blogging as a social function, connecting with friends and peers, but if I'm linking it to direct returns then it's another example of monetising relationships that Brian is so uneasy about.

But when I've tried to make the case for digital scholarship and tenure, it is something along these lines that I want to argue - that the impact you see here is as valid as the impact we have chosen to recognise and measure. So we need to start thinking about what these numbers do mean for academics - we just need to make sure we don't turn them into the thing itself, with all of us chasing and parading numbers. But maybe that's inevitable as soon as you shine a light on them, in which case, ignore this post.

The random REF

<Image from gringer http://www.flickr.com/photos/gringer/5096129532/>

As a senior member of staff (I know, how did that happen?), part of my duties currently include reading papers for the UK research assessment exercise, the REF. I've moaned about how this reinforces traditional publication models previously. This is a more general moan.

I think the REF has muddled objectives. It aims to make academics accountable to some extent, but primarily acts as a means of allocating research funds. It may have some secondary aims, or you could classify these as indirect consequences, such as reinforcing the position of the main publishers and maintaining the status of Russell Group universities.

When you have multiple objectives the resultant process is often unsatisfactory. So if we ignore the secondary aims (although I wish the Finch report had been brave enough to make clear reference to the REF in undermining the move to open access), then how might the first two be best served?

Let's assume for now that you want to go along with accountability drive. There are of course plenty of reasons why this may not be a good idea, but that's for a different post. As someone who has to read papers and try and judge them, the real problem with the REF is that it brings in quality judgements. I thought that educational technology was a special case in that it didn't fit well into a broader education category and so the people making assessments weren't sufficiently well acquainted with the subject area to make decent judgements. But I've spoken to people in other disciplines and everyone feels the same. The specialisation of research means that no categorisation can be fine enough to really capture it. And it's a fruitless task anyway, who can judge the impact or quality of a paper? I think you can spot a poor paper easily, and maybe a really excellent one, but 90% of the rest sit in the middle so judging whether something is a 2* or 3* is essentially a coin toss.

So, my suggestion (if you want to do the accountability thing) is to simplify it and just apply a simple gateway threshold. So, for example you could take the REF guidelines - in four years someone who has a research part to their contract should be expected to publish four refereed articles, have got one piece of research funding in and supervised one student. Against this simple criteria you can then have accepted trade-offs: running large university projects, digital scholarship equivalents, personal consultancy as funding, etc. But you remove the quality judgement, it's a simple measure that this is what a research active should be expected to deliver. I don't necessarily agree with it, but we'd all know what the rules were.

The next element is to allocate research funding. You could simply do away with direct funding and have it all come through research councils, as they do in some countries. Or you could distribute it evenly, or according to number of staff who've passed the threshold above. But I've got a more interesting suggestion - there was some maths research (from those crazy Italians) recently that argued that random selection of some, but not all, politicians would benefit democracy:

"the introduction of a variable percentage of randomly selected independent legislators can increase the global efficiency of a Legislature, in terms of both the number of laws passed and the average social welfare obtained."

I wonder if the same might apply to research funding? So some is allocated according to worth (but using a simple system), but a percentage is allocated randomly, on the proviso it is used for research. This might well generate innovative research we wouldn't see otherwise. And it prevents a self-reinforcing group who get caught in version of group think. And it'd be more fun.

I know what you're thinking - why isn't he in charge of the next REF? We can only shake our heads.

The research Hussites

[This is another from my catalogue of strained metaphors, and my grasp of religious history is rather tenuous, so I'm sure people who are better acquainted with the subtleties of Hussite history can point out lots of flaws with it. But take the surface points as of interest.]

I've been giving a talk recently called "Digital Scholarship: 10 Lessons in 10 Videos". I'll blog these in detail later, but one of my lessons is that we should rethink research. By this I mean we have a certain attitude towards how research is conducted, which was shaped prior to the arrival of digital, networked and open technologies. Some of that attitude still works, but there are also a host of possibilities that remaining wedded solely to that view prohibits.

An aspect of this that interests me is the Do It Yourself (or Do It Ourselves as Tim O'Reilly has it), do it now approach. In my talk I give three examples from my own experience:

• Start your own journal - for example my MetaEdtech Journal
• Build an app - eg our Facebook apps
• Interrogate data - such as Tony's playing around with tools and social data

But you can think of many other examples: Jim Groom's DS106 mega experiment or the MOOC approach from George Siemens and co can be seen as research and experimentation in the open.

What I find fascinating about these is that they didn't need permission. As Larry Lessig points out in his review of The Social Network, it is this removal of the permission filter that is really significant:

"what’s important here is that Zuckerberg’s genius could be embraced by half-a-billion people within six years of its first being launched, without (and here is the critical bit) asking permission of anyone. The real story is not the invention. It is the platform that makes the invention sing."

Back to research then, and our ingrained attitude goes something like this:

1. Come up with an idea
2. Write a proposal
3. Get funding
4. Do research
5. Publish

Obtaining the funding is often an absolutely necessary step - you can't build a Large Hadron Collider in your back yard after all. But it also performs another function - it validates research as something worthwhile. So ingrained is this approach, that if you don't get funding, it doesn't count as research. We don't question it anymore. 

But with the arrival of new easy to use tools, open data, a network of peers and cheap or free storage, you can do a lot of research now without funding. The video I show for this lesson is this one from Derek Sivers. He's talking about starting a business, but a lot of it could apply to research also:

So what's all this got to do with a 14th Century Czech Priest? Well, Jan Hus argued that we didn't need priests, that everyone formed part of the church and the Papal hierarchy should be undermined. I'm not making any comment on his religious argument here (he says, trying to avoid offending anyone). There is an analogy with this approach to research though. For the Catholic churches read research councils. We are all the research councils, we don't need permission to conduct our practice and we don't need their approved channels to reach the audience. 

My argument is not to overthrow research councils, we still need them, but to propose that often there are low to zero cost alternatives available which might get at some of your research question. We should consider these at least without always defaulting to the funding = research model.

Mind you, poor Jan didn't have a happy ending, so maybe I should stay quiet.

Bridge 2 Success

This is the third in the what I'm up to at the OU posts. A while back we were part of a consortium that won a bid in the Next Generation Learning Challenges funded by the Bill and Melinda Gates Foundation, Educause, & The Hewlett Foundation. The partners are us (IET at the OU), MIT, the Anne Arundel Community College and University of Maryland University College (UMUC) - the project is Bridge to Success.

It's aim is to use OERs to help community college students who struggle with study skills, and math(s) in particular. So we've taken some OU courses and these have been adapted in the US, and made available through our OpenLearn labspace. But this is about more than just creating content, so these materials are used in classroom instruction, and we are researching how students and instructors find the process.

My role is to head up the research work programme (Patrick McAndrew is the OU lead and does all the real work). One of the things I'm interested here is in the process of adaptation. We have given our materials to US partners to adapt, and what we are looking at is the type and degree of adaptation that occurs. There are at least four types of adaptation that take place:

1. Simple context - changing £ for $ is a simple way of thinking about this
2. Language adaptation - changing the nature of the instruction that is given in terms of the style used
3. Content adaptation - reworking, deleting or adding content so that it meets the needs of a US maths curriculum
4. Pedagogic adaptation - altering the nature of activities and instruction based on knowledge of the types of students

Now, you probably want to get away with just doing type 1 adaptation as that is fairly straightforward and takes the least time. But you still need subject experts to do this, and it quickly becomes apparent that it bleeds across to the other types of adaptation too. It is all context. 

So what will be interesting to see is just how much adaptation takes place, whether some instructors adapt more than others, and if this has any impact on student performance. There undoubtedly comes a point when you have adapted something so much, it would have taken as much time to create it from scratch. The argument here then is whether the OER approach is worthwhile, if it takes as long to adapt as to create? Even if this is the case (and we haven't found that to be so yet, but it's still early), the use of OER may lead to better material than creating from scratch - this would be a tough thing to test, but maybe for another project.

Those are the kinds of things I'm interested in, beyond the more practical (and worthwhile) outcomes of hopefully improving student performance, confidence and understanding.

Here is a presentation from MIT's Brandon Muramatsu on the project if you want to know more:

Demystifying OER and Bridge to Success

View more presentations from Brandon Muramatsu