The Ed Techie

Ten digital scholarship questions for researchers

<Image http://www.flickr.com/photos/elycefeliz/3262326159/>

In my talk the other day for the Change MOOC, and in my book, I argue that digital scholarship has the potential to alter the way we conduct research, or at least to add some new tools to the toolbox.

Now checklists should be treated with even more caution than a finding that 3D improves learning, so take this as some things to think about rather than a definite checklist you can submit to your research funder. I'd be interested to hear of other suggestions, and also whether this is all stuff people are doing anyway. My feeling is that when it comes to writing research proposals we're all locked into a set of existing behaviours, and more significantly, the process is so arduous that you focus on surviving the process itself and not on thinking beyond it. In other words if the research funders aren't asking for it, we don't consider it. But if we (and by we, I mean researchers) want to take ownership of our own practice, then maybe we should.

So here's my list of digital scholarship type stuff to consider for your project:

1. Do you have an open access publishing policy for the outputs of the project?
2. Will you make the data openly available, or is some of it not appropriate to do so? If so, where will you put it?
3. Will there be one main individual in charge of social media communication or is it just distributed across the group?
4. Will you have specific twitter/blog/youtube accounts for the project or do individuals in the project have a high online reputation it would be better to utilise?
5. How will you incorporate analytics into the project? Do you expect to get a certain number of views, dwell time, global distribution, etc on a main site?
6. What media will you use? For example, will you create a 'trailer' video for the project and an overview of the findings?
7. How will you archive discussion around the research, eg twitter hashtag conversations?
8. Will you provide a curation service, eg a Scoop-It page of relevant resources as you go along?
9. Are there new methodologies you will employ, eg crowdsourcing?
10. Is there a planned release for findings throughout the project? Will any aspects be not open for dissemination eg via twitter or blogs?

I'll stop there because ten is a nice number.

A quick and dirty REF

I am currently engaged with working on the REF submission for my Institute. I fully understand why we do this, in cash strapped time in particular, research money will make a big difference. It is also almost impossible not to play the game if you want to be recognised as a research institution and attract staff.

But it is a horrible, pernicious process that seems deliberately designed to kill innovation, perpetuate existing models, waste time, focus energy on gaming the system and keep publishers happy. If you wanted to design a process with the sole aim of restricting the adoption of new methods and technology, then you could do a lot worse that the REF.

Not every country has an equivalent of the REF, so we shouldn't view it as a necessary evil. But, let's say that for whatever reason, something like it is going to persist. Two questions then arise: can it be better and can it be easier?

I'm not going to answer the first of these questions. It would take too long, and I've covered it lots of times before. But in a conversation with Tony the other day we pondered how the REF might be made easier and a hell of a lot cheaper. The total (estimated) cost of the 2008 RAE in England was £56.7 million, which compares with the £1.5 billion of QR research funding. They estimate costs for the REF will be similar. My guess is this figure is low, and may not take into account all of the meeting and individual time associated with the process, and probably not opportunity costs also. In a review of the 2001 RAE, for more teaching oriented institutions it was found that the "RAE is over 16 times less efficient than the norm". I don't think the REF can be argued to be more efficient as it largely follows the same methodology.

So here is my (well Tony's mainly) proposal for a quick and dirty REF:

• Research councils monitor where money goes and evaluate research projects
• Publishers have databases of authors and citations

The REF then centrally datawarehouses these sources, adds in various weighting factors, creating a score for individuals, which is aggregated for the institution to give their REF evaluation (maybe by subject or maybe across a whole institution).

It uses proxies for research quality, and is focused on research funding and formal publications, but hey, so is the current REF. If we're going to be forced to play that game, then couldn't we at least do it in an efficient manner? It would at least free up a lot of time to you know, actually do research.

It would at least be worth doing a pilot based on this to see how far it differs from the very labour intensive system we have currently, don't you think?

 

More brains! The perpetuation of the zombie scholar

With Jim Groom I'm writing a chapter for the Zombies in the Academy book. It's at an early draft stage, but thought I would share it in this wiki.

My argument is that the context we have set up in education for all aspects of scholarship is like a contagion that works against innovation. This was an idea I explored in my researchers and new technology post and which forms a theme of my upcoming book (but with no zombies). There is an analogy here with the manner in which zombies turn all-comers into one of their own, which is exemplified by this clip from Dan O'Bannon's Return of the Living Dead:

 

The scholarly discipline is in ever more need of fresh brains to feed itself.

But I propose that, unlike zombification, there may be an antidote: the adoption of the fast, cheap and out of control technologies espoused by Brian Lamb.

Anyway, take a look at the paper and if you feel like contributing let me know.

The lack of uptake of new technology by researchers

Yesterday I gave a presentation for George and Stephen's open course PLENK. You can see the recording of the session here. The slidedeck is below:

Research, technology & networks

View more presentations from Martin Weller.

I have been writing a chapter on research and how researchers are (or rather aren't) using new technologies for my digital scholarship book. Several good surveys and reviews have been published recently looking at this, and the overall picture is a rather depressing one (see some references in my Mendeley list). While there are islands of innovation, generally researchers are making little use of new technologies and are very cautious and conservative in their adoption.

I talked about why this might be so and suggested the following reasons:

• The stranglehold of publishing - tenure and reward are still linked to publishing in academic journals, so spending time on other forms of communication is not directly encouraged. This is reinforced through research quality assessment exercises which are closely allied to publishing, such as the REF.
• Wasting new blood - related to the above, in other sectors new recruits are seen as the people who bring in new ideas. Yet in education because tenure is linked so closely to traditional outputs, new researchers are positively discouraged from spending time on other endeavours. So the very people who might engage in transforming practice are discouraged from doing so. This leaves it to tenured academics to engage in innovation, but these have mostly been successful through more traditional means so their history tells them to carry on this way. The pool of potential adopters then is decreased by the research context we have in place. I suggested this was quite a harmful environment to create.
• Antithetical to research ethos - I also put forward the suggestion that in science in particular, research is about predicting outcomes, and much of web 2.0 (or whatever you want to call it) is about unpredictability - you don't know what will happen with your outputs, or through a network.
• Research is the core practice - despite how much academics might say they value teaching, or public engagement, it is still research that is most highly valued. This means it is the area people are least willing to risk their reputation in and want to control most tightly.

I then suggested some themes that I thought were emerging, but that is for another post.

Of course one might argue it doesn't matter. Researchers will adopt new technologies if they are effective, and there is an underlying assumption in my talk that adoption is a good thing, which isn't necessarily true. This may well be the case, but there are three responses to this which at least bear consideration:

• The impact of the negative context - if research adoption of new technologies (and associated practices eg open publishing) is being impaired by the financial/reward structure we have in place, then we need to address whether that structure is doing research a disservice in the long term
• Powerful tools - I know through personal experience that even when people have been dismissive, or cautious of a particular tool, when it is demonstrated in the right context, or they appreciate its possibilities, then they will find uses for it in their academic work that I had never anticipated. This isn't true for everyone or for all uses, but it is worth chipping away at misconceptions and demonstrating potential.
• The role of the researcher - as higher education seeks to demonstrate its value to society, it strikes me that it is potentially dangerous not to be leading on the use and development of new tools, and particularly on what they mean for knowledge construction and sharing, rather than leaving it to others.

OU conference - evaluation

Following on from the previous two posts looking at the OU conference, this final one in the trilogy looks at some evaluation. I am particularly indebted to Karen Cropper, Rebecca Ferguson and Juliette Culver for doing much of the analysis for this post.

Evaluation took four main forms:

1. A questionnaire of attendees in surveymonkey
2. Statistics from cloudworks
3. Analysis of twitter users adopting the #OUConf10 hashtag
4. Analysis of the elluminate sessions

Questionnaire

There were 102 responses to the questionnaire. Below are some charts representing salient issues:

 

Most attendees were central staff, but there was a significant audience that had no connection to the OU, and a mix across other categories.

 

Attendance was split evenly across most sessions.

 

Most people would not have attended had it been face to face.

 

People usually combined it with an element of work.

 

The content, technology and organisation were all rated good to excellent, with discussion a bit below these.

 

In general participants thought that the conference allowed a community to form, and thought that the open nature of the conference worked well. Opinion was divided as to whether interaction was less than with a face to face conference.

 

There was strong agreement that they would attend another online conference.

We asked attendees to give us three words to describe the conference and the resultant word cloud is below:

 

Some issues raised by participants included:

• Separating out time to attend an online conference
• Perception an online conference is ‘techie’
• Possibly having shorter events but spread out over more time
• Allowing more open sessions and not just presentation

Some representative quotes from respondees are:

• "liked the accessibility factor - it saved a lot of money having to travel to Milton Keynes”
• "The moderators were excellent; very engaging, kept things going well, did a sterling job. The discussions in the text box during sessions was very good indeed. Generally the speakers and the topics covered were of great interest and engaging.”
• “This was an exceptional way to involve people outside the University and particuarly useful for CPD for those who can't afford to travel or whose institutions won't pay for them to go to conferences. A real credit to the OU's open learning ethos.”
• “thanks for the opportunity to take part. Not being part of the OU myself I welcomed the 'open invitation' and got loads out of it. Am already spreading the word!"
• “Most conferences should be presented like this in future. As well as Elluminate the various social network backchannels can be used for even more interaction”
• “Just wanted to thank all concerned for their hard work and the effort they put in to make this such a rewarding experience. I've developed some very important new contacts from the conference. I interacted much more than I would at a face-to-face conference.”
• “Excellent - from this distance I made better contact than I ever have at a conference before.”
• “It worked! - That's quite exciting! - To know that you can participate without too much stress is a big achievement for me - it takes some of the "fear factor" away”

Cloudworks Statistics

The following data was taken from the cloudworks site and are for the conference ‘cloudscape’ which is comprised of different elements, or ‘clouds’.

• No. clouds - 46
• No. comments - 168
• No. embeds - 59
• No. link - 80
• No. views of cloudscape - 2352
• No. registered users viewing - 392
• No. guests - 2852

Twitter analysis

The conference used the #OUConf10 hashtag, so anyone tweeting with this would have their tweet stored in an archive (using TwapperKeeper).

• No. twitterers using hashtag - 141
• No. tweets using hashtag - 766
• Av. no tweets per user - 5.43
• No. followers added over conference - 150

Elluminate analysis

The following data was gathered from analysing the elluminate sessions over the two days. Overall 287 people participated in at least one session.

The number of people who participated in each session are given below.

• Session 1 - 202
• Session 2 - 141
• Session 3 - 108
• Session 4 - 97

Conclusion

Overall the technology worked well, and most participants found the conference a useful and interesting experience. The number of participants was in line (if not a bit higher) than previous year's face to face conference, but the costs were considerably lower. In terms of benefit from the conference most participants gained as much, if not more, from this form.

A few small points it would be interesting to explore:

• I would like to perform more detailed analysis of the discussions in cloudworks, elluminate and twitter, but haven't had time yet.
• Some people commented that they felt more at liberty to ask questions in text chat than they do in a face to face conference.
• The cost and green comparison with previous face to face versions of the conference.

Project report as coffee table book

I've mentioned before I was part of the Sidecap project, which looked at OER use in Fiji, West Indies and Mauritius. The project has now finished and instead of producing the standard EU report (which, let's face it, no-one ever reads) Frank Rennie from UHI decided to create it as a book in Blurb. We all contributed sections and gathered photos from the duration of the project. My book turned up today:

 

It's a small, and undoubtedly obvious point, but the glossiness of it and the approachability of the layout made me want to read it. In almost exactly opposite to the way I'd approach a normal project report. With almost no shame I am going to mention affordances - a nice glossy book affords you picking it up and looking through it, it extends an invite to you. A project report printed on A4 paper and stuffed in a binder says the opposite, it says 'move on, nothing to see here.'

You can order the book if you want online (no profit goes to us, it just covers the cost of printing). I think all publicly funded projects should mandate this as a deliverable. On my output as collateral damage riff, this is exactly the sort of thing that means a project report destined to be tucked away in the database equivalent of the final scene of Raiders of the Last Ark, becomes a findable, accessible resource.

Or was everyone doing this anyway and I've just caught up?

Do you have to do social media to get social media?

Christian, aka Documentally has a post entitled "Understand Social Technology Through Participation", in which he says:

"How much can you understand from just watching and not participating?
Last night I met a Professor who although used Facebook and other platforms seemed proud to state he did not use twitter.
His job was to study social Technology and although obviously a very smart chap It made me wonder.. Is it possible to comprehend something as complex as social Technology by not participating in a platform like twitter. Can you glean just as much insight from only using sites like Facebook?..
All of my insights into Social Technology have come from my uses of it. "

I agree with him, it's been a bit of a bugbear of mine, the way for example, when Facebook was a'happening it was suddenly inundated with education, psychology, media, business and social science researchers who all wanted to have a look at this phenomenon. They did their research, wrote it up in traditional journal papers or media outlets and then disappeared. And I kept thinking 'they don't get it.'

But first I ought to put the contrary case:

It is perfectly possible to research any subject or group of individuals without using an active participation method. For example I can research the behaviour of vets, football hooligans or stockmarket traders without becoming a vet/hooligan/trader. Indeed not only is it possible, but it is often desirable to have the objectivity that research requires.

It is also the case that not everyone has to use every technology to 'qualify'. I may not use Second Life but that doesn't mean I can't comment on social media in general. And we don't want to have some kind of qualification to start with where people starting wearing a McDonalds type star badge to demonstrate their social-media expertise.

But having said all of that I think that academics and journalists who don't use the new media should be treated with caution. Here's why:

1. It's a fast changing world - the way technologies are used changes, so the survey you did last year won't apply this year
2. The use and benefits are very subtle, you have to spend some time to understand them and you only really do that by participating
3. There is an interconnection between many different tools, so you cannot cherry pick one in isolation without understanding how it might exist within an ecosystem of tools for an individual
4. It is only through participation that you begin to see the types of interesting things to ask. By using the tools you will formulate more meaningful research questions
5. If you simply research it then you see it as a distinct entity or community eg 'it's a teen thing'
6. If you want people to engage with your research then you have to earn some points in the reciprocity economy

So, in general I already dismiss any journalist article that begins "I've never used [technology X] but I know this...". But I am also suspicious of any academic article on social media where the author doesn't have a blog, or twitter id.

The REF - a user's guide

This is the first of two posts looking at the Research Excellence Framework (REF). Apologies to non-UK readers, this is a bit parochial. I needed to look through the REF in relation to the digital scholarship work I am doing. In the next post I will comment on it from a digital scholarship perspective, but having read it all, I thought I'd give an overview of the key points in this post, just to provide a quick review for those who can't be bothered to read the whole thing (ie, any sane person). So in this post I'm not commenting on the REF itself, either in detail, or as an exercise as a whole. I'll do that in the next post.

I read the REF consultation over at WriteToReply, and have left a number of comments there. I found the breakdown and formatting there a good way to read it, so if you do want to go through it, I'd recommend that route. So here are what I have determined to be the main elements of the proposal (although don't sue me if you submit according to this and I've missed something out).

Change to REF from RAE

The REF does not appear radically different from the RAE. It is an adjustment in process rather than a complete overhaul. The aim seems to be to simplify the RAE. This is made clear from the start:

"The underlying policy of allocating research funding selectively on the basis of quality remained unchanged; the intention was that the mechanisms should become simpler and less burdensome."

One substantial change is more of an emphasis on 'impact' - be it social or economic. This is probably a nod towards justifying the money spent on research and its relevance to the UK population:

"the Secretary of State emphasised that the REF should take better account of the impact research makes on the economy and society, and gave further guidance on particular activities that the REF should encourage:

'The REF should continue to incentivise research excellence, but also reflect the quality of researchers’ contribution to public policy making and to public engagement, and not create disincentives to researchers moving between academia and the private sector.’ "

Aims

So what is the overall aim of the REF? They sum it up as:

"Support and encourage innovative and curiosity-driven research, including new approaches, new fields and interdisciplinary work."

We should bear this aim in mind, as I found myself losing sight of it amongst the discussion of the process.

What is research?

Their definition of research is:

"a process of investigation leading to new insights effectively shared"

Assessing researchers

They assess units of researchers, not individuals. These units need to demonstrate:

• "a portfolio of high-quality, original and rigorous research, including work which is world-leading in moving the discipline forward, innovative work pursuing new lines of enquiry, and activity effectively building on this to achieve impact beyond the discipline, benefiting the economy or society"
• "Effective sharing of its research findings with a range of audiences"
• "a range of activity leading to benefits to the economy and society, "
• "A high-quality, forward-looking research environment"
• "Significant contributions to the sustainability and vitality of the research base"

There are three main elements they will assess (not equally weighted, see later):

• Output quality
• Impact
• Environment

Greatest weight in the assessment will be given to output quality - they say this three times so I think they are making a point. Their current thinking is a weighting of Output = 60%, Impact = 25%, Environment = 15%.

As with the RAE assessment will be performed by expert panels, although there will also be the use of some citation metrics for the sciences in the REF.

Outputs

The institution selects the staff it wishes to be submitted for the REF in each of the units of assessment. There is no pressure to include a minimum or maximum number of staff:

"It remains our view that the proportion of staff selected should not be a significant factor in assessing quality in the REF."

They propose 3-4 outputs per individual. It looks likely that they will come down to 3 to reduce burden on the panels.

There is some indication that it is not just the traditional outputs which will be considered (although you might want to see my next post):

"All types of outputs from research that meets the Frascati principles (involving original investigation leading to new insights) will be eligible for submission. This includes ‘grey literature’ and outputs that are not in conventional published form, such as confidential reports to government or business, software, designs, performances and artefacts"

They will assess outputs "against criteria of ‘rigour, originality and significance’." Assessing significance may be difficult and they propose "to assist in assessing user significance (beyond the academic sphere), institutions will be invited to include a short statement with any output for which they believe that such significance may convincingly be asserted."

They will use a similar 1-4 star rating as the RAE. They then provide some 'definitions' of excellence, which are self-referential:

• Four star      Exceptional: Quality that is world-leading and meets the highest standards of excellence in terms of originality, significance and rigour
• Three star     Excellent: Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence
• etc.

Metrics

A pilot study was conducted with citation metrics. They conclude that it is not robust enough to replace the expert panel, but can form a useful part of the review, particularly in the sciences.

The citation metrics need to relate to the selected outputs, ie not to an individual or a project overall.

The citation data will be limited to only 3-4 pre-selected databases:

"In the pilot exercise we used two databases, the Web of Science and Scopus. For the REF, we will procure one or more databases through a rigorous procurement process"

There is a strong hint that these citation metrics should act as the filter for who is submitted:

"Given that a number of expert panels will make use of citation data to inform their judgements, we anticipate that institutions may wish to make use of such data to inform their selection of outputs"

In order to reduce the load on panels they suggest some outputs may be 'double-weighted'.

Impact

This is now an explicit element, a change from the RAE. They stress that it needs to impact based on the research, not impact arising from other activity. They propose a wide definition of impact, "including economic, social, public policy, cultural and quality of life." They explicitly state that impact does not cover influence on academic knowledge - that is encompassed in the outputs element. This is much more about the broader, social impact of research. It can relate to teaching where "it can be shown that high-quality research has informed practice, not just course content".

Impact will not be quantified, instead "expert panels will review narrative evidence supported by appropriate indicators, and produce graded impact sub-profiles for each submission".

Each unit of assessment will need to provide evidence of impact comprising case studies and an overarching impact statement. These both need to use a number of indicators:

• "of research income generated from key categories of research users
• of the amount and extent of collaboration with the full range of research users
• that may be particular to specific UOAs, selected from a common ‘menu’."

Research environment

This doesn't really affect individuals, as it will be a narrative produced by the unit of assessment indicating how their research environment covers areas such as resourcing, management and engagement.

Panels and Units of Assessment

The workload on panels and consistency between them is a major concern in the REF document. They propose having fewer units of assessment and with "fewer fluid boundaries between them than in previous assessment exercises". Each panel will consist of about 20 members with 15-20 associate members.

Interdisciplinary research gets an explicit mention: "we aim to ensure that whichever panel interdisciplinary research is submitted to, there will be effective mechanisms for ensuring it is reviewed fairly by people with appropriate expertise"

Timetable

These are the suggested dates:

• Sep – Dec 2009     Consultation exercise. Initiate impact pilot exercise
• Spring 2010     Announce high level consultation outcomes. Invite nominations for panels. Start developing REF data collection system
• Autumn 2010     Conclude impact pilot exercise. Publish guidance on submissions
• Late 2010     Appoint panels
• 2011     Panels consult on and publish criteria. Complete REF data collection system
• 2012     HEIs make submissions
• 2013     Panels assess submissions
• Dec 2013     Publish outcomes
• Feb 2014     Determine funding outcomes

Costs

There was a review after the last RAE. The total (estimated) cost of the 2008 RAE in England was £56.7 million, which compares with the £1.5 billion of QR research funding. They estimate costs for the REF will be similar.

Science 2.0 workshop

I spent a couple of days in Nice at the ECTEL Science 2.0 workshop organised by Peter Scott and Erik Duval. I've live-blogged it over in Cloudworks, and below is a video I shot over the course of it, just to give a flavour really:

Erik has blogged some thoughts about it, I'll add a few here:

i) We are only beginning to appreciate what this data might tell us. There were a few demos about visualisations and research on twitter use at conferences (good paper from Martin Ebner and Wolfgang Reinhardt). I think we need to do more of this sort of work, because it will both push us to get more data and also we will explore different levels of interpretation. So there was some work on representing collaboration from different countries, and also on creating networks of researchers through examining citations.

ii) Mendeley is based on the concept of a 'LastFM for research articles'. It looks interesting, but it is more interesting for what it potentially highlights - people will upload articles in exchange for good data, that a good system such as this could disintermediate publishers, and peer review can be post review also.

iii) I wonder if there's not a dilemma with science in particular and some of the web 2.0 stuff: scientists by their (or I should say, our) very nature seek to predict the future. That's what science does - if we have these variables then these outcomes will ensue (or these outcomes are a result of these input variables). But the benefits of many 'web 2.0' ways of working are wrapped up in unpredictability. I don't know which blog posts will be popular, I can share ideas on twitter but I can't predict who will take them up, I can release research data but I won't know what the uses for it will be. Might it be the case then that scientists in particular want predictable benefits and outcomes from engaging in this type of activity?

What would ALT-REF look like?

So in the UK the Research Assessment Exercise is being replaced by the REF (Research Exercise Framework - see it's got a different name?). This will evaluate the research standing of individuals and departments and allocate money accordingly. Here's what they say:

"The REF will consist of a single unified framework for the funding and assessment of research across all subjects. It will make greater use of quantitative indicators in the assessment of research quality than the RAE, while taking account of key differences between the different disciplines."

In a twitter exchange with Joss Winn and Brian Kelly this morning I talked about creating an alternative to the REF:

I have in mind something that takes in the sort of distributed identity we have online, so measures activity in blogging, delicious, slideshare, YouTube, twitter, etc. It would need to measure not just activity but influence, impact, etc in some data driven manner.

A key principle must be that it's easy to do and doesn't involve expert panels, peer review etc. What I would be interested then is seeing how an ALT-REF would compare with REF? Would there be any overlap in the top researchers? Would the fields be the same? Would online reputation have any correlation to research funding?

I don't have the right knowledge for this though - it needs data and metrics experts, but it'd be an interesting experiment wouldn't it?