The Ed Techie

An interview with the future

I am one of the 'presenters' at the JISC 09 online conference. I don't actually present, but instead have been asked to create a short video for the session looking at Do educational institutions have a future? Graham Attwell and Rob Howe are the other presenters. We had a chat about the session and I mentioned that I had considered doing an interview with a future version of myself, as a means of exploring the issues. They liked the idea so we have decided all to adopt it, with myself taking the academic perspective, Rob the learner one and Graham the institutional view.

Below is my video, although it's meant as a bit of fun I hope there are some issues in it that we can explore in the session. Please sign up for the conference if you haven't already, it should be fun (it costs £50 I think).

BTW - I think everyone should do an interview with their future selves, so if you do one let me know.

Remote conference participation - flash debate

<Image Waiting for it all to begin by Unhindered by Talent http://www.flickr.com/photos/nicmcphee/2380602382/>

One of the things I am interested in is the subtle ways in which new technology begins to alter standard practice over time, and without these changes being planned. In the academic world I think the conference is one such area. The academic conference can be seen as one of the core practices in higher education. It achieves many vital functions in academic practice, including:

• Knowledge sharing - you get to present and listen to other talks
• Validation - by sharing research and ideas within a subject community you get validation
• Networking - you establish a network of peers
• Recognition - publishing conference papers is often a first step for researchers to publishing papers and are recognised outputs
• Socialising - slightly different from networking, there is a social element to conferences which make them enjoyable

Over the past few years remote participation has become more commonplace. By remote participation I don't mean solely online conferences, but rather the sort of vicarious, casual participation many of us undertake. This type of participation is often unofficial, and uses low key, free technology. It is often a hybrid of the following examples:

• Twitter hashtags
• Live streaming (whether official or via an individual)
• Blogging
• Live-blogging
• Video/audio updates
• Flickr streams
• Slideshare presentations
• Cloudworks/Friendfeed aggregations

There are a few things that interest me about this. The first is how does it change the nature of the conference to have this broader participation? Secondly, how can conference organisers and presenters best take advantage of it and incorporate it into the conference? Thirdly, what is the experience like for the remote participant compared with the 'real thing'? For this last question, I have created a quick 5 question survey to get a feeling for how remote participation compares with real attendance.

I think there is a lot to discuss here, so I have set up a space in cloudworks for a flash debate. I'd be interested in your thoughts on any of the three questions above, or other opinions. Of course you can add comments here too - I'll post them over in Cloudworks also.

The REF - a digital scholarship perspective

Having given an overview of the REF in my last post, in this one I will provide a commentary on it from a digital scholarship perspective.

As readers will probably know, I'm not a fan of such exercises in general - the inevitable experimenter's effect phenomena comes into play, particularly when there is considerable money involved, with the result that we don't encourage new, exploratory types of behaviour. But let's put aside these more general reservations, and look at the REF proposal itself. My particular take on this is to what extent does it reward, recognise, encourage activities which we might broadly term digital scholarship?

From this perspective the REF starts well with the aim to "support and encourage innovative and curiosity-driven research, including new approaches, new fields and interdisciplinary work." So there is a very explicit aim to 'support and encourage' new approaches, new fields, interdisciplinary work. I was rubbing my digital scholarship hands at this point.

Similarly their definition of research is sufficiently broad to encompass digital scholarship: "a process of investigation leading to new insights effectively shared". That phrase 'effectively shared' surely harks at blogs, social networks, data visualisation, wikis, etc. 

So feeling encouraged, I read on. Sadly the rest of the document does not live up to these opening intentions, indeed it seems to actively undermine them in places.

Making digital activity explicit

Given that part of the aim of the REF is to place UK research at the fore and to demonstrate both its excellence and relevance to society, I would suggest that encouraging new forms of scholarly activity would be an explicit aim. Particularly as this would tie in with the aim set out in the Digital Britain report about establishing the UK as an economic hub in the digital age. Yet anything resembling digital scholarship is noticeable by its absence in the document.

It is noticeable for instance that the two drivers for change to the RAE were to make it less cumbersome and to have a unified approach. These seem rather unambitious – I would have liked to see something like ‘to recognise the changing nature of research and dissemination in a digital age and to both reward and encourage this’.

For example, it does make a gesture towards moving away from the traditional article as the sole output when it states

"All types of outputs from research that meets the Frascati principles (involving original investigation leading to new insights) will be eligible for submission. This includes ‘grey literature’ and outputs that are not in conventional published form, such as confidential reports to government or business, software, designs, performances and artefacts"

But this would have been a prime opportunity to explicitly recognise new forms of output - blogs, video, podcasts, etc.

The wrong metrics

Having looked at the possible use of metrics to inform the panels they conclude that they are not robust enough, but that some citation metrics will be used. These however are limited to 3-4 pre-approved databases. This is not even a forward step for digital scholarship - by limiting it to these databases and then suggesting that staff are selected on this basis, they are effectively limiting outputs to journal articles.

Although they do argue that "This approach to the assessment of outputs retains scope for the assessment of grey literature and work published in non-standard forms (for which citation data are unlikely to be available)" my suspicion is that the presence of metrics in the sciences will effectively become a selection filter.

There is no suggestion here of embracing the broader world of data and metrics which new forms of activity would be well represented by.

Blurring boundaries

Despite it being mentioned in the high level aim of the REF, interdisciplinary research will not be well served. They are reducing the number of units of assessment, which will mean more researchers being forced into inappropriate categories. In addition they state that fluidity between the units should be reduced:

"For the REF we propose to have substantially fewer UOAs with fewer fluid boundaries between them than in previous assessment exercises"

The REF must be the only body that feels the way to embrace the digital age is to have less fluid boundaries.

As with the use of citation metrics restricting outputs to journal articles, they anticipate the criticism of interdisciplinary work:

"we aim to ensure that whichever panel interdisciplinary research is submitted to, there will be effective mechanisms for ensuring it is reviewed fairly by people with appropriate expertise."

But as with the outputs, the actual proposal seems to undermine this, and merely stating it as an objective will not ensure that it happens.

Impact

I think the inclusion of impact could be an area that digital scholars can excel at - we have data about blog readership, numbers of views and embeds on videos, podcast downloads, etc. In Appendix D they list some of the evidence of impact that will be admissible. These are disappointingly conservative however - eg Staff movement between academia and industry, Research contracts and income from industry, Research income from government organisations, etc.

Openness

Openness doesn't get a mention in the REF, and you would have liked to see this as a key theme. Indeed there is an element of a closed world of researchers that pervades much of the document, despite the talk of impact and relevance. There is no mention for instance of recognising data as an output that should be released. The use of citations is limited to pre-defined ones that are purchased. There is no encouragement to publish openly, or the use of open APIs to explore different metrics.

Conclusion

Overall I found it a highly frustrating, deeply conservative and mildly schizophrenic document.

I wonder if part of this schizophrenia arises from not wanting to be seen to be directing research too much, or a denial that such exercises do exactly that. I think they should acknowledge the experimenter effect (it is ironic that a research exercise ignores it) and embrace it. If we are to have exercises such as the REF, which effectively control the direction of UK research, then would it not be better for them to have some worthy goals and vision that would see research be competitive, highly regarded and relevant?

So an opportunity to have a vision that encourages new forms of scholarship, to embrace the potential of digital technologies, to make openness a central theme in UK research is missed. What we have is a curious mix of strategy document and justification of the difficulties of the process. Having decided to measure something they now come up against all the difficulties that entails, such as imposing categories, regulating workload, etc. The process becomes the artefact and thus instead of promoting interdisciplinary work, fluid boundaries, new forms of output which a strategic proposal would surely do,  all of these are curtailed because that makes the process more manageable.

As a document it seems to consistently undermine its own ambition, and whenever it approaches new forms of scholarship it veers away at the last moment and reverts to the comfort of what it knows best.

The REF - a user's guide

This is the first of two posts looking at the Research Excellence Framework (REF). Apologies to non-UK readers, this is a bit parochial. I needed to look through the REF in relation to the digital scholarship work I am doing. In the next post I will comment on it from a digital scholarship perspective, but having read it all, I thought I'd give an overview of the key points in this post, just to provide a quick review for those who can't be bothered to read the whole thing (ie, any sane person). So in this post I'm not commenting on the REF itself, either in detail, or as an exercise as a whole. I'll do that in the next post.

I read the REF consultation over at WriteToReply, and have left a number of comments there. I found the breakdown and formatting there a good way to read it, so if you do want to go through it, I'd recommend that route. So here are what I have determined to be the main elements of the proposal (although don't sue me if you submit according to this and I've missed something out).

Change to REF from RAE

The REF does not appear radically different from the RAE. It is an adjustment in process rather than a complete overhaul. The aim seems to be to simplify the RAE. This is made clear from the start:

"The underlying policy of allocating research funding selectively on the basis of quality remained unchanged; the intention was that the mechanisms should become simpler and less burdensome."

One substantial change is more of an emphasis on 'impact' - be it social or economic. This is probably a nod towards justifying the money spent on research and its relevance to the UK population:

"the Secretary of State emphasised that the REF should take better account of the impact research makes on the economy and society, and gave further guidance on particular activities that the REF should encourage:

'The REF should continue to incentivise research excellence, but also reflect the quality of researchers’ contribution to public policy making and to public engagement, and not create disincentives to researchers moving between academia and the private sector.’ "

Aims

So what is the overall aim of the REF? They sum it up as:

"Support and encourage innovative and curiosity-driven research, including new approaches, new fields and interdisciplinary work."

We should bear this aim in mind, as I found myself losing sight of it amongst the discussion of the process.

What is research?

Their definition of research is:

"a process of investigation leading to new insights effectively shared"

Assessing researchers

They assess units of researchers, not individuals. These units need to demonstrate:

• "a portfolio of high-quality, original and rigorous research, including work which is world-leading in moving the discipline forward, innovative work pursuing new lines of enquiry, and activity effectively building on this to achieve impact beyond the discipline, benefiting the economy or society"
• "Effective sharing of its research findings with a range of audiences"
• "a range of activity leading to benefits to the economy and society, "
• "A high-quality, forward-looking research environment"
• "Significant contributions to the sustainability and vitality of the research base"

There are three main elements they will assess (not equally weighted, see later):

• Output quality
• Impact
• Environment

Greatest weight in the assessment will be given to output quality - they say this three times so I think they are making a point. Their current thinking is a weighting of Output = 60%, Impact = 25%, Environment = 15%.

As with the RAE assessment will be performed by expert panels, although there will also be the use of some citation metrics for the sciences in the REF.

Outputs

The institution selects the staff it wishes to be submitted for the REF in each of the units of assessment. There is no pressure to include a minimum or maximum number of staff:

"It remains our view that the proportion of staff selected should not be a significant factor in assessing quality in the REF."

They propose 3-4 outputs per individual. It looks likely that they will come down to 3 to reduce burden on the panels.

There is some indication that it is not just the traditional outputs which will be considered (although you might want to see my next post):

"All types of outputs from research that meets the Frascati principles (involving original investigation leading to new insights) will be eligible for submission. This includes ‘grey literature’ and outputs that are not in conventional published form, such as confidential reports to government or business, software, designs, performances and artefacts"

They will assess outputs "against criteria of ‘rigour, originality and significance’." Assessing significance may be difficult and they propose "to assist in assessing user significance (beyond the academic sphere), institutions will be invited to include a short statement with any output for which they believe that such significance may convincingly be asserted."

They will use a similar 1-4 star rating as the RAE. They then provide some 'definitions' of excellence, which are self-referential:

• Four star      Exceptional: Quality that is world-leading and meets the highest standards of excellence in terms of originality, significance and rigour
• Three star     Excellent: Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence
• etc.

Metrics

A pilot study was conducted with citation metrics. They conclude that it is not robust enough to replace the expert panel, but can form a useful part of the review, particularly in the sciences.

The citation metrics need to relate to the selected outputs, ie not to an individual or a project overall.

The citation data will be limited to only 3-4 pre-selected databases:

"In the pilot exercise we used two databases, the Web of Science and Scopus. For the REF, we will procure one or more databases through a rigorous procurement process"

There is a strong hint that these citation metrics should act as the filter for who is submitted:

"Given that a number of expert panels will make use of citation data to inform their judgements, we anticipate that institutions may wish to make use of such data to inform their selection of outputs"

In order to reduce the load on panels they suggest some outputs may be 'double-weighted'.

Impact

This is now an explicit element, a change from the RAE. They stress that it needs to impact based on the research, not impact arising from other activity. They propose a wide definition of impact, "including economic, social, public policy, cultural and quality of life." They explicitly state that impact does not cover influence on academic knowledge - that is encompassed in the outputs element. This is much more about the broader, social impact of research. It can relate to teaching where "it can be shown that high-quality research has informed practice, not just course content".

Impact will not be quantified, instead "expert panels will review narrative evidence supported by appropriate indicators, and produce graded impact sub-profiles for each submission".

Each unit of assessment will need to provide evidence of impact comprising case studies and an overarching impact statement. These both need to use a number of indicators:

• "of research income generated from key categories of research users
• of the amount and extent of collaboration with the full range of research users
• that may be particular to specific UOAs, selected from a common ‘menu’."

Research environment

This doesn't really affect individuals, as it will be a narrative produced by the unit of assessment indicating how their research environment covers areas such as resourcing, management and engagement.

Panels and Units of Assessment

The workload on panels and consistency between them is a major concern in the REF document. They propose having fewer units of assessment and with "fewer fluid boundaries between them than in previous assessment exercises". Each panel will consist of about 20 members with 15-20 associate members.

Interdisciplinary research gets an explicit mention: "we aim to ensure that whichever panel interdisciplinary research is submitted to, there will be effective mechanisms for ensuring it is reviewed fairly by people with appropriate expertise"

Timetable

These are the suggested dates:

• Sep – Dec 2009     Consultation exercise. Initiate impact pilot exercise
• Spring 2010     Announce high level consultation outcomes. Invite nominations for panels. Start developing REF data collection system
• Autumn 2010     Conclude impact pilot exercise. Publish guidance on submissions
• Late 2010     Appoint panels
• 2011     Panels consult on and publish criteria. Complete REF data collection system
• 2012     HEIs make submissions
• 2013     Panels assess submissions
• Dec 2013     Publish outcomes
• Feb 2014     Determine funding outcomes

Costs

There was a review after the last RAE. The total (estimated) cost of the 2008 RAE in England was £56.7 million, which compares with the £1.5 billion of QR research funding. They estimate costs for the REF will be similar.

Becoming a multi-blogger

One of the (vague, ill-defined, unstructured) tasks I set myself this year was to see if my blogging/social networking activity could be bent some way to a more focused institutional benefit. I can make lots of (unsupported, wildly inaccurate, ego-driven) claims about the benefit of this blog to the Open University, for example it acts as a form of staff development, it forms part of an institutional dialogue, it raises the university's profile, it demonstrates the university's engagement with new media, etc.

But I also feel that my blogging activity has been well supported without any specific aim, and it has now reached a reasonably robust state where can I make the above claims without everyone in the room laughing (only some of them). So, I wanted to investigate whether this practice could become more central and foregrounded in what I do and in its institutional benefit.

One result of this has been the initiation of the digital scholarship work at the OU. We have created a site as a part of this activity (knocked together very quickly at our 2-day hackfest). After another iteration of this, I may propose to senior management that it is open to all (in the spirit of things), not just OU staff, but it may be that it only really works with the specific OU focus. One element of this is that it has a central blog for news relevant to digital scholarship. This is rather like my tumblr blog but with a bit more comment. I am also taking over some blogging duties over at the SocialLearn blog.

It is going to be interesting to see whether I can blog effectively for these different audiences. I started this blog as a separate entity from the OU, and although it is obviously related, I have never really had to worry about a very specific intention behind the blog. Hence I can spout half-formed ideas about music, football, elearning and general bobbins. This isn't going to cut it in my new (on-brand, suit-wearing, performance criteria achieving) role. It will also be interesting to see if I can spread my already meagre intellectual jam thinly across three blogs, and whether it impacts upon this one.

I guess what this really comes back to is identity again and the personal-professional tension. I believe a bit of the personal is essential to give a blog meaning. I created this intellectually challenging graph to illustrate my point:

(Graph produced at crappygraphs.com)

So this mini-personal experiment will examine whether I can blog to a different audience, and still retain enough personal element to make it meaningful for me and the reader.

Science 2.0 workshop

I spent a couple of days in Nice at the ECTEL Science 2.0 workshop organised by Peter Scott and Erik Duval. I've live-blogged it over in Cloudworks, and below is a video I shot over the course of it, just to give a flavour really:

Erik has blogged some thoughts about it, I'll add a few here:

i) We are only beginning to appreciate what this data might tell us. There were a few demos about visualisations and research on twitter use at conferences (good paper from Martin Ebner and Wolfgang Reinhardt). I think we need to do more of this sort of work, because it will both push us to get more data and also we will explore different levels of interpretation. So there was some work on representing collaboration from different countries, and also on creating networks of researchers through examining citations.

ii) Mendeley is based on the concept of a 'LastFM for research articles'. It looks interesting, but it is more interesting for what it potentially highlights - people will upload articles in exchange for good data, that a good system such as this could disintermediate publishers, and peer review can be post review also.

iii) I wonder if there's not a dilemma with science in particular and some of the web 2.0 stuff: scientists by their (or I should say, our) very nature seek to predict the future. That's what science does - if we have these variables then these outcomes will ensue (or these outcomes are a result of these input variables). But the benefits of many 'web 2.0' ways of working are wrapped up in unpredictability. I don't know which blog posts will be popular, I can share ideas on twitter but I can't predict who will take them up, I can release research data but I won't know what the uses for it will be. Might it be the case then that scientists in particular want predictable benefits and outcomes from engaging in this type of activity?

10 reasons to use open content

I gave this presentation to set the scene at a workshop recently at The Open University. The aim of the day was to promote the use of third party resources, so during the day the teams had to devise a learning activity based around third party content.

I set out ten reasons (with an extra one for luck), in this slidecast below:

10 reasons to use open content (in teaching)

View more presentations from mweller.