Who wants the TEF?

I'll say this for the White Paper on Higher Education "Success as a Knowledge Economy": it's not as bad as the Green Paper that preceded it. The Green Paper had me abandoning my Christmas shopping for furious tirades against the errors and illogicality that were scattered among the exhausted clichés and management speak (see here, here, here, here and here). So appalled was I at the shoddy standards evident in the Green Paper that I actually went through all the sources quoted in the first section of the White Paper to contact the authors to ask if they were happy with how their work had been reported. I'm pleased to say that out of 12 responses I got, ten were entirely satisfied, and one had just a minor quibble. But what about the twelfth, you ask. What indeed?

When justifying the need for a Teaching Excellence Framework (TEF) last November, Jo Johnson used some extremely dodgy statistical analysis of the National Student Survey to support his case that teaching in some quarters was 'lamentable'. I was pleased to see that this reference was expunged from the White Paper. But that left a motheaten hole in the fabric of the argument: if students aren't dissatisfied, then do we really need a TEF?  One could imagine the civil servants rushing around desperate to find a suitably negative statistic. And so they did, citing the 2015 HEPI-HEA Student Academic Experience Survey as showing that "Many students are dissatisfied with the provision they receive, with over 60% of students feeling that all or some elements of their course are worse than expected and a third of these attributing this to concerns with teaching quality." (p 8, para 5).  The same report is subsequently cited as showing that: ".. applicants are currently poorly-informed about the content and teaching structure of courses, as well as the job prospects they can expect. This can lead to regret: the recent Higher Education Academy (HEA)–Higher Education Policy Institute (HEPI) Student Academic Experience Survey found that over one third of undergraduates in England believe their course represents very poor or poor value for money." The trouble is, both of these quotes again use spin and dodgy statistics.

Let's take the 60% dissatisfaction statistic first. The executive summary of the report stated; "Most students are satisfied with their course, with 87% saying that they are very or fairly satisfied, and only 12% feeling that their course is worse than they expected. However, for those students who feel that their course is worse than expected, or worse in some ways and better than others, the number one reason is not the number of contact hours, the size of classes or any problems with feedback but the lack of effort they themselves put in." So how do we get to 60% dissatisfied? This number is arrived at from the finding that 12% said that their experience had been worse than expected, 49% said that it had been better in some ways and worse in others. So it is literally true that there is dissatisfaction with 'some or all elements', but the presentation of the data is clearly biased to accentuate the negative. One is reminded of Hugh in 'The Thick of It' saying "I did not knowingly not tell the truth".

But it gets worse: As pointed out on the Wonkhe blog, among 'key facts' in a briefing note accompanying the White Paper, the claim was reworded to say “over 60% of students said they feel their course is worse than expected”. The author of the blogpost referred to this as substantial misrepresentation of the survey. This is serious because it appears that in order to make a political point, the government is spreading falsehoods that could cause reputational damage to Universities.

Moving on to perceptions of 'value for money', there are two reasons for giving this low ratings  - you are paying a reasonable amount for something of poor quality, or you are paying an unreasonable amount for something of good quality. Alex Buckley, one of the authors of the report replied to my query to say that while the numeric data were presented accurately, crucial context was omitted. This made it crystal clear it was the money side of the equation that concerned students. He wrote:

"Figure 11 on page 17 of the 2015 HEPI-HEA survey report shows that students from England (paying £9k) and students from Scotland studying in Scotland (paying no fees) have very different perceptions of value for money. And Figure 12 shows that the perceptions of value for money of students from England plummeted at the time of the increase in fees. Half of 2nd year students from England in 2013 thought they were getting good or very good value for money. In 2014, when 2nd years were paying £9k, that figure was a third. (Other global perceptions of quality - satisfaction etc. - did not change). There is something troubling about the Government citing students' perceptions of value for money as a problem for the sector, when they appear to be substantially determined by Government policy, i.e. the level of fees. The survey suggests that an easy way to improve students' perceptions of the value for money of their degree would be to reduce the level of fees - presumably not the message that the Government is trying to get across."

So do students want the TEF? All the indicators say no. Chris Havergal wrote yesterday in the Times Higher about a report by David Greatbatch and Jane Holland in which students in focus groups gave decidedly lukewarm responses to questions about the usefulness of TEF. Insofar as anyone wants information about teaching quality, they want it at the level of courses rather than institutions, but, as an ONS interim review pointed out, the data is mostly too sparse to reliably differentiate among institutions at the subject level. Meanwhile, the NUS has recommended boycotting the National Student Survey, which forms a key part of the metrics to be used by TEF.

This is all rather rum, given that the government claims its reforms will put students at the heart of higher education. It seems that they have underestimated the intelligence of students, who can see through the weasel words and recognise that the main outcome of all the reforms will be further increases in fees.

It's widely anticipated that fees will rise because of the market competition that the White Paper lauds as a positive stimulus to the sector, and it was clear in the Green Paper that one goal of the reforms was to tie the TEF to a regulatory mechanism that would allow higher fees to be set by those with good TEF scores. Perhaps less widely appreciated is that the plan is for the new Office for Students to be funded largely by subscriptions paid by Higher Education Providers. They will have to find the money somewhere, and the obvious way to raise the cash will be by raising fees. So students will be in the heart of the reforms in the sense that having already endured dramatic rises in fees and loss of the maintenance grant, they will now also be picking up the bill for a new regulatory apparatus whose main function is to satisfy a need for information that they do not want.

On the need for clarity of purpose in the REF and TEF

©CartoonStock.com

The UK’s Research Evaluation Framework (REF) has come in for a lot of criticism. It is now under review by a panel chaired by Nicholas Stern, with a call for evidence that closes later this month. At the same time, we have a Green Paper setting out plans  for a Teaching Excellence Framework (TEF). This is motivated in part by the view that the attention given to research and teaching has got out of balance. REF has provided universities with strong incentives to put resources into research, and teaching has consequently been neglected, goes the argument (though see here). So what do we need to even things up? A TEF.

The problem for both REF and TEF is that, at the end of the day, they aim for a single scale on which universities can be rank ordered so we can compare quality. But everyone agrees that the things we are measuring, research and teaching excellence, are complex and multifactorial.

There are basically two ways forward. Option A is to use some kind of proxy measure, recognising its limitations but taking the view that it is good enough for purpose. Option B involves trying to measure the complex multifactorial construct in all its richness.

There are a number of factors that influence choice of approach. Because everyone recognises that things are complex, Option A is unlikely to be acceptable to the academic community. Simple measures are often easy to game. On the other hand, the complex multifactorial measures of Option B can be debated endlessly, often involve elements of subjective judgement, are not immune to gaming, can be extremely expensive to administer, and can be hard to integrate into a single ranking.

James Wilsdon has noted with regard to the REF, before deciding which system of measurement to use, we have to have a clear idea of what we are trying to achieve.  As far as the REF goes, its purpose has changed and mutated over the years. It started out with a pretty simple goal: to find a formula to determine allocation of quality-related (QR) funding from central government to universities. However, as Wilsdon notes, it has subsequently been used for four additional purposes: to demonstrate accountability, to provide a measure of reputation, to influence research culture, and as a tool within universities for managing academics. He notes that: “If all we want from the REF is a QR allocation tool, then we can certainly do that in an algorithmic, metric-based way”(i.e. Option A). But he argues the REF needs to fulfil the other functions too, and, as was amply demonstrated in his report the Metric Tide, for those other purposes, a simple metrics-based system is inadequate.

I agree with much of what Wilsdon says, but I think we could save ourselves a lot of trouble by reverting to the original purpose of the REF, i.e. treat it purely as a mechanism for allocating funding. As I have argued previously, if that is all you want to do, then you don’t even need to bother with metrics of the kind discussed in his report. A simple measure of the number of active researchers present in a department gives a remarkably high correlation with the amount of QR funding received – and this works well for most subjects in arts and humanities as well as sciences.

But what about gaming? When I proposed this idea a couple of years ago, people said, wouldn’t universities just designate the departmental cleaner as an active researcher, or take on more research staff? I don’t see these problems as insuperable. It would be important to specify stringent criteria for research staff to meet: these would include terms of employment (casual staff would be excluded), as well as evidence of research activity. If one counted only those staff who had been employed at the institution for some minimum period, such as 3-4 years, this should prevent institutions catapulting in overseas researchers on Mickey Mouse contracts, or taking on short-term staff to give a temporary blip in researcher numbers.

A more serious objection to my proposal is that there is no explicit measure of research quality – an institution could take on a large number of weak researchers and look as good as a competitor with an equal number of excellent researchers. But would this happen? Remember, researchers would need to be on the institutional payroll for a period of 3-4 years prior to the evaluation, so the institution would need to commit to the expense of employing them. This would not be worthwhile if staff then failed to meet the criteria set for research-active staff. Academics who did not count as active researchers would end up being a net cost to the institution.

I’m not saying that it would be easy to fine-tune such a system to avoid gaming or unintended consequences, just that it could be done, and I suspect would be much less difficult than devising an entirely separate system for evaluating research quality.

My case falls apart if, like Wilsdon (and many other people who have been involved in REF) you think REF should fulfil additional purposes. Then, because no one measure is suitable for all purposes, you need something much more complicated. But I do agree with Wilsdon that, if that’s what you want, you need to be clear about it – and about the need for a diverse set of measures appropriate to different goals.

What about TEF? Well, when you dig beneath the surface, you find that the parallels between REF and TEF are purely superficial. The purpose of TEF is not to allocate funding – there is no funding to allocate. The stated purposes are as complex and multifactorial as the notion of teaching excellence itself: to help students select courses, to increase access of under-represented groups to higher education, to provide a basis for allowing universities to raise fees, and to provide criteria for ‘new entrants’ (i.e. private institutions) that wish to enter the higher education market. According to a recent BIS Select Committee report, it’s also intended to provide incentives: “to ensure that higher education institutions meet student expectations and improve on their leading international position.” Quite what it means to improve on a leading international position is not specified.

In attempting to develop a measure that will cover all these functions, those promoting the TEF have tied themselves in knots, as illustrated by this wonderfully circular statement from the same Select Committee report:

In the absence of any agreed definition or recognised measures of teaching quality, the Government is proposing to use measures, or metrics, as proxies for teaching quality. Therefore the challenge is to identify those metrics which most reliably and accurately measure teaching quality, as opposed to other factors that contribute to the results achieved by students.

This is worrying. The only positive thing one can say is that there are signs that government may be starting to recognise some of the problems. The Select Committee report cautions the need not to rush into a TEF, and notes reservations both about the measures proposed and the proposed link between TEF and fee-raising powers. The report concludes by encouraging academics to work with BIS to develop appropriate metrics for TEF – the impression is that government is aware if they get it wrong then universities may just decide not to play ball. One of the members of the Select Committee, Amanda Milling, wrote in the Times Higher that “the higher education sector has a responsibility to engage with TEF to make it work.”

But do we? I would argue that the responsibility lies with the Minister, to make a proper case for the TEF.

As the Select Committee report points out: “It is important to note the high quality of teaching generally available in our higher education system at present…..The debate around teaching excellence should therefore be viewed within the context of enhancing an already excellent system or, as the Minister for Universities and Science put it, ‘to continue to make a great sector greater still’”. These weasel words mean that if universities resist TEF, they can be accused of complacency. But where’s the evidence that TEF will ‘make a great sector greater still’? A considerable amount of time and money will be sucked up by this exercise, which has multiple confused aims and has potential to tie up a great sector in pointless bureaucracy and waffle. The whole idea is seriously misconceived and has been rushed through without adequate justification or cost-benefit analysis.

We are now being told that TEF will be introduced by degrees, with measures being developed over time, but I am not reassured. If the government wants academics on side, it needs to demonstrate more coherent arguments, with clear specification of the goals of the TEF, and evidence of validity of the measures it proposes to achieve those goals. And most of all, it needs to show us that more good than harm will result from this exercise.