What works in 'What Works'?

What works in 'What Works'?

Director of Impact Mary Hartshorne discusses the latest survey results from users of The Communication Trust's What Works platform

What Works has been up, running and growing for nearly eight years. Many of you are regular users, but if this is the first time you’ve heard about What Works … it’s an online searchable database of evidenced interventions for children and young people’s speech, language and communication. In essence, it’s a practical tool to help early years practitioners, speech and language therapists, school staff – really anyone who works with children or young people – to make decisions about how to plan the best, well-evidenced, support for them.

There’s lots more information about what it does and doesn’t do here – or you can listen to our What Works podcast here.

Every couple of years or so, a survey of What Works users helps the moderating group and the team at I CAN with planning out how the site can be developed, improved, changed so that it is as useful as possible. For example, the infographics and animation explaining effect sizes came about after one survey showed users wanted clear, accessible information about the strength of evidence. Broadly, the survey looks at who uses the database, for what purpose, how easy it is to use and the impact of their visit to the site.

Cristina McKean from Newcastle University led the 2019 survey, which threw up some interesting findings: some expected, others less so but all useful prompting discussion and an action plan!

An impressive 857 people responded to the survey. Overall, respondents were positive: they find What Works useful and it influences their practice.

A first look at the results told us some fairly predictable things:

The majority of respondents were speech and language therapists (63%). Perhaps this is no surprise given the nature of the interventions listed in the database. However, it’s good to see a small but significant number of education staff responding, particularly specialist teachers. 94% of people completing the survey were based in the UK or Ireland, but there was a small number from further afield, many of whom were academics rather than practitioners.

The majority of users reported using What Works two or three times a year, mainly to check evidence or search for different interventions – and many (84%) felt it supported their professional development and influenced their practice (76%).

So, aside from those headlines, what key themes prompted discussion?

There’s a generational difference in usage.

Over half (53%) of respondents had been five year or less in their role, and only 18% had been in role for more than 15 years. People most recently in role tended to use What Works more frequently, and had embedded its use in their CPD. The opposite was true of people longer in role.

good job Higher Education Institutions are doing a good job signposting What Works and this may, in part explain the generational difference. The second highest group of respondents (11%) were students, and maybe this early experience embeds a culture for their later practice.

A need to understand and better support educational usage.

Although the growing number of users from schools and early years settings is encouraging, What Works is not a first point of call for them. There are questions around the accessibility of the platform for these users, and whether it’s possible to improve this. In the past, we’ve considered adding a ‘What Works in education’ section, or specifically targeting submissions based on studies carried out in schools and nurseries. But the influence of other evidence databases such as the Education Endowment Foundation (EEF) toolkit for teaching and learning may be more relevant for teachers and school leaders.

How can What Works be used to influence Commissioners?

Very few (4.2%) respondents were in a role where there was responsibility for commissioning speech and language services: speech and language therapy managers or school leaders. The context for planning and funding services is ever-changing, but alongside this is the need to ensure that service design is based on evidence of effectiveness. What Works is not directly aimed at commissioners, but could be a useful tool to inform decisions. Particularly if the evidence was communicated in a different way.

Synthesis of evidence

Some of the free text responses to questions suggested that there may be ways that evidence could be interpreted or summarised for users so that it could be more easily applied in practice.

Communicating What Works

There was a view that more people should know about and use What Works, and communications should be increased so that its profile was raised – particularly to non-speech and language therapy audiences.

The nature of Evidence-Based Practice

This probably generated most discussion!

What Works is clearly being used to inform practice, but further analysis looked at what this actually meant. When asked how they used the database, the most popular responses were to ‘check evidence of an intervention we are using’ and ‘to look for new interventions’, while fewer than half of respondents reported using the database for practitioner-based research. Nearly half did not respond when asked whether they used it to shape service design or delivery, nor to whether they used it to share research across their team – these null responses suggest uncertainty around these questions.

Looking also at the free text responses, some respondents talked about using What Works to inform headteachers that the interventions being recommended for use in schools are evidence-based. There was a sense that What Works was used to select or identify interventions, rather than as contributing to the collaborative decision-making process which characterises evidence-based practice (EBP) – or to wider evidence-based service design.

This may have been a result of the questionnaire and limited choice of responses, but when reflecting on the initial findings we felt we needed to know more. We wanted to get a sense of what EBP looks like for practitioners, especially for more experienced practitioners, and what role What Works plays in this.

Next steps

The survey prompted discussion, and the first item on the action plan arising from it was the need for some further investigation by Cristina and the team.

Yes, we wanted to find out more about the nature of EBP on the ground and how What Works can support the pipeline from research to practice. But we were also interested in questioning some of the less frequent users: the education staff, more experienced clinicians and commissioners, for example, to find out what would make What Works more accessible and useful to them. Cristina is currently doing this via a series of semi-structured interviews. We’ll keep you informed!