How do we measure the success of prioritisation processes? What more do we need to learn in this field?

How do we measure the success of prioritisation processes? What more do we need to learn in this field?

High quality stakeholder engagement, transparency, and more!

This post is written by Paul Kellner as part of Good Questions Review, a living literature review about how social science can be useful for making decisions. It was made possible through support from Open Philanthropy.

Note: this post is continuously updated as relevant articles are added to Good Questions Review. An important part of this project is archiving substantive edits to posts. To do this we create a digital object identifier for the original post as well as subsequent versions. Please find the archived version of this post here.

The “right” research question may vary quite significantly when considered from different stakeholder perspectives, especially when time and other resources are limited. Based on this, there is growing academic study about the methods we can use to better prioritise questions to meet the needs of various knowledge users and other people impacted by research. This post explores what might constitute “success” for prioritisation processes. Additionally, it compares the markers of success to current knowledge about the current ways that prioritisation takes place, as well as considers the direction that some researchers and other stakeholder feel that methodological research should go in this field.

NOTE: Articles about rapid reviews are disproportionately represented in health sciences, and so this post will focus on those derived primarily from health. The intention is to understand the state of the art of these methods and possibly understand where the field might be going within and beyond the health sciences. Some of the methods described are from the social sciences or at least relevant to them.

What are the usual components of prioritisation?

Understanding what a typical prioritisation process involves is likely a good first step in exploring what success might look like. Fadlallah et al.'s (2020) systematic review of prioritisation processes with the purpose of developing evidence synthesis[i] topics related to health. Based on the 31 included studies, effectively all processes include 2 key steps:

  • Generating a list of topics (97% of included papers)
  • Ranking or organising topics in some other way to show their priority, e.g. pooling topics into high and low priority (97% of included papers)

Although the above steps are nearly universal, there are several specific methods or components of relevance to the topics of this post.

In the pre-prioritisation stage, the most common method of generating a list of candidate questions or topics is by gathering feedback from stakeholders (79%). This stage sometimes also features components focused on establishing a set of guiding or ethical principles for the process (31%), and on a few occasions has involved the collation of other data sources to inform the prioritisation process (17%).

During the prioritisation itself, undertaking gap analysis was the most frequently mentioned method (83%), aside from the act of prioritising the questions or topics itself. Gap analyses usually involved prioritisation participants assessing what research topics have already been considered by review and primary studies.

The ranking approaches varied across studies with some ranking full lists of topics/questions, while others placed topics/questions into categories (e.g. high and low priority questions). Of note here is that 48% of studies used criteria of some type to inform the prioritisation process. Fadlallah et al. (2020) analysed the type of criteria used and found that they “…clustered in 10 domains:

  1. problem-related considerations;
  2. practice considerations;
  3. existing systematic reviews;
  4. existing primary studies;
  5. question is amenable to review;
  6. urgency;
  7. implementation considerations;
  8. ethical and moral considerations;
  9. interest of the topic at different levels; and
  10. expected impact of applying evidence…” (Fadlallah et al., 2020, pg 75)

In the post prioritisation phase, there was less consistency across studies about the types of follow-up activities that take place or are reported. Forty-one percent of studies described dissemination or implementation of the priorities and 24% discussed a process to refine the topics into actionable review questions. On this later point, Fadlallah et al. (2020) observed the critical importance of refining broad topics into actionable questions because it supports appropriate interpretation of any research that flows from the prioritisation process. Lastly, it should be noted that only a few processes involved a revision or appeals step in the process of finalising a priority list.

From this work, the authors concluded that there is room for greater methodological guidance on how to undertake, and report findings from, prioritisation processes.

What do experts regard to be good measures of success in prioritisation?

Moving to how we measure success in prioritisation processes. An older article[ii] provides the most direct, substantial, and empirically-grounded article about the topics of this post. Sibbald et al. (2009) describes a rigorous process to explore what constitutes success in priority setting processes in the health sector.

Sibbald et al. (2009) undertook three related empirical studies to understand success from several relevant stakeholders’ perspectives. First a Delphi Study[iii] was run to iteratively build consensus among scholars and healthcare decision makers. Next, the authors undertook a series of interviews with senior decision and policy makers who function on various levels of the health system – for instance on a Macro level, Ministry of Health officials, or on a Micro level, clinical managers. Finally, focus groups with patients, caregivers, and policy makers were held. The data from the three studies were first analysed separately, then the findings were synthesised and analysed in aggregate. A key component of all three studies was the fact that participants had the opportunity to comment on and verify the provisional findings for reasonableness, and/or to determine if revision or further analysis was required.

The project resulted in a list of ten separate but related elements of successful priority setting. Some were outcomes concepts:

  • stakeholder understanding, meaning that stakeholders understand the process and organisation running beyond a basic level;
  • shifted priorities/reallocation of resources, meaning that real shifts in resources and strategies are an outcome of the prioritisation process;
  • decision making quality refers to better use of evidence, consistent reasoning, and implementing the priority setting in accordance with plans;
  • stakeholder acceptance and satisfaction, which can be measured directly (e.g. surveys) or indirectly (e.g. stakeholders continuing to engage in a process); and
  • positive externalities, which can take many forms, for instance, others emulating the prioritisation approach, positive coverage about the process in the media, or changes in legislation. (Sibbald et al., 2009, p. 7)

Others measures of success were process concepts:

  • stakeholder engagement, meaning identifying and involving all relevant stakeholders;
  • use of explicit process, meaning using a process that is transparent for all participants;
  • information management, refers to effective information sharing with participants both about data to inform the process, and also how data about the process is collected and analysed;
  • consideration of values and context, refers to decisions being made in a way that is rooted in clearly stated values; and
  • revision or appeals mechanism, refers to a range of approaches to ensure that the rules of the process are communicated, participants have a means to correct errors and suggest revisions, and overall allow the process to be responsive to participant feedback. (Sibbald et al., 2009, p. 7)

If we consider the common methods for and components of prioritisation (Fadlallah et al., 2020)  and this assessment of what constitutes success (Sibbald et al., 2009), there are some valuable overlaps that tell us how well research reports prioritisation success and how researchers can strengthen knowledge in this area in the future. First, it is clear that recruiting and engaging stakeholders is both a marker of success and a common practice. However, the measures of success in Sibbald et al. (2009) indicate that quality-related details are important to assessing success, e.g. are all relevant stakeholders involved? Are all stakeholders given the tools to participate actively in the process? Do they have recourse to provide follow-up feedback? Fadlallah et al. (2020) does not set out to address these questions in detail, so some of this information may be reported in individual studies. Future updates to this post may include recent examples of primary studies that bring to light the latest practices on this front. Another overlap is that about half of the papers in Fadlallah et al. (2020) indicated that criteria were used to support the ranking process. Criteria are an ideal means to address the markers of success related to explicit processes, consistent reasoning, and consideration of values. Criteria can make a process explicit by ensuring that participants know how the different criteria contribute to the overall prioritisation process, and in some cases, participants may be able to provide insight into which criteria should be most important. Criteria support consistent reasoning by provided a pre-set framework through which the ultimate outcome of the prioritisation process is reached. Lastly, criteria can help to make values explicit by showing which factors are most important in making a prioritisation decision (e.g. where to factors like potential for research impact, undertaking ethical research, or cost-effectiveness rank in the minds of the researchers and participants minds).

What are priority methodological research questions for researchers and other stakeholders in this field?

The Fadlallah et al. (2020) paper found that relatively fewer papers described follow-up, implementation, or feedback steps of prioritisation. I speculate that this fact may be, in part, a reporting issue for journal articles. Articles that focus on the prioritisation process and its immediate outcomes make for a concise, clear agenda-setting article after which the authors can publish an article about the research project that resulted from one of the selected topics or questions. Articles that focus on the meta-science of long-term outcomes from the research process itself are somewhat less common. Nonetheless, based on the measures of success described in Sibbald et al. (2009), future work in this area might benefit from articles that assess and discuss the longer-term outcomes of prioritisation processes.

Finally, we’ll consider a recent article about setting the agenda for future research on prioritisation methodologies resulting in rapid review[iv] topics (Beecher, 2022). This article will help to solidify the current consensus on where this field can and should be headed. The study used a James Lind Alliance Priority Setting Partnership (JLAPSP) approach, which brings together a range of stakeholders to identify uncertainties and unanswered questions, build consensus around the most important uncertainties, and then publicise the methods and results[v]. In this instance, the research team used online surveys and a prioritisation workshop, in which the public, reviewers, researchers, clinicians, policymakers, and funders participated (Beecher, 2022, pg. 151). The outcomes very much reflect the findings above about stakeholder engagement.

NOTE: The links in this section of text will connect you to a website with more detailed descriptions of the key questions resulting from this study.

“Two of the top three prioritized questions ask how people or groups who will use the results of a rapid review (e.g., stakeholders such as patients and the public, clinicians, policymakers), underserved stakeholder groups (e.g., ethnic minorities, socioeconomically disadvantaged), and stakeholders from underrepresented countries (e.g., countries of different income levels) can be identified and have meaningful involvement in planning and doing a rapid review, and in reporting and sharing the findings.” (Beecher et al., 2022, p. 158)

I would add to this observation that another question asked “What is the best composition of a rapid review team (e.g. expertise)?” In my own rapid review work, engaging content matter experts has often been a valuable step not only for analysing and interpreting the findings of studies, but also to determine how best to make the results useful to relevant users because they can help share findings among professional networks, have awareness of knowledge of current policy and research discussions in the sector, or know key nuances about how to communicate to various audiences. Further, it should be mentioned that my view is that “expertise” could extend beyond in-depth academic or professional knowledge, and that on some occasions lived experiences and other forms of knowledge could often enhance the quality and relevance of rapid review outputs. However, the practicalities of integrating these forms of expertise requires additional study.

Another question that was prioritised by this JLAPSP, asked about the most useful processes for developing a review question. This strand of research reflects the already growing work discussing how the use of criteria supports transparent processes. Moreover, this strand could support undertaking research on the best ways to develop feedback mechanisms for participants in prioritisation processes.

Lastly, the participants in the JLAPSP, also prioritised the question “What training or supports are needed to help people to plan, do and share the findings of rapid reviews?” This insight directly relates to a wide range of the markers of success highlighted by Sibbald et al. (2009), namely: a strong focus on stakeholder engagement, use of explicit process, information management, consideration of values and context, and stakeholder acceptance and satisfaction. In other words, undertaking and reporting on more research about how to ensure that prioritisation participants have all the tools they need to adequately participate, support follow up, hold researchers accountable, among others, looks like a critical topic of research for addressing quite a few markers of success.

Conclusion

In this post I’ve explored what constitutes success in a prioritisation process related to research and decision making in the health sector.  Based on the two reviews included in this post to date, some aspects of successful prioritisation processes (e.g. a focus on identifying and engaging a range of key stakeholders) are already a core focus of researchers in this space, whereas some areas (e.g. more research on how the outcomes of prioritisation are implemented and/or follow up steps with stakeholders).  In the future as this post is updated, I’ll aim to incorporate more primary studies that might provide other examples of ways that success might be measured or where gaps in knowledge may be getting filled.

[i] Evidence synthesis refers to a “type of research method that allows researchers to bring together all relevant information on a research question”, https://www.lshtm.ac.uk/research/centres/centre-evaluation/evidence-synthesis

[ii] Where possible I favour more recent articles on the Good Questions Review, as long as they are also high-quality and relevant to the question. These usually incorporate the insights of other relevant work. However, in this case, the best-suited article to respond the guiding question of this article in my searching thus far was from 2009. Given the nature of this living literature review, I will surely add to this post when I encounter other, more recent articles.

[iii] Defined in a previous post as “…a well-established method that involves asking experts to engage in several rounds of surveys to reveal consensus…”.

[iv] To my knowledge there is no consensus on the definition of a rapid review, however Garritty et al. (2020) as “…a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting specific methods to produce evidence for stakeholders in a resource-efficient manner”

[v] https://www.jla.nihr.ac.uk/jla-guidebook/chapter-2/what-is-a-priority-setting-partnership.htm

Articles cited

Beecher, C., Toomey, E., Maeso, B., Whiting, C., Stewart, D. C., Worrall, A., Elliott, J., Smith, M., Tierney, T., Blackwood, B., Maguire, T., Kampman, M., Ling, B., Gill, C., Healy, P., Houghton, C., Booth, A., Garritty, C., Thomas, J., … Devane, D. (2022). Priority III: Top 10 rapid review methodology research priorities identified using a James Lind Alliance Priority Setting Partnership. Journal of Clinical Epidemiology, 151, 151–160. https://doi.org/10.1016/j.jclinepi.2022.08.002

Fadlallah, R., El-Harakeh, A., Bou-Karroum, L., Lotfi, T., El-Jardali, F., Hishi, L., & Akl, E. A. (2020). A common framework of steps and criteria for prioritizing topics for evidence syntheses: A systematic review. Journal of Clinical Epidemiology, 120, 67–85. https://doi.org/10.1016/j.jclinepi.2019.12.009

Garritty, C., Hamel, C., Trivella, M., Gartlehner, G., Nussbaumer-Streit, B., Devane, D., Kamel, C., Griebler, U., & King, V. J. (2024). Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness. BMJ, e076335. https://doi.org/10.1136/bmj-2023-076335

Sibbald, S. L., Singer, P. A., Upshur, R., & Martin, D. K. (2009). Priority setting: What constitutes success? A conceptual framework for successful priority setting. BMC Health Services Research, 9(1), 43. https://doi.org/10.1186/1472-6963-9-43