What do we know about how organisational factors shape individual academics’ research agendas and how do we know it?

What do we know about how organisational factors shape individual academics’ research agendas and how do we know it?

This post is written by Paul Kellner as part of Good Questions Review, a living literature review about how social science can be useful for making decisions. It was made possible through support from Open Philanthropy.

Note: this post is continuously updated as relevant articles are added to Good Questions Review. An important part of this project is archiving substantive edits to posts. To do this we create a digital object identifier for the original post as well as subsequent versions. Please find the archived version of this post here.

There are several posts on Good Questions Review that provide some insight into what might influence academics’ research agendas – for instance, demand for research and perceptions of policy relevance or identifying under researched topics. Learning about the factors that shape individual researchers’ agendas can help us better understand, among other things, the conditions that foster high-quality, insightful, pioneering, and/or useful research. This post unpacks some relatively recent research on if and how organisational factors might shape social scientists research agendas.

Note: As with many other topics covered on Good Questions Review, this topic is potentially quite expansive. For instance, this discussion could include nuanced considerations of research and reporting about a number of national research quality frameworks, the UK’s Research Excellence Framework (REF). So, it’s important to note that this post focuses on data on how individual academics describe their research agendas and the organisational factors that influence them. Moreover, as is common on other Good Questions Review posts, a description of how researchers undertake this work is featured almost equally alongside the studies’ findings.  

Organisational factors associated with different research agenda types in the social sciences

The work of João M. Santos, Hugo Horta, and colleagues has provided several substantial contributions to this topic in recent years. They have published several articles using a survey tool they have developed called the Multidimensional Research Agendas Inventory (MDRAI)[i]. The purpose of the tool is to provide a quantitative way to identify the factors the influence individuals’ research agendas, breaking them down into the following list:

  1. “Scientific ambition is associated with the willingness to be recognised in a field of knowledge and thus to obtain prestige and increased access to resources,
  2. Convergence refers to a preference for disciplinary bounded research agendas, and indicates a preference to avoid shifting the foci of research (stability)…
  3. Divergence refers to a preference for research agendas that address themes from a multidisciplinary perspective
  4. Discovery is associated with a preference for a risk-inclined research agenda, manifested in the choice of research topic or a propensity for emerging topics with uncertain outcomes.
  5. Conservative refers to a risk-aversion preference for research, and to choosing topics and fields well covered in the literature where uncertainty is less prevalent.
  6. Tolerance to low funding is a measure of the risk tolerance associated with opting for a research focus that may have very little funding and is expected to be sensitive to organisational pressure towards the acquisition of competitive research funding
  7. Collaboration refers to the interest in engaging in collaborative research agendas and can be influenced by institutional pressure to collaborate,
  8. Mentor influence refers to the degree to which individual research agendas are influenced by the PhD supervisor” (Horta & Santos, 2020a, p. 2386)

The authors have used an impressive recruitment strategy for some of the studies using the MDRAI. In these studies, they’ve undertaken a comprehensive search for journal articles in a field (e.g. higher education) over a period of time (e.g. 10 years). They then have invited all the corresponding authors from all articles to participate. This means distributing thousands of invitations to participate.

Several of their studies are designed to analyse researchers’ responses to the MDRAI survey tool in conjunction with other survey tools with the purpose of understanding the relationship between research agendas and other factors, e.g. organisational characteristics or personality traits. One such study explores how “organisational factors related to the working research environments of universities associated with the research agendas of academics in the social sciences?” (Horta & Santos, 2020, p. 2383). In this study they ended up getting responses from 735 corresponding authors.

The people who filled in the MDRAI for this study also filled in Multi-Dimensional University Research Workplace Inventory (Santos, 2018), to learn about the researchers’ organisational factors/characteristics across five dimensions:

  • “Organisational commitment refers to the degree to which an individual identifies with and is committed to an organisation [broken into belonging, willingness to stay, and satisfaction with leadership sub-dimensions]…
  • ”resources…[which relates to] perception of access to resources in the department/university…”
  • “…social satisfaction, which refers to the level of satisfaction to academic has for his colleagues…”
  • “…autonomy…the perceived level of autonomy the academic believed to have in their department/university…”
  • “…unconstraint, which measures the lack of institutional constraints and obligations unrelated to research [e.g. teaching]…” (Horta & Santos, 2020a, p. 2387)

The analysis used the MDRAI (individual research agenda) factors as dependent variables and the MDURWI (organisational characteristics) as explanatory variables. In other words, it was organised to understand how organisational factors might explain different research agenda dimensions. Horta & Santos (2020) also asked participants to respond to an additional survey to help control for individual personality characteristics. They used the BFI-10, a well-established, reliable survey used for this purpose (see Costa Mastruscusa et al. 2023). Including BFI-10 allowed the study to account of the in influence that individual personality traits like openness to experience, conscientiousness, agreeableness, neuroticism, and extraversion, may have on the agenda setting process, which can help to better isolate the real influence of organisational characteristics on individual social scientists’ research agendas.

Note: This study design can allow us to know a lot about associations between characteristics, but cannot tell is if one causes another.

Their findings provide several insights. First, on a basic level, the study showed that organisational factors are, in fact, associated with individuals’ research agendas. In other words, the organisational context in which researchers work may shape the research agendas they choose – a finding consistent with previous studies. More substantively, the study found that social scientists working in organisations that provide more autonomy and a more collegial environment reported that their researcher agendas were “…more multidisciplinary, collaborative and disruptive” (Horta & Santos, 2020, p. 2391). Related to these findings organisations that foster a sense of autonomy were found to be associated with more ambitious research agendas, they types of which, the authors suggest, may result in higher researcher output and recognition seeking behaviour.

Autonomy was also found to positively impact researchers’ reported scores on the discovery and tolerance for low funding dimensions of their research agendas. The authors suggested that these aspects of research agendas may result in research that has a higher potential for significant breakthroughs, but also involves greater risk of failure. Conversely, when researchers had high scores on organisational commitment – which includes researchers’ commitment to the organisation’s managerial policies – this was associated with research agendas that have greater conformity, take less risks, and are less likely to be disruptive.

Interestingly, the authors also found that a lack of pressure to work on tasks outside of research reduced the multidisciplinary and collaborativeness of individuals’ research agendas. In other words, organisations that place a strong, primary focus on social scientists spending most of their time doing research – rather than a mix of research, teaching, and other outreach activities – may be creating an environment where researchers stay within their disciplinary silos and/or work less collaboratively.

Based on these findings, Horta & Santos (2020) reflected that research quality frameworks and research performance management approaches that reduce researchers autonomy by closely regulating the types of research undertaken by researchers and where it is communicated (e.g. by incentivising publishing in high ranking journals) may not yield the results desired by those frameworks and systems. They suggest that, in some cases, such close regulation may even be counterproductive to the goals of some systems – e.g. if they are trying to foster research that yields transformative breakthroughs or deals with complex, multidisciplinary challenges.

A case study from Italy

Building on the last observation from Horta & Santos (2020), the next paper I’ll look at presents a case study about a new performance management system in one Italian university in the context of several years of applying a national trend towards a new public management style approach. Guarini et al. (2020) developed a case study based on a public university to understand how academic staff adapted to the new performance measurement system and assessment approach emphasising, among other things, increased publication in top-ranking journals. I’ll look at the findings of this case study mainly in reference to the Horta & Santos (2020) article.

The case study in the Guarini et al (2020) paper was selected because it was an outlier amongst its peers – implementing performance management systems that aligned very closely with national government metrics developed for the same purpose. The article provides a thorough consideration of the new public management approach and the recent history of several research assessment frameworks, for which there isn’t room in this post. However, it should be noted that this trend has led some researchers to argue that these approaches have led universities to adapt “…their structures around the idealised model of corporate performance to enhance research excellence and impact…” (Guarini et al., 2020, p. 112). They are structured in a way that may provide more central guidance about how to measure research performance and often have strong in-built systems for transparency and peer review. There have likely been several impacts of systems like this including, potentially, increases in overall research productivity. However, researchers also lodge criticisms ranging from reduced academic freedom, the approach incentivising high ranking publications rather than knowledge dissemination more broadly, among others. Again, this post will not be able to provide a full account of this debate.

In that context, Guarini et al. (2020) undertook face-to-face interviews with participants from eight departments – four from hard science (e.g. physics) and four from social science (e.g. sociology). Departments that were seen to be high performing and low performing within the new performance management system were selected for both the hard and social science groups. They used two different interview guides in the study, one for department heads and one for faculty members.

On a high level, the findings show that aligning university-level performance management with national performance metrics and associated government funding schemes can affect the way universities allocate resources and how researchers set their agendas. This is similar to Horta & Santos (2020) finding that organisational characteristics were associated with individual research agenda. Although this is again unsurprising, it is valuable to confirm with data that organisational characteristics are connected to how individual researchers develop their agendas.

The interview findings are also largely consistent with the survey findings from Horta & Santos (2020) about the way that organisational characteristics relate to research agendas. The new performance management system, which placed emphasis on producing publications in high-ranking journals, seemed to be associated with some significant changes in the research behaviours of academics. The impact of the system was likely greater for social scientists and in departments that were seen to be underperforming. The authors found that a much larger proportion of the social science participants reported a meaningful change to their research agenda.

Some interviewees observed that although the new system did have positive impacts for transparency and objectivity, they also felt that some procedures and measurement approaches (like journal rankings) may not be ideal from their perspectives. Several felt that the focus on high-ranking journals was seen by some as potentially counterproductive to the overall advancement and dissemination of knowledge. Interviewees reported targeting different journals and more actively structuring research projects in a manner that results in optimal outcomes in relation to the publications incentivised by the system. Some also felt that this dynamic extended to the topics of research as well – with researchers increasingly focusing on “fashionable” topics to ensure high ranking publications. Specific to social scientists, some interviewees reflected that the system may have also resulted in researchers undertaking less research on topics of national importance and/or producing publications that might reach critical audiences outside of academia via writing articles for newspapers or writing in Italian. One interviewee placed a fine point on this by saying that some of his publications would have been better suited for lower-ranked publications (that may be in Italian or simply lower ranking journals), but he instead focused on getting them into the top-ranked journals because it was incentivised by the system. This change extended to disciplinary traditions, as social science academics also noted that publishing in books and book chapters was disincentivised, which is historically a common format for some social sciences.

Regarding collegiality, previous systems in the university had relied on assessing researchers’ performance based on the values of each discipline. However, the new system framed performance increasingly in terms of, among other factors, winning of external grants and the journal publications described above. There was also competition between departments and individuals to access internal and external funding. Interviewees, in some cases, reoriented their research agendas to maximise their success in reference to these funds.

The authors summarised the interview findings by saying that the new performance management system “…created a “closed incentive system” that values only individual research performance. In such a system, performance metrics become an instrumental device for valuing time allocation to research, teaching and third mission activities. Although this incentive scheme may be rewarding for highly motivated researchers, it may surreptitiously alter the commonly accepted notions of academic work and the university mission” (pg 128).  In their assessment, the authors also underlined the potential unintended impacts of the management system that other studies previously identified – reductions in academic freedom, less innovation, and more conformity in research agendas.

Conclusions

If we take the findings of these two articles which focused specifically on how organisational characteristics relate to individual research agendas, a few key themes emerge. First, organisations that provide researchers with autonomy likely foster greater diversity in research agendas. This likely means that, among those agendas, there are more risky ones that may not result in high productivity or performance. However, those riskier agendas may also transcend disciplinary boundaries to potentially meaningfully engage with the complex problems facing our societies (e.g. the diverse interconnected impacts of climate change). Closer regulation of research outputs, like pressure to publish research in high-ranking journals, may also displace other important goals of universities – namely to teach students and effectively disseminate knowledge to the public. Finally, in relation to collegiality, these papers seem to indicate that organisations that foster collegial environments may ultimately result in research agendas that reach across disciplinary boundaries are potentially transformative.

These things being said, it is important to note a few things about this post. Although these studies above describe associations between organisational characteristics and research agendas, it is important to say that the study designs cannot tell us anything about causality. Also, I’ve done my best to avoid making any claims about what the ultimate purpose of a university should be – rather, attempting to present what some researchers found about how organisations are structured and how researchers develop their agendas within those structures. Depending on the way that our communities and societies see the role of universities, this may have implications for what organisational characteristics might foster the research agendas that feed that purpose. Lastly, this post will be placed on a high priority list for updating, as well as for developing complementary posts, because this is a nuanced and complex issue about which I’ll seek additional scholarship.

Thanks for reading!

[i] This tool was originally designed for the social sciences and revised for use with all disciplines in the MDRAI-R, which can be found here. This incorporates additional dimensions based on the authors previous use of the MDRAI and other relevant scholarship.

Articles cited

Costa Mastrascusa, R., de Oliveira Fenili Antunes, M.L., de Albuquerque, N.S. et al. Evaluating the complete (44-item), short (20-item) and ultra-short (10-item) versions of the Big Five Inventory (BFI) in the Brazilian population. Sci Rep 13, 7372 (2023). https://doi.org/10.1038/s41598-023-34504-1

Guarini, E., Magli, F., & Francesconi, A. (2020). Academic logics in changing performance measurement systems: An exploration in a university setting. Qualitative Research in Accounting & Management, 17(1), 109–142. https://doi.org/10.1108/QRAM-06-2019-0076

Horta, H., & Santos, J. M. (2020). Organisational factors and academic research agendas: An analysis of academics in the social sciences. Studies in Higher Education, 45(12), 2382–2397. https://doi.org/10.1080/03075079.2019.1612351

Santos, J. M. (2018). Development and Validation of the Multi-dimensional University Research Workplace Inventory (MDURWI). Higher Education Policy, 31(3), 381–404. https://doi.org/10.1057/s41307-017-0064-6