How can we be better aware of under researched topics? What role is there for “policy irrelevant” research?
How can we be better aware of under researched topics? What role is there for 'policy irrelevant' research?
The highly policy relevant field of forced migration studies has some unique insights.
This post is written by Paul Kellner as part of Good Questions Review, a living literature review about how social science can be useful for making decisions. It was made possible through support from Open Philanthropy.
Note: this post is continuously updated as relevant articles are added to Good Questions Review. An important part of this project is archiving substantive edits to posts. To do this we create a digital object identifier for the original post as well as subsequent versions. Please find the archived version of this post here.
The categories and frames of reference that researchers bring with them into their work can have a meaningful impact on the questions that they ask and the answers that they find. The categories and frames that researchers use are shaped by a wide range of factors including their disciplinary background, methodological skills, their funders’ priorities, the categories and frameworks used by decision makers who might use their research, among many others. Developing a research project and having the opportunity to undertake the work can often involve finding a balance point between these various factors. Sometimes these factors can even be at odds – for example, a project that best provides insights to a near-term policy decision may be quite different that is highly innovative for a particular academic discipline. These dynamics can contribute to some topics being “over researched” and “under researched”. This post describes a contemporary method for compiling and analysing data about what subjects get researched and what might need more focus in the field of forced migration. It will show applying additional data to a range of documents can help us map what categories are most present in the research and which ones may merit further emphasis or exploration. Finally, it will raise some key questions about the potential role for “policy irrelevant” questions.
Before discussing the articles, it is valuable to know that both articles in V1 of this post originate from the field of forced migration studies. Forced migration refers to situations when people move due to circumstances or forces beyond their control, whether that is within or between countries. Forced migrants are often categorised as refugees, internally displaced people (people who have been forced to leave their homes due to a range of circumstances, but remain within their country’s borders), asylum seekers, or people who have experienced trafficking, among others. There are important legal definitions of some of these terms that are often very relevant to what topics get researched and how data are analysed.
Research outputs from forced migration studies are often directly targeted towards influencing policy and policy makers. This makes logical sense – it is natural that researchers working in this field are very much interested in how to use their research outputs to improve the circumstances of people who have been affected by conflicts, natural disasters, or other very difficult events. Beyond this, issues related to forced migration are often at the centre of public discourse, e.g. immigration policy. Therefore, again, researchers’ work in this field is often in very close conversation the issues of the day.
The Australian Research on Refugee Integration Database
A recent article from Olliff et al. (2023) presents a compelling approach to better understanding what subjects are most emphasised in refugee integration research in Australia, as well as the categories and concepts that are most used. The authors describe the establishment of the Australian Research on Refugee Integration Database (ARRID) and use data from ARRID to explore what factors frame, promote, or constrain knowledge production in their field.
The database contains bibliographic information for more than 2000 journal articles, books, theses, and reports. The team that compiled the database supplemented the existing bibliographic data from these documents with metadata about thematic categories and sub-categories developed on the basis of the UK Home Office’s Indicators of Integration. An example of these categories is “Markers and Means” with its subcategories consisting of work, housing, education, leisure, or health and social care. The creators of ARRID refined some of the categories for their local context and added the ability to search by author and other data like those pertaining to mediating factors, e.g. age, ethnicity, visa type. The database was created, and the categories were added, with the purpose of facilitating knowledge exchange between researchers, policymakers, and other community stakeholders, as well as to explore how integration is researched and conceptualised in Australia. Databases like ARRID are an invaluable tool for researchers to build their understanding of knowledge production. Analysing data like those added by the ARRID team makes it possible to see how various aspects of refugee integration research may reflect trends from other data sources or aspects of social life like demographics, media discourse, or shifts in policies, legislation, or regulations. Through such analysis, researchers can determine what topics might be under researched; consider where there might be “blind spots” in knowledge, practice, or policies; identify topics that might benefit from studies using another methodology; among others.
The analysis in Olliff et al. (2023) uses descriptive quantitative analysis at a document-level to highlight trends within ARRID. This means that their analysis did not explore the contents of each document in detail. Rather, the analysis focused on the body of work comprising the totality of ARRID itself. This means that the paper can tell us things like how much research there is on various categories of integration research, where categories might co-occur (e.g. mental health and service access), or if an author produced multiple works on the same topics on a period. The descriptive nature of this work means that determining how or why these trends might be occurring would require further study, often with additional data gathering/methods. To put things another way, the methods used in the article cannot provide a causal argument for why particular trends in the category and other ARRID data are observed. However, analysis like the one presented there can then help researchers begin theorising better what issues are or should be on the research agenda, as well as how and why some things are getting prioritised over others.
At this point it may be interesting to dive into a few of the findings and what they might mean for further study. For instance, more than a quarter (26.9%) of the documents pertained to refugees’ experiences of general institutions like schools or healthcare providers, and another 14.1% focused on refugee specialist services. The authors interpreted this finding to indicate that a large portion of research is focused on policy and practice related questions that underline engagement with basic services as a critical element of integration. In other words, much of the research on engagement with institutions was focused on the practicalities of becoming integrated into systems and processes of government. The authors noted that comparatively less research focused on topics like voting behaviour or participation in political parties. The authors argue that the comparative lack of these other civic engagement focused categories may potentially reflect a narrative about refugees as being victims and reinforce deficits-oriented view of their role in society. This argument could lead to the conclusion that future research should interrogate if this deficits framing is present in the research, what such a framing might have on refugees day-to-day lived experience of services or undertaking more strengths- or assets-oriented research in the future.
Another insight was based on the fact that ARRID has the ability to filter by author. Olliff et al. (2023) found that of the 122 people who had authored doctoral theses in ARRID, less than half (49.2%) went on to produce other outputs in the database. Olliff et al. (2023) found that this may raise important questions about why people who write theses about refugee integration are not continuing to publish in this field either in scholarly journals or in plain language practice-oriented documents, which are also included in ARRID. The authors also ask what the impact might be of losing the knowledge that these authors have to other sectors or strands of work. Further, they draw on an insight from a scoping review undertaken by Squires (2018), that found that much of the conceptually innovative work on refugee integration in Australia was being done by students writing theses, and such innovation was less present in other document types. Based on this, it may be even more important to identify why thesis authors have not continued to publish on the subject of their doctoral work.
The analysis also found that the subjects of research were, maybe unsurprisingly, aligned with the disciplinary interests or predispositions of the researchers who were undertaking the work. To provide an example, a large proportion of the items in the database (9.3%) focused on wellbeing. Within this well being literature, 62.1% of these documents related specifically to mental health in formal service provision settings. Whereas only 10.9% focused on a more diverse array of work on potentially less formal aspects of supporting refugee wellbeing related to topics like social cohesion, community inclusion, befriending, among others. Olliff et al. (2023) reflect that these findings may be, in part, a function of the fact that several authors have training in social psychology and publish in social psychology journals, a field that may frame well being closely around mental health. Based on this, further study may be merited on how wellbeing is conceptualised in research on refugee integration with a particular focus on if wellbeing should narrowly pertain primarily to mental health, or if it is a concept that should be measured broadly across multiple dimensions.
Finally, they observed several demographic trends that did not necessarily align with the demographics of refugees in Australia. For instance, they found that young people (14.3%) and children (6.4%) were much more frequently the focus of research than older refugees (0.4%). Although young people make up a larger proportion of refugees, the disparity is not this wide. Similarly, women were the primary focus of much more research (11.0%) than men (2.0%). They also found that country of origin shaped what likely gets researched noting a strong focus on young people of African origin, a community that have been the focus on public discourse from politicians and the media. Overall, the authors observed key “silences” in ARRID on topics like identifying what issues refugees themselves think are important for integration, religion as a mediating factor for integration experiences, older refugees, refugees with a disability, and refugees who have a place of origin in the Americas.
Based on the above, I hope that it is clear that data like those compiled in ARRID can help bring to light several potential assumptions, categories, frames of reference that may be influencing a body of research, but also that these descriptive techniques are only the first step in confirming if and how these factors are shaping work in this field.
Highlighting the importance of “policy irrelevant” research
As mentioned in the previous section, the methods used in the Olliff et al. (2023) does not provide data that allows us to make any claims about how or why particular trends are present, but it does help us know where to look to begin asking those questions. Based on this finding, Olliff et al. (2023) highlight the argument for the importance of “policy irrelevant” research[i] made in an influential article from Oliver Bakewell (2008) “….warning that the emphasis on policy relevance privileges the worldview and assumptions of policymakers and…institutions that are the instruments of those policies…” (Olliff et al., 2023, pg 420). Moreover, a focus on policy relevance “…also engenders a narrow focus on characteristics deemed essential or unique to refugees, limiting the potential to apply insights from broader theories of social processes or transformations…” (Olliff et al., 2023, pg 420). Briefly describing the basis for the arguments made in Bakewell’s (2008) article may help to further underline the value of building a resource like ARRID.
Bakewell's (2008) is not fully summarised in this post, in part, because the full article provides rich and important context and details about Bakewell’s in-depth field work in Zambian borderlands about the repatriation of refugees. A summary would likely not do the full work justice. Thus, only the critical details are shared here. When Bakewell embarked on the fieldwork described in his article, he quickly realised that the research should shift away from a policy relevant frame of reference with which he had started the work – namely a focus on the repatriation of refugees. He found when doing the research in the Zambian borderlands that these formal institutional or legal definitions like refugee status and candidates for repatriation, were not as salient for the research, and may be problematic. He found a research context in which refugee status may a) be a category that some research participants would not ascribe to themselves even if they met the legal definition and, more importantly, b) it may not have been socially safe for people to identify as refugees or for other people to identify them as refugees. In short, the research presented several reasons for Bakewell to be guided by the data he collected more so than the institutional or legal categories that he carried with him into the field. Based on this, he argues that creating ways to put “…aside policy relevance and stepping outside the categories, we may be able to get a sideways look at policy and practice from a new angle…” (Bakewell , 2008, pg 449). Put another way, questioning the categories and frameworks that researchers bring into a research project can help them gain insights that they might not be able to see if they were only applying those categories and frameworks.
As mentioned in the introduction, Bakewell argues that research questions development in practice is often driven by factors like:
- the overlap between the researchers interests and a funder’s willingness to pay for the research;
- determining what can be done in a methodologically robust, ethical, and practical way; and
- and what the researcher might be capable of within their own body of knowledge related to the fields with which they are familiar.
Bakewell (2008) argues that attempting to be free of focus on policy relevance (i.e. the first bullet point above) may make researchers more able to understand assumptions that they may have taken for granted. He goes on to say that such an approach may enable researchers to better observe which parts of people’s social world are important to them independent of the categories applied to them institutions, thereby making more parts of their lives visible to the research. Such insights, in some cases, when brought back into conversation with policy conversations may help to revise frames of references, categories, and definitions in a manner that could not have been achieved without taking a “policy irrelevant” approach. This is indeed a potentially unique and positive role that academic researchers can play in policy spaces. Of course, there is also a balance to be struck, because true “policy irrelevant” work means that research outputs may not get used by policy makers and practitioners to benefit people in real need of support.
Conclusion
A database like ARRID is a great example of a starting point for critically assessing that trends and dynamics present in the field of refugee integration in Australia, while remaining “in conversation” with policy and practice decisions. Although ARRID applied existing, policy relevant categories (the UK Home Office’s Indicators of Integration), those categories were adapted for local needs and created a source of data that can be used to reflect on the assumptions inherent in those categories and the endeavour of refugee integration in Australia more broadly. Moreover, its descriptive insights can help researchers identify new opportunities to understand how concepts and categories exist across the whole field of study in reference to and outside of the Indicators of Integration, as well as how those categories are related, interact, overlap, or are invisible to researchers, policy makers and practitioners. It can help identify an important gap in knowledge, concepts that may need translation between disciplines or sectors, thereby beginning to explore and address several of the issues raised in Oliver Bakewell’s (2008) argument for “policy irrelevant” research.
Note: As with all posts on Good Questions Review, I’ll continue to update this post by looking for other recent papers that provide some empirical examples that respond to this post’s guiding questions. In this instance, updates will likely include other methods for gaining awareness of key gaps in research. Moreover, other posts will engage in the wider questions asked by Bakewell (2008), namely, what role is there for policy relevant questions and how can we put them into practice in ethical, practical, impactful ways.
[i] Research in which researchers, at least, attempt to design their research and analyse their data that is in some way “outside of” or independent of relevant institutional categories and frameworks. Full disclosure, I found the Bakewell (2008) article to make a very compelling argument when I first read it many years ago, and it was the starting point for my thinking about this post topic.
Articles cited
Bakewell, O. (2008). Research Beyond the Categories: The Importance of Policy Irrelevant Research into Forced Migration. Journal of Refugee Studies, 21(4), 432–453. https://doi.org/10.1093/jrs/fen042
Olliff, L., Block, K., Baker, S., Edwards, C., & Tran, J. (2023). The Australian Research on Refugee Integration Database (ARRID): A Platform and Conceptual Framework to Map, Connect and Share Knowledge. Journal of International Migration and Integration, 24(1), 403–423. https://doi.org/10.1007/s12134-022-00949-y
Squires, P. (2018). A scoping review of Australian studies of refugee integration: Popular definitions of integration in the Australian literature. Migration Studies. https://doi.org/10.1093/migration/mny032