Collecting qualitative data as part of your monitoring and evaluation

What is Qualitative data and how can it be valuable?

Qualitative data are measures of ‘types’ or ‘categorical’ information and whilst this data might be able to be analysed using some quantitative approaches (such as frequency) they require a numeric value to be applied to them first. Qualitative data is descriptive in nature; capturing qualities and characteristics which can not be quantified numerically. Collecting descriptive data supports a greater understanding of why or how things of interest to the project/program evaluation behave or change.

Whilst quantitative data collection is important for the statistical analysis of evaluation evidence to determine the effects of a project/program, it doesn’t commonly provide evaluators with the information necessary to understand why such effects occur (or don’t). A mixed methods evaluation approach which collects quantitative data and qualitative data can provide a more holistic analysis of a project/program’s effectiveness.

How do you collect qualitative data?

Qualitative data can be collected as part of the evaluation in many different ways. When designing an evaluation plan, consider the most appropriate ways to collect the qualitative data related to the context of the project/program being evaluated. For instance, some common considerations are whether there are any challenges associated with the availability/access to people who are a part of the project/program, the most appropriate format for data collection from project/program participants (eg. in person vs online), the ease and willingness of participants to share information, the level and depth of qualitative data required for the evaluation (eg. high level grouped thematic data or more detailed and individualised qualitative data that is much richer and contextualised to smaller numbers of participants), available resources and time.

There are many ways that qualitative data can be collected, which should be considered in the evaluation planning such as:

Use of open text survey questions in a survey can provide some narrative, personal experiences, and diversity in perspectives for a large group of participants. Data can be used to identify themes which can then be further qualitatively explored through other approaches such as interviews or focus groups.
One on one or small-group conversations probing around themes of interest to gain a rich detailed understanding of their experiences, perspectives and emotions. Interviews can be structured with fixed questions allowing only minimal topic deviation; semi-structured with some predefined questions yet flexibility to explore different topics at different depths and to ask new questions as themes emerge from responses; or unstructured using open ended conversation allowing the participant to largely guide the discussion.
Facilitated discussion with small groups, providing an opportunity for participants to interact and discuss with each other on themes of interest. Can assist in determining if themes are perceived in a similar way or differently by seeking consensus and or agreement as to the collective views on a theme.
Interactive sessions facilitated by the evaluation team to support collaboration and idea generation, the sharing of experience and/or the solving of problems using a participatory approach. eg. Data may reveal a problem that the workshops can then further explore to develop a solution to. Evaluators can record the process of problem identification by participants, and the ideation towards the design of a solution.
Participants record their thoughts, experiences, feelings etc over a set period, providing and sharing their insights and reflection on a project, program etc.
Typically non-intrusive real-time observing of participants in a target environment (e.g. classroom). Either structured, observing and recording specific activity, interactions or behaviours or unstructured observing and understanding behaviours and actions of participants as they occur.
Participants take photos or create visual prototypes to represent their experiences with opportunity for them to further discuss or explain the depth and context of their visuals. A useful way to prompt and explore sensitivity in ways that may suit some participant groups by helping to bridge language or cultural differences between participants.
Examining existing material such as reports, emails, meeting minutes or other documents to extract key themes, insights or other data of value.
Immersive evaluation where the evaluator actively engages with the group of participants or community to understand the cultural and social dynamics of the participant group and explore other contexts that the evaluator hopes to understand.
In depth exploration of a phenomenon, activity or other thing of interest, can involve multiple data collection across different participants around the same focus area to understand how the phenomenon works and its uniqueness.

How do you analyse qualitative data?

The type of data collected and the evaluation questions to be addressed will inform and guide qualitative analysis approaches. There are various qualitative analysis methods some of which can be combined or layered to provide additional value to the intelligence gathered from the analysis.

One of the first steps to consider is to code the qualitative data to make it easier to group together and compare. Establishing a coding plan for the data early in the evaluation can support the identification of information of interest to the evaluation and support identification of information gaps. Ideally, developing a coding frame or rubric is valuable to headline the main categories/themes or patterns of interest. Under these main themes/categories further sub themes can be identified. The sub-themes are less likely to be well defined from the outset and may only emerge as through the collection and analysis of the data.

Developing a framework for coding can better ensure your coding is applied in a systemised consistent manner across evaluators and collection/analysis timepoints. You can test and refine the coding framework with some sample data first to ensure all evaluators apply it consistently. Where coding inconsistency is identified, evaluators as a group can determine how to better categorise the coding framework to establish consistently.

Once the data has been coded, the following common analysis approaches may be appropriate:

Explores patterns and themes in your coded data, to identify further similar or different patterns or themes across your data. This analysis is often combined with content analysis to explore the frequency of coded themes across data or according to particular participant group characteristics, timing of the data or other factors of value to the evaluation.
A systemised way to break down data into smaller units for its categorisation and coding. It can help to understand patterns, themes, trends or other meaning to the broader data. Content could be as small as a particular word or could be an entire paragraph that talks to a theme of interest e.g. participant satisfaction. Conceptual analysis focuses on certain concepts in the data and relational analysis examines the relationships between the concepts. Content analysis can be 'inductive' reasoning allowing themes to emerge from the data or deductive reasoning based on pre-existing hypothesis, theories or frameworks for the concepts and themes being explored.
Structuring participant data into a story of personal account eg. how and why a participant perceived something of evaluation interest in a particular way and how that narrative may differ to a different participant. Comparing narratives for similarities and differences and learning about why differences exist.
Analyses language and communication to understand the way meaning is constructed. Particularly valuable for studying issues of ideology, inequality, and cultural practices. It aims to look at the deeper contextual meaning of language such as the social power dynamics it reveals or the influence of 'culture' on how language is used. The approach theorises that historical (time and history), social (relationships, class, gender and power dynamics), cultural (cultural norms, values and understandings) and power/ideology (language which maintains of challenges power or dominant social narratives and how groups are presented or misrepresented) contexts all influence use of language. eg how two different media reports might portray the same group of people (eg. refugees) with one portraying them as victims for instance and another portraying them as dangerous.
The approach recognises that reality is subjective and that the meaning of experiences is personal to each person. Analysis focuses on understanding and describing the lived experience of participants according to their ‘own’ subjective perspective. It can assist you to explore and explain participant experiences and their meanings. Analysis aims to look for commonality and difference across participant accounts and put aside the preconceptions and biases of the evaluator (bracketing). It can prevent you as an evaluator imposing your own interpretations of data.
Allows you to combine deductive (beginning w/ determined codes and themes) and inductive analysis (allowing for themes to emerge and be coded as the analysis progresses) by organising data into a structured framework based on themes and categories identified through the evaluation process. Provides a logical analysis approach to complex phenomena to assist show rigour and make the analysis replicable. It can be used in a theory-driven evaluation approach or in an exploratory but explanatory manner. The framework comes from ordering the data into rows and columns with rows representing the cases/participants and the columns representing the themes so that comparison across cases or themes can be systematically compared. The data in the matrix can then be further analysed to identify patterns, relationships and/or contradictions related to the key evaluation questions.
A method used to explore and understand complex aspects of a case within a particular system or process. It aims to explore and investigate the context, dynamics and other details of the case in very fine detail in order to provide a comprehensive understanding of the case and associated contextual phenomenon. It commonly uses multiple data sources e.g. interviews, observation and document review to assist triangulate an understanding of the case under investigation. Case studies can be sued in many different evaluative ways to highlight for instance the uniqueness of characteristics or value/importance of the information collected; to draw attention and provide insights into a theory of interest; to help demonstrate a larger phenomenon of interest; to compare and explore a broader phenomenon by comparing multiple cases and looking for patterns and differences; to explain causal relationships as to why and/or how certain outcomes occur.
Used to explore the structure and patterns of spoken interaction to understand how conversations are organised and participants use language to create meaning, manage their interactions and create mutual understanding. Aspects such as turn-taking and interruptions are analysed to understand how conversation unfolds. observations may reveal how misunderstandings, errors or other breakdowns in communication are repaired to maintain flow of the interaction. It also explores how context around the conversation influences the interaction. Voice or video data is typically collected and the evaluator transcribes the information into non-verbal and verbal elements such as pauses, intonations and other intricacies of the conversation. This type of analysis might be used for analysing a teacher student interaction in a classroom to explore how the conversation flows (or doesn't) as well as how teacher and student react throughout the interaction.