What is an evaluation plan, a protocol and a matrix?

A brief summary of each

Evaluation plan – Typically a stakeholder-facing, overarching guide covering all aspects of the evaluation from project/program objectives to evaluation reporting

A plan provides a high level overview of how the evaluation will be conducted, and is commonly developed for both small and larger evaluations. In smaller size evaluations the plan may be used by evaluators as well as an evaluation operational guide, where development of a full evaluation protocol is not feasible due to resource constraints, time or the inherent nature of the evaluation.

Evaluation protocol – An evaluation team-facing document, providing operational and detailed procedural instructions to the evaluation team as to 'how' the evaluation will be conducted (detailed data collection and analysis methodology and how reporting will be synthesised)

Evaluation matrix – A high level tool which organises and displays the key components of the evaluation in a table/spreadsheet

A matrix provides a visual, easily-read display of the relationship between key evaluation questions, outcomes, data sources, data indicators/themes, data collection methods and timepoints for data collection. Essentially it displays what is being measured, how it will be measured, how it will be collected, and by whom.

How do they differ?

- An ‘Evaluation’ Plan should accompany and support the ‘Project/Program’ Plan¹. The ‘evaluation plan’ connects into this ‘project/program plan’ how the project will be evaluated across all phases of project implementation to completion. It may be a section of the ‘project plan’, part of the appendix or a separate document depending on the size and scope of the project/program and associated evaluation.
- An Evaluation Plan provides information about the governance structure for monitoring and evaluation reporting of findings and frequency of such reporting. Further, how and when findings will be presented to key stakeholder groups to inform their decision making related to the program/project under monitoring and evaluation. Details the high level analysis plans for data and any limitations of analysis or the evaluation methodology more generally, and how potential biases in evaluation will be avoided. The plan includes for stakeholders the Theory of Change, Program logic, what the key evaluation questions (KEQs) are that are to be answered. It may also detail evaluation rubrics to assess project/program success.
¹A project/program plan details: what the project is (scope), why it is to be implemented (justification), how it will be implemented (instructions), governance structure, roles and responsibilities of project team, milestones, risk register and budget.
- An ‘Evaluation Protocol’ details how evaluation and monitoring will be conducted in greater methodological detail. As well as detailing the theory of change, program logic, and the key evaluation questions (KEQs), it provides detail on the metrics/indicators that will be collected for various outputs and outcomes, provides a schedule of timepoints for different data collection, details what qualitative data will be collected (for example, interviews, desktop review, or focus groups), how data (quantitative and qualitative) will be collated, cleaned and analysed (quantitative analysis methods and/or quantitative statistical methods), the evaluation methodology that will be used across the evaluation (process, impact, summative, developmental) and a schedule of monitoring and reporting activity.
- It also details the tools, questionnaires (and their justification for selection and/or design) and other evaluation methods that will be used to capture data, as well as detailing the themes of interest for evaluation exploration related to the key evaluation questions to guide interviews, focus groups etc.
- The evaluation protocol connects into the evaluation delivery any key evaluation related policies, frameworks, standards and/or guidelines related to the safe and responsible operation of the evaluation that the evaluation team should consider and follow. (eg. requirement for ethics approval, consent to provide data, policies related to data privacy and protection, cultural safety frameworks etc.)
- Having a detailed evaluation protocol ensures the evaluation is replicable by others, supporting standardisation of evaluation approach, quality and accountability of evaluation delivered and supports consistency in the evaluation over time.
- An evaluation matrix is usually included as an appendix or section added to the evaluation plan and/or protocol, mapping in a simple and visual way how key evaluation activity connects up to answer the key evaluation questions.
- It is usually set up as a table or spreadsheet with the evaluation questions guiding the population of other matrix components.
- The matrix can be as simple or as detailed as is necessary for the size, scope and complexity of the evaluation but should provide to stakeholders an easy way to scan over how components of the evaluation connect up.
- The evaluation matrix can be as simple as providing the key evaluation questions, the indicators for success for those questions and details of the data which will be analysed and used as evidence to answer the evaluation questions.
- A more complex and detailed evaluation matrix might also present where data will be sourced from (eg. survey, database, interviews etc), key metrics of interest in the data, data identifiers/groups of interest (students, teachers, employers etc), evaluation key evaluation question themes that will be explored at different timepoints and stages of the evaluation process and identification of who is responsible for data collection at each point.

Do I need all three?

The complexity and the size of the program/project will determine how detailed and extensive the evaluation preparation should be. A general guide is as program size and complexity increases so should the evaluation. A more complex and larger-scale evaluation has more potential for things to go wrong or for something to be missed. Involving stakeholders in the development of an ‘evaluation plan’ can reduce confusion and ensure expectations around the objectives and scope of the evaluation are clear.

Generally a quality evaluation plan, accompanied by an evaluation protocol which details high quality evaluation methodology, will better enable evaluators to report on the robustness of the results from the evaluation. Essentially together the 3 tools can ensure the evaluation is set up to succeed, creating for example the ideal of a large enough data sample for the statistical analysis tests desired to answer the key evaluation questions. Such planning can also better avoid data attrition over time, better prove the validity and reliability of the analysis findings, improve transparency and accountability of evaluation approach and generally reduce potential evaluation bias to ensure the evaluation supplies robust intelligence.

Additional resources

Education, New South Wales Government - Evaluation Hub

Centres for Disease Control and Prevention (CDC), United States Government - CDC Program Evaluation Framework, 2024

Intelligence and Insights