Diverging assessment: Assessment as learning

There were three key inspirations for diverging assessment.

1. First, no learning opportunity for students during the assessment period;
2. Second, the lack of authenticity for a lot of assessments and;
3. Third, addressing students’ exam stress.

That led me to start rethinking my assessments to be more authentic, the more authentic the better.

I believe any assessment task should create learning opportunities as opposed to ‘assessment of learning’ in which we assess our students about their learning, like exams. I don't look at the assessments as a means for only evaluating students but as an opportunity for students to learn during the assessment.

Students should be responsible for their own assessment and practise learning at any step in the assessment. That includes reading the assessment task, seeking feedback from the peers or from any teaching team members, because that's really what professionals do in their job.

Diverging assessments are designed to generate learning opportunities that foster students' meta-cognition, encouraging students to work together as a group, but also address concerns about academic integrity and the use of generative-AI-based tools.

Setting up diverging assessments is a bit technical, but it's doable. I use Moodle quiz which has auto marking, which means once the initial assessment has been set up, there is minimal involvement required for tutors or teaching team members for the marking process.

First I generate more than one set of input data and upload all them along with one single set of quiz questions to a question bank. Then I set up the Moodle quiz that contains the single quiz questions and then Moodle would randomise each student’s access to one of the generated input data.

For example, in the context of FIT9137 (Introduction to computer architecture and networks), for a class of around 500 students, I prepared 50 unique files i.e. versions of the quiz question. I do this by setting a mock environment and running different tasks, for example sending an email with different attachments, visiting a website, browsing the internet, doing some online shopping, uploading something online to the cloud, and so on and so forth. All of these actions will be captured and stored in a network traffic file (pcap file), which I upload to a Google drive file to be given as input data to each student.

As I generated the artefacts myself, I knew the answers to each version of the question. Using XML format, I import them as quiz questions into the Moodle quiz question bank.

When students attempt the quiz, each randomised Moodle quiz question contains a link to a pcap file. Students download the linked file, and using that unique data set, they apply the tools that they have learned in the class to extract specific information. For example, students must look at the network and extract specific information, such as the size of an email attachment, address of the sender, IP of the receiver, etc.

For each student, the questions are the same but the file assigned to each student is different, so the correct answers vary from student to student. Students share this common framework, and we encourage them to work together. Students can seek feedback from myself, from the teaching team or their peers. The assessment task really encourages peer learning and incentivises them to work together.

I have run such quizzes in different formats, e.g. both time and untimed. For the untimed quiz, the quiz will be open for students for two weeks. The timed quiz is open for an entire day. Students have three attempts (for the timed quiz each attempt is limited to two hours, similar to an exam), so if they realise they can do better and potentially improve their mark, they can attempt the assessment again. The catch is that if students start another attempt, they will be given a new batch of data. The data set is very different from student to student, it prevents any sort of academic integrity concerns or issues.

The assessment is open book, there is no invigilation required and we encourage students to seek feedback from myself, from the teaching team or their peers. Students can use generative AI as tools and share strategies, but also learn the limitations of generative AI.

For example, the pcap files are very special in format, so current generative AI platforms are unable to process the information, and asking ChatGPT to do the test for you won’t give students the correct answers. Students need to know how to apply special tools and techniques to arrive at a solution.

I have units where I have completely replaced exams with three or four diverging assessments. Exams are usually 50-60% of the entire grade, so each of the quizzes are at least 20% (less than 10% is not worth the work that goes into setting up diverging assessment quizzes). I surveyed students about how they felt about the assessment compared to sitting an exam and the results showed that many symptoms of stress have been reduced compared to final exams. I’m proud to say that my units have become well-known for their assessments.

Click on the tabs to view the the comparison of students’ experience of stress while sitting final exams compared to taking our designed quizzes.

Diverging assessments are not only applicable to cyber security or networking. They have been implemented within a few units already inside the Faculty of Information Technology at Monash. As well as three other universities including RMIT, University of Wollongong and University of New England.

For example, I've done it in a unit in cybersecurity as well as networking computer networks. This is being done at both undergraduate and graduate level.

I believe there is a lot of scope to implement diverging assessments in engineering, maths and statistics. For example software development units, operating systems units or ethical hacking units.

Key characteristics of diverging assessments

	It should at least target the ‘apply’ level of the Bloom's taxonomy.
	Students are given some data, and then they are required to do some tests or drive some information or visualise something out of the data using a software tool and/or technique.

If you think your assessment meets the above requirements and need assistance with setting up a diverging assessment, you can contact me.

• Ensure that the expected input for the quiz questions is very clear. Quiz auto-marking doesn’t register answers as correct if there are typos or variations. For example, when the question asks for the size of an email attachment and the answer is 56KB. I expect students to enter ‘56’, but any kind of deviation will not be recognised by the quiz’s automatic marking, which then has to be manually marked correct. You can minimise this if you make your questions very clear.
• Don't be afraid of generative AI. Instead of waiting for university to tell you how to engage with it, try it yourself. If you tell your students how to use it, you will be surprised as this is often the quickest way for students to understand the limitations of generative AI.