Innovative code review education study accepted at premier software engineering conference

23 February 2021

Acknowledging the gap between industry needs and university offerings, Dr Chun Yong Chong and Dr Chakkrit Tantithamthavorn (Monash University) partnered with Dr Patanamon Thongtanunam (The University of Melbourne) to conduct a large-scale investigation into how teaching code review can be improved from a pedagogical perspective.

As the very first paper of its kind from the Faculty of IT across Monash University Australia and Malaysia, this puts the university at the forefront of world-class, innovative SE education.

‘We strive to be a world-leading group in all aspects of software engineering. This achievement clearly highlights the strengths and excellence of Monash’s education in the discipline.’ said Dr Tantithamthavorn, Lead of the Software Engineering group in the Faculty of IT at Monash.

First coined a branch of engineering at the 1968 NATO conference, software engineering (SE) has become one of the fastest-growing and evolving disciplines. Naturally, it’s now widely acknowledged that the quality of SE education drives the quality of software engineers in the ICT workforce.

Smoothing the learning curve

An independent, mature discipline on its own, academics are actively proposing innovative ways to improve the delivery of SE education. A range of policies and guidelines are also being put forward to adapt to the ever-changing software development landscape.

But industries are evolving rapidly due to the strong demand for digital transformation. With business and technology requirements changing dramatically every decade, educators must stay on top of the trends by addressing outdated processes and approaches in SE education.

Code review is a widely-used practice in software development companies to identify software defects. It’s therefore included in SE curricula taught at universities worldwide.

But teaching code review is still a challenging task because its ultimate effectiveness depends on the code reading and analytical skills of the reviewer.

‘Code review is critical for high-quality software engineering, especially in safety and security critical domains,” stated Professor John Grundy, Australian Laureate Fellow and awardee of the 2014 IEEE Technical Council on Software Engineering Distinguished Educator.

But code review is inadequately taught – or taught not at all – in many computer science courses, resulting in poor industrial practices.’

Figure 1: An overview of the code review process

Checklist-based code review can be a valuable way of learning for inexperienced students, since checklists encourage students to focus on important issues or defects during their evaluations.

A checklist is a set of questions related to potential issues or software defects that might occur in code reviews (refer to Figure 2).

Figure 2: An example of code review checklists from Google's Engineering Practices documentation

The checklist dilemma

Although checklist-based code review is an effective method to train students' reading skills, providing it to students may not facilitate the development of their analytical skills to anticipate potential issues. This is because learners may use the provided list without understanding and appreciating the key motivations behind checklist-based code review.

Additionally, current practices in industry generally don't provide code checklists. Instead, reviewers must depend on their own experience and domain knowledge to anticipate issues (ie. using their own mental checklists) when performing code reviews.

Having students develop checklists for themselves should stimulate the growth of their critical and analytical skills, allowing them to anticipate issues or defects in software artefacts that they will review. Yet, it’s unclear whether they will be able to predict potential issues given their limited programming experience and mistakes that learners might make when forecasting issues for code reviews. For example, students tend to focus on obvious code defects such as typos, formatting issues and coding style rather than code semantics which are equally important.

Embarking on innovative education

Dr Chong, Dr Tantithamthavorn and Dr Thongtanunam’s research investigated whether developing a checklist can be part of code review learning activities. This will help scaffold students’ skills and identify their common mistakes when developing a checklist.

It’s widely acknowledged that software engineering graduates often face difficulties when they start their career due to misalignment between the skills learned in university and what is needed in industry. To address this issue, we explored the missing link and uncovered how we can elevate software engineering.’ said Dr Chong.

The researchers conducted a longitudinal study to collect and analyse assignment submissions from students who enrolled in FIT2107: Software Quality and Testing unit offered at Monash University Australia and Malaysia in 2018 and 2019.

1,791 checklist questions from 394 students across the two selected cohorts were collected.

Encouraging results. An abundance of opportunities.

The researchers found that with basic knowledge of algorithms and programming, 76% of the checklist questions prepared by students were suitable to guide reviewers in finding defects. The results suggested that students tend to be able to anticipate potential issues and can develop good code review checklists, though they may make some mistakes in the process.

The first two common mistakes, such as unclear and irrelevant questions, partly stem from misconceptions about the quality specification of the code review checklist. Thus it’s recommended that educators make a clear specification of the checklist.

The other two common mistakes, testing-dependent and static-analysis-dependent, are partly due to misperceptions about the goal of the code review process. It’s therefore recommended that teachers emphasise the key role of code review in the SE process, as well as a clear distinction of software quality assurance activities.

Where to from here?

The study affirmed that code review checklists, which should stimulate students in developing their analytical skills for code reviews, can be an effective and innovative educational strategy to revolutionise how code reviews are conducted at universities worldwide to be more reflective of industry.

‘Code review is a common task that software engineers need to do when developing software, and it’s an important step for quality assurance. But doing code reviews well requires both expertise and experience. By practising developing a code checklist, students will get early feedback and correct any misconceptions, enabling them to do better code reviews.’ explained Dr Thongtanunam.