Latrobe Valley Better Health Project: Evaluation of the Injury Prevention Program 1992-1996
Monash University Accident Research Centre - Report #114 - 1997
Authors: L. Day, J. Ozanne-Smith, E. Cassell & A. McGrath
Full report in .pdf format [5.3MB]
Evidence for the effectiveness of the community based approach to "all age all injury prevention" applied in the Australian context is limited. This study's aim was to evaluate the Latrobe Valley Better Health Injury Prevention Program, a community based intervention in south-east Victoria, Australia.
The evaluation design was quasi-experimental including pre- and post-intervention observations in a population of approximately 75,000. There was no single comparison community, rather comparative data was used where possible. Process measures included key informant interviews with local organisation representatives. Impact evaluation relied mainly on self-reported changes in injury risk and protective factors, gathered by a random telephone survey. Outcome evaluation was based on five years of emergency department injury surveillance data for the Latrobe Valley. Modelling of injury rate data was performed using both Poisson and logistic regression.
The program built strategic partnerships, increasing the emphasis on safety at the local level. Promotional and educational activities were implemented in the targeted areas of home, sport, and playground injuries, and alcohol misuse among the youth. Some 51,000 educational contacts were made with the community and 7470 resource items distributed. There was a 7.3% increase in the proportion of households purchasing home safety items (p=0.55). Unsafe equipment was replaced and undersurfacing upgraded in municipal playgrounds. The demand for and availability of protective equipment for sport increased. The age standardised rate per 100,000 persons for emergency department presentations for all targeted injury fell from 6594 in the first program year to 4821 in 1995/96 (p=0.017). There were significant decreases in the presentation rates for home injuries among all age groups except for those 65 years and over, playground injuries among 5-14, 15-24 and 25-64 year olds, and sport injury among 15-24 year olds only. The direct program cost per injury prevented was $272.
There were no decreases in alcohol purchases by liquor outlets, and the rate of arrest for being drunk and disorderly increased among 10-24 year olds. However, significant reductions were observed for assaults among 10-24 year olds compared to those over 25 years.
Most program objectives were met to some extent. The lack of a comparison community and injury data limits the conclusions which can be drawn about the association between the program and the injury reductions observed. However, the reductions were associated to some extent with changes in injury risk and protective factors and were greatest for the injury issues subjected to the most intense activity. There is merit in monitoring the impact of the ongoing injury prevention program, only with improvements to the evaluation design including a comparison community.
The application of the community based approach to "all age all injury prevention" has been applied increasingly in various parts of the world, following the first successful reports in Sweden during the early 1980's. Controlled evaluations of these programs have identified varying degrees of success with respect to the intended health outcome ie., frequency and severity of injury. Significant decreases in injury relative to comparison communities have been reported in Sweden and Norway. In Australia, the only controlled evaluation of an all age all injury prevention program demonstrated a significant effect on self reported injury only, compared with the comparison community after six years. This document reports on the evaluation of the first four years of the Latrobe Valley Better Health (LVBH) Injury Prevention Program, a community based program, modelled on the Swedish experience, in the Latrobe Valley, Victoria.
The aim of the LVBH Injury Prevention Program was to utilise a community intervention approach to prevent injuries, reduce hazards and increase public awareness of measures to reduce the incidence and severity of injuries in the Latrobe Valley community (population 75,500 - 76,560). The program focussed on four main areas of activity: home, sports and playground injury prevention, and alcohol misuse among the youth.
The evaluation is a quasi-experimental uncontrolled design which includes pre and post intervention observations. There is no single comparison community in which a range of equivalent measures have been recorded due to budgetary constraints. However, where possible, comparative data from a range of sources has been included. The current evaluation covers the period from May 1992 to June 1996. The following recognised phases of evaluation are included: process, impact and outcome.
Data for process evaluation was obtained from program reports, the program officer's diaries and media file, and through interviews and discussions with key individuals from other local organisations.
The major method utilised for the impact evaluation was random household telephone surveying. Two surveys of 1.5% of the total population were conducted in April 1992 and April 1995, in conjunction with the Centre for Health, Education and Social Sciences (Monash University, Gippsland). The questions were designed to determine changes in knowledge, attitudes and practices and were modelled on questionnaires developed for the evaluation of a similar community-based injury prevention program. Differences between responses in the pre and post-intervention surveys were tested using the chi-square function in Excel.
Program records, and data from other organisations, including VicRoads, the Liquor Licensing Commission, were also used in the impact evaluation.
Outcome evaluation was intended to include three sources of injury data: self-reported injury, emergency department presentations and hospital admissions. However, hospital admissions data presented significant methodological difficulties and was therefore not used. This had a significant disadvantage for the evaluation, since the hospital admissions data provided the opportunity for comparison of injury trends with other similar areas of Victoria. The other sources of injury data did not provide a similar opportunity.
Self reported injury data was gathered in the random household telephone surveys. Injury rates for the two week period were calculated using the total population surveyed as the denominator.
Emergency department presentations data for five years (1991/92 - 1995/96) was obtained through the Victorian Injury Surveillance System (VISS) operating at both campuses of the Latrobe Regional Hospital, the only public hospital within the relatively well defined geographic area of the Latrobe Valley. VISS captures high known proportions of patients who present with an acute injury to the emergency department. There were no emergency department surveillance data for a comparable time period available for any other Victorian non-metropolitan region which could be used for comparison.
Data for targeted and untargeted injury categories were extracted by age group, for those residing in program post code areas. The untargeted injury category served as a type of comparison since it encompasses injury not targeted by the program or by other sectors (such as transport and occupational injury). Injury frequencies were adjusted for annual capture rates at each campus of the hospital. Injury rates per 100,000 population were calculated using the Australian Bureau of Statistics resident population estimates and were age standardised by the direct method to the Victorian population.
Two other sources of outcome data were used to evaluate the program component relating to alcohol misuse among youth. Data for fatal and serious casualty crashes among those up to 25 years of age for the program region and comparison regions in rural Victoria were extracted from the VicRoads Crash Database for the years 1984-1995. The data used were limited to high alcohol hours ie., periods during which fatal and serious casualty crashes are 7 times more likely to involve blood alcohol contents exceeding 0.05%. Drunk and disorderly arrests by age, sex and postcode of residence were collected by the Victorian police regional headquarters.
Trends in emergency department presentation rates, motor vehicle crash rates, and drunk and disorderly arrest rates were modelled using established regression techniques. Program effect was assumed to be cumulative in nature, based on Swedish theory of community based injury prevention.
In all the statistical analyses, p values less than 0.05 were considered to be significant. In this evaluation, therefore, a statistically significant result has only a 5% probability of occurring by chance alone. The outcome of a statistical test is not in itself definitive, and should be considered with all other relevant information.
Cost per injury prevented was calculated from the total direct program costs and an estimate of the number of injuries prevented, assuming that the injury rate would have remained constant at the pre-program level, in the absence of the program. The difference between the total expected injury frequency and the actual injury frequency was reduced by 7.9%, to account for the background injury reduction observed in non-targeted injury.
A range of promotional, educational and awareness raising activities were undertaken in the home, playground and sports injury areas. There were at least 46,000 educational contacts with the community on the subject of home injury prevention of which 1% were structured educational sessions. In addition, home injury prevention education sessions for professional groups were attended by 170 people. There were more than 6000 resource items distributed and consistent media exposure was obtained. Strategic educational sessions were undertaken on the subject of playground safety with the Latrobe Valley Primary Principals' Association and tertiary education students. Injury prevention materials have been incorporated into the local football coaching courses and trainers programs, which reached approximately 365 coaches and trainers. A major sports sponsorship promoting sports safety was secured.
These activities appeared to result in an increase in community awareness, with significant changes in some knowledge measures. A significantly higher proportion of respondents in the post intervention survey was able to list safety features of their home (65.9%, 79.3% p<0.05), however, there was no significant difference in the numbers of features reported per household (1.9, 2.2 p=0.83). Seventy-two percent of respondents in the post-intervention survey reported knowing where to purchase safety items, compared with 62% in the pre-intervention survey (p<0.01). This improved knowledge did not appear to have been translated into action, with similar proportions of respondents reporting that safety items for the family or house had been purchased in the previous 12 months (42.7%, 45.8%, p=0.55).
Unsafe playground equipment was removed and playground undersurfacing was upgraded. In the former City of Traralgon, one new playground per year has been constructed and a total of 6-7 pieces of hazardous equipment removed. Undersurfacing was replaced if necessary twice each year, and more frequently in heavily used areas. In the former City of Moe, 5 new playgrounds were built, and new equipment was installed in 3-4 playgrounds . The former City of Morwell implemented its already existing strategy which required a playground to be located within 500 metres of every household. Since the formation of the new Shire, over $100,000 has been expended on tan bark undersurfacing for municipal playgrounds, and a maintenance crew conducts monthly playground audits using tick box check lists.
The use of safety devices and equipment was increased for some sporting activities, particularly football. The demand for helmet use in junior football has increased, as has the availability of mouth guards.
Injury reductions were observed in both self-reported injury and emergency department presentation data. There was a non significant decrease in the rate of self reported injuries recorded in the telephone surveys (62.7/1000 persons, 48.2/1000 persons, p=0.19). The age standardised rate per 100,000 persons for emergency department presentations for all targeted unintentional injury fell from 6594 in the first programmed year to 4821 in the last evaluation year. This decrease was significant in the Poisson regression analysis (p=0.017), in contrast to a non-significant decrease observed for non-targeted injury (p=0.08). In the logistic regression analysis, the decreasing trend in all targeted unintentional injury was significantly different to that observed for untargeted unintentional injury, among all age groups tested (0-4, 5-14, 15-24, 25-64, 65+ years).
There were decreases in emergency department presentations for each of the targeted unintentional injury categories, which were significant for:
- home injuries among all age groups except those 65 years and over
- playground injuries among 5-14, 15-24, and 25-64 year olds
- sport injury among 15-24 year olds only
The reductions were of borderline significance for playground equipment injury emergency department presentations.
In the area of alcohol misuse among youth, working parties were established in collaboration with the Police Community Consultative Committees, through which a range of initiatives were undertaken, these included increasing the acceptance and availability of non/low alcohol alternatives with promotions at 20 major events, responsible serving of alcohol courses in which 35 staff from 10 local venues participated, participation in a code of conduct by all licensed premises in the area, and changes in local legislation prohibiting consumption of alcohol in central business districts.
The impact of these activities on alcohol misuse was difficult to determine with the measures used. Litres of regular alcohol purchased by retail liquor outlets in the program region decreased marginally, while light alcohol purchases increased by 16.5%. In comparison, in country Victoria, purchases of both regular and light alcohol products have decreased by 4.8%. There were non significant increases in the rates of arrest for being drunk and disorderly for both the 15-24 year age group (targeted with alcohol misuse programs) and the over 25 year age group. There were significant decreases in the age group crude rates of motor vehicle crashes during high alcohol times in the both the program and comparison regions of Victoria. In both regions, the decrease occurred during the pre program years.
There were however, significant reductions in intentional injury emergency department presentations for the 10-24 year age group, in contrast with a non significant decrease for those over 25 years.
The direct program costs per injury saved were $272. This does not take into account the considerable in kind contribution of personnel and resources from local organisations. Nor does it include the unpaid additional hours worked by the program officer. There was insufficient data for pre-program years to determine the likely injury rate trend if the program had not been implemented. The estimate was made assuming that the rate of injury would have remained constant at the 1991/92 level in the absence of the program. The number of injuries prevented would therefore have been underestimated if the trend in targeted injuries were increasing, and over-estimated if the trend in targeted injuries was decreasing more rapidly in the absence of the program than that for untargeted injuries, which is unlikely.
Overall the program achieved the aims of utilising a community intervention approach, developing collaborative relationships with local organisations and achieving some structural and organisational changes which had the potential to provide cumulative benefit. There is some evidence that components of the program have been incorporated into the activities of other local organisations, with the program itself ultimately becoming part of the new local government structure.
Evaluation design and methods: Despite some limitations, mainly relating to the absence of a comparison community and comparable injury data, the evaluation design and methods have been strengthened by a number of features including the use of process, impact and outcome measures, the combination of qualitative and quantitative methods, and a relatively small likelihood of contamination from neighbouring programs and activities. One of the strongest features of the evaluation is the prospective injury surveillance system operated at the Latrobe Regional Hospital by the Victorian Injury Surveillance System (VISS). The data collection method remained unchanged for the five year period and regular audits identified the capture rates for both campuses of the hospital, allowing adjustment for variations in capture. The coding system used by VISS allowed disaggregation of the data into the same categories targeted by the program. Age standardisation of injury rates adjusted for population changes during the course of the program, an important feature since the Latrobe Valley has undergone some characteristic demographic changes during the program period.
The emergence of comparable results from two statistical approaches each making slightly different assumptions about the emergency department data indicates a degree of robustness in the finding of injury reductions.
Program objectives and injury reductions: The majority of program objectives, that were evaluated, were met to some extent and there were examples where the program objectives were expanded as opportunities arose. It should be noted however, that none of the stated program objectives extended to include quantitative targets. Therefore, while there is evidence that most objectives were met, there is no measure of the extent to which the objectives were met and some objectives were met for a limited period of time.
Injury reductions were observed in both self-reported injury and emergency department presentation data for all of the three targeted unintentional injury categories. Changes in patient presentation patterns or capture rates are unlikely to account for the decreases, which appear to be real.
The important question for the Latrobe Valley Better Health Injury Prevention Program then, and for injury prevention more generally, is whether all or part of these reductions can be attributed to the program. In the absence of a comparison region, two other avenues can be explored. These are firstly changes in risk or protective factors which could have led to the injury reductions, and secondly changes in non-targeted injury.
There is some evidence to suggest an association between program activities and the observed reductions in targeted unintentional injury emergency department presentations. Awareness of household safety features increased, as did knowledge about where to purchase safety items. There was no evidence though for any increases in knowledge about how to improve home safety, numbers of safety features per house, or the proportion of households having purchased safety items in the preceding 12 months. Consequently, if the significant reductions in home injury emergency department presentations are attributable to the program, the mechanism was behavioural, rather than environmental, modification.
The significant reduction in playground injury emergency department presentations was associated with the reduction of hazardous equipment in municipal playgrounds and improved maintenance schedules, factors which can be attributed to the program, at least in part. Reduced exposure could account for a lowered playground injury rate. However, in most instances the hazardous equipment was replaced, and in two former municipalities the amount of playground equipment actually increased. Reduced exposure could therefore only explain the injury reductions if there was a decrease in playground use by the community, which is unlikely.
Changes in risk and protective factors for sports injury are more difficult to detect. There was some evidence for a change in the culture of sports clubs, with an increased emphasis on safety being incorporated. The use of protective equipment, particularly in junior football, appeared to increase. Although the reduction in sport injury emergency department presentations was statistically significant for 15 - 24 year olds only, there is some evidence that the reduction could be attributed at least in part to program activities.
The conclusion that the injury reductions are associated, at least in part, with program activities is supported by three additional observations:
- the reductions were significant in both statistical models in the two program areas in which activities were implemented for a longer period ie., home and playground
- the reductions were incremental from the first program year for home and playground injuries
- a small insignificant reduction was observed for non-targeted injury
It remains, however, that the lack of comparative injury data constrains the strength of the conclusions which can be drawn about the association of the program with the reduction in less severe injuries. A state wide or regional downward trend in emergency department presentation rates may have been occurring at the same time.
An association between program activities and any decrease in alcohol misuse by young people is more difficult to determine, due in part to imperfect measures. There was no significant reduction in serious injury motor vehicle crashes in high alcohol hours in the program region compared to other rural areas of Victoria. This may be more due to changes in drinking and driving patterns among young people across Victoria, than due to changes in alcohol misuse. There was, however, a marked reduction in the rate of intentional injury by others among 10-24 year olds, compared to little change in the rate for those over 25 years. This is an encouraging result, although it may be due to factors other than a decrease in alcohol misuse.
Future directions: The program has been re-named La Trobe Safe Communities and incorporated into the local government structure, bringing new challenges and opportunities. There would be considerable merit in building on the increased levels of awareness, knowledge and training by increasing the emphasis on environmental, legislative and policy changes, particularly in the home and sports injury prevention components. The incorporation of the program into local government provides the opportunity to examine all aspects of local government operations with a view to incorporating safety into routine activities and services.
Given the encouraging results of the evaluation to date, there is also merit in monitoring the impact of the ongoing injury prevention program, only with improvements to the evaluation design, as there would be little additional benefit in continuing with the current design. These should include a comparison community in which changes in injury risk and protective factors are measured and retrospective and prospective injury data is available.
The continued availability of high quality emergency department injury surveillance data from the Latrobe Regional Hospital is critical, not only for continued evaluation, but also for program implementation. Local injury data has been a powerful motivating tool for the program, and a cornerstone of the evaluation.
Institutionalisation of program components and activities into local government, and into other relevant local organisations, should be monitored and evaluated.
It would be highly desirable to extend the cost benefit analysis of the Latrobe Valley program, using more sophisticated methods building on other current MUARC research. Regardless of the size of injury reductions achievable with this approach, the investment in such programs should be justifiable at least in terms of financial returns. Such a study would provide significant results for the field of community based injury prevention nationally and internationally.
Further evaluations of the type reported here will provide reducing returns. The limitations of the quasi-experimental design, even if a comparison community is included, are significant. Future developments should focus on controlled trials of multiple communities randomly assigned to treatment and control groups, consisting of more than one community. The financial and logistical considerations of such a trial would be substantial and challenging.
Sponsors: Victorian Health Promotion Foundation & Public Health Research and Development Committee, NHMRC