Community Based Injury Prevention Evaluation Report - Shire of Bulla SafeLiving Program
Monash University Accident Research Centre - Report #66 - 1994
Authors: J. Ozanne-Smith, J. Sherrard, I. Brumen & P. Vulcan
Full report in .pdf format [6.3MB]
This report documents the evaluation of the Shire of Bulla Safe Living Program at the end of the first three years of implementation. The Program is a community based, all age, all injury prevention program which commenced in 1991 in an outer metropolitan Melbourne municipality with a population of approximately 39,400. It aims to demonstrate the effectiveness of the community intervention approach in preventing injuries, reducing hazards, and increasing public awareness of injury prevention by comparison with a demographically matched Shire and the State of Victoria. Program strategies were targeted particularly at high risk groups including children and seniors in the home, at school, and on the road.
Process evaluation for Program reach and community participation indicates that 45% of the population was aware of the Program and that approximately 6.4% of the adult community received some education and training in safety and injury prevention. The Program implemented a wide range of interventions in all settings. Reach of the Program was moderate. Usage of traffic education in the Shire of Bulla schools was greater than the comparison community. Additional process measures are documented.
Impact evaluation for changes in injury risk factors indicates a reduction in exposure to hazards, an increase in the use of protective items, an increase in the sales of safety products, and environmental changes. Community knowledge and awareness of safety and injury prevention were raised. Additional impact measures are documented.
Outcome evaluation for changes in the incidence and severity of injury indicates no statistically significant evidence for an effect of the Program on overall injury rates or hospital bed days in the Shire of Bulla. However, one sub category of injury, motorcycle casualty hospitalisations, decreased significantly. Injury reported by residents in the telephone survey also decreased significantly. Several issues relating to the outcome evaluation are documented.
Recommendations include an increase in Program reach and community participation, introduction of new interventions to address high frequency high severity injury, institutionalisation of successful interventions, and continuation of outcome measures.
This report documents the evaluation of the Shire of Bulla Safe Living Program at the end of the first three years of implementation. It should be read in conjunction with the Program report 'The First Three Years: Final Report of the First Three Years of the Shire of Bulla's Safe Living Program (Hennessy, Harvey & Arnold, 1994), which describes the Program structure and implementation.
The Shire of Bulla, located 25 km north west of Melbourne, had an estimated population of approximately 39,400 in 1991, growing rapidly to approximately 43,900 in 1993. The Program is a community based, all age, all injury prevention program, which commenced in 1991, and is based on the Swedish model pioneered in Falkoping. It aims to demonstrate the effectiveness of the community intervention approach in preventing injuries, reducing hazards, and increasing public awareness of injury prevention. Program strategies have been targeted particularly at high risk groups including children and seniors in the home, at school, and on the road. Overall, 113 strategy activities were developed covering all ages and all injury settings.
The objectives for the Program are:
- to increase community awareness of injury prevention and create a "safer community" environment.
- to develop strategies which will provide an overall reduction in the number and severity of injuries within the Shire of Bulla.
- to reduce the number of hospital bed days as the result of injuries.
- to reduce the frequency and severity of injuries requiring hospital attendance or admission.
- to reduce the incidence of accidental deaths.
- to reduce hazards.
- to increase the use of safety devices and equipment.
A demographically matched shire, the Shire of Melton, was used as a comparison for evaluation measures.
The results of the evaluation of the Shire of Bulla Safe Living Program are presented in three sections, which reflect the development and maturation of the Program. The first section, process evaluation, deals with measures of Program reach and community participation. The second deals with measures of impact of the Program and includes measures of changes in hazards, environment, and community knowledge attitude and behaviour. The third section deals with the outcome measures of change in injury morbidity and mortality.
Process evaluation - Measures of program reach and community participation
The promotion of the Safe Living Program in local newspapers achieved the Program target of an overall average level of 1 item/week in weekly papers at no cost. The participation rate of volunteers in the working groups which were responsible for a substantial role in the planning and implementation of the Program was 0.5% (151) of the community aged 20 years and over. About 300 volunteers overall (1% of the adult population 20 years and over) were involved in the implementation of the program.
Approximately 6.4% of the adult community participated in Program seminars and courses in injury prevention and control. The participation rate for seniors was approximately 15%. The usage of the Traffic Safety Education Programs by schools in the Shire of Bulla was greater than for the Shire of Melton.
Outlets for the sale of safety products increased for the township of Sunbury in the Shire of Bulla but not for the other centres in the Shire. Successful applications for sports grants from the Victorian Health Promotion Foundation were achieved. The seniors activities guide achieved a good reach of 58% to the Shire residents aged over 50 years, though its relevance to injury prevention may be marginal. The Family Safety Guide was hand delivered to all urban homes in the Shire.
The pre- and post-intervention telephone surveys which measured Program effects showed a moderate level of increases in awareness in several areas. These included recall of the borne safety package delivered to all homes (28%), awareness of the Safe Living Program (45%) and receipt of Safe Living News (27%). Measurement of the level of participation in community activities or meetings related to safety indicated no statistically significant change when compared with the Shire of Melton.
The Safe Living Program has contributed to raising awareness of injury prevention at regional, state and national levels. Reference should be made to the companion volume to this report, which describes the application for YMO accreditation and networking activities with other communities by means of hosting and participating in Safe Communities national meetings. [Editorial note: WHO Safe Community status was conferred in 1994].
Additional process measures are documented.
The Shire of Bulla Safe Living Program achieved increased levels of sales of children's safety seats and restraints, sales of smoke detectors, usage of the Early Childhood Injury Prevention Program, wearing of helmets, and training in child safety as a result of Train the Trainers Courses.
The Shire of Bulla Council response to advocacy for traffic engineering, recommendations from the road audit, and Safe Routes to Schools initiatives indicate a high level of commitment to action. Council also introduced footpath cycling, and supported a safety audit of school playgrounds as a result of advocacy by the Safe Living Program. Overall, the schools implemented more than 50% of the recommendations of the playground safety audit.
An increase in the level of knowledge of the Safe Living Program, of the purchase location of safety items, and of injury prevention in the household were achieved. Change in behaviour, as a direct result of the increase in knowledge and awareness of the Program, are difficult to assess.
The Program successfully promoted (at no cost to the Program) the construction by a local builder of a Safe Living House incorporating safe design features.
Additional impact measures are documented.
Coroner's data indicated a total of 17 injury deaths in each of the Shires of Bulla and Melton in the two years of available data, 1989/90-1990/91. The numbers of deaths were too small to allow meaningful comparisons.
Analysis of hospital admissions data from the Victorian Inpatient Minimum Database (VAM) shows no significant difference in all injury rates between the Shires of Bulla and Melton from 1987/88 to 1992/93. Hospital admissions in categories relating to motor vehicles, cyclists, pedestrians, collisions in sport, and falls in seniors and children were unchanged. Length of stay in hospital, a measure of injury severity, was also unchanged for both Shires.
Vic Roads data indicates a reduction in motor cyclist injury in the Shire of Bulla compared with the Shire of Melton, but no significant changes for vehicle occupants, pedestrians or bicyclists.
Analysis of the Victorian Injury Surveillance System emergency department presentations for children shows no significant difference in frequency of injury between the Shires.
Occupational injury decreased substantially for the Shire of Bulla Council employees but no data was available from other local government for comparison.
In summary, there is evidence to indicate achievement of four of the objectives of the Program. These are, increased community awareness of injury prevention, development of injury prevention strategies, hazard reduction, and increased use of safety devices and equipment. Furthermore, the telephone survey found a considerable reduction in the number of respondents who reported sustaining an injury in the previous two weeks for the Shire of Bulla relative to Melton. However, there is little evidence so far, of achievement for the three Program objectives relating to injury reduction. These objectives include overall reduction in injury requiring hospital attendance or admission and a reduction in bed days. A difference of approximately 20% would have been required to demonstrate a statistically significant change in hospital admissions.
Comparison with the limited literature on studies describing the process of implementation of community based programs indicates that the level of publicity for the Shire of Bulla Program could be improved and other methods investigated to determine alternative approaches for promotion.
The exposure of the Shire of Bulla community to education and training in injury prevention was 6.4% for adults. This was less than some reported studies for prevention generally, suggesting that an increase in this exposure would be advantageous.
Community volunteers participated in Program planning and implementation, at varying levels, some on a long term basis and some for specific very short term strategies only, Comparison with the literature is limited as detail of this nature appears to be rarely documented.
While a large number of measures to reduce hazards have been implemented, it seems that there may be scope for increasing coverage of the community, particularly if resources are available. This may be possible by targeting some of the more important hazards and setting objectives to reach a higher proportion of the total population.
Comparison with other programs is limited by their lack of data relating to process and impact thereby reducing useful comparison of outcome data.
The number of deaths remained the same in both the Shires of Bulla and Melton and further years of data are needed to establish whether the program has had an effect on the death rate. By the middle of 1993, effectively 2 and a half years after commencement of the Program, the Shire of Bulla hospital admission rate for injury had not decreased nor had the length of stay for injury hospitalisations. In Falkoping, bed days reduced after 3 years of the Program indicating a reduction in injury severity. An additional complexity relates to the Shire of Bulla having a substantially lower admission rate for injury than the Shire of Melton before the Program commenced. This difference may pose a problem in demonstrating any significant reduction in injury rate due to the phenomenon of regression to the mean.
The overall rate and the frequency of emergency department presentations for a number of categories of injury targeted by the Program have not changed significantly in relation to injury in the Shire of Melton.
It should be noted that the Shire of Bulla had the lowest rate of road casualties of all the Shires in the region prior to commencement of the Program. Some encouraging results were found with both motorcycle casualties and telephone reported injury reduction in the Shire of Bulla relative to the Shire of Melton. The telephone survey result is the only source of low severity injury data, (in the absence of general practice injury presentation data and school injury data), which limits the ability to measure Program effects of less severe (but more frequent) injury, and which also reduces the statistical power overall to detect a significant effect.
Any demonstrable effect on injury incidence and severity may depend on outcome measures conducted over a longer time. Therefore the true nature of the Safe Living Program effects can only be more fully assessed by extension of outcome evaluation over several years.
The formal evaluation of the Safe Living Program, three years after its commencement should not necessarily be seen as a final report. It provides a valuable opportunity to revise and improve the Program. The Shire Council has indicated its commitment to the Program by providing two further years of funding and including in its Corporate Plan a number of objectives for Safe Living activities.
1. Increase program reach and community participation
2. Continue to introduce new interventions focusing on high frequency, high severity injury issues
3. Continue successful interventions and institutionalise these where appropriate
4. Investigate funding to continue process and outcome evaluation for three further years
5. Develop formal implementation guidelines for other municipalities, and determine a distribution policy.
Sponsors: Victorian Health Promotion Foundation and VicRoads