|
||||||||||||||
Expert Network : Home Safety Literacy
|
||||||||||||||
|
Fire Safety Literacy Project Evaluation Dr. David Leitner and Pacific Research Associates conducted the formative evaluation of the Fire Safety Literacy Project for the Home Safety Council and its partners ProLiteracy Worldwide and Oklahoma State University’s Fire Protection Publications. The Evaluation Report was completed in February 2005. A field test of the Fire Safety Literacy Project was implemented in seven locations by teams of literacy programs and local fire departments. The results of this field test are being used to revise program materials and delivery for distribution to literacy organizations and fire departments interested in replicating or adapting the project. Pacific Research Associates administered surveys and conducted interviews with participants at the pilot sites, including literacy program managers, teachers/tutors, and adult learners in literacy programs and project coordinators in fire departments.
Read about field test participants and their training. Teams of representatives from seven literacy programs and local fire departments conducted the field test. These pilot sites represent a broad geographic range and demographic diversity and include communities served by both career and volunteer fire departments. The fire service and literacy providers attended a two-day training in September 2004 at the National Fire Training Academy in Emmitsburg MD. The program developers from the Home Safety Council and ProLiteracy Worldwide led the training. They presented the program materials, including the Orientation Guide, Teacher’s Manual, readers, activities, and pictographs to the providers and showed them how the materials could be used with adult learners. In addition, the trainers presented information about fire safety, literacy in America, fire departments, and key fire safety messages from the project. The trainers set aside some time for the fire service and literacy provider teams to develop a plan to implement the pilot project in their local communities.
At the training, the evaluator from Pacific Research Associates reviewed the data collection instruments and answered the questions that fire service and literacy providers had about the evaluation. The evaluator also administered a survey to collect feedback from participants about the training.
To conduct the evaluation, Pacific Research developed a series of surveys and interview protocols for literacy program managers, teachers/tutors, adult learners, and fire department project coordinators. The evaluation involved both quantitative and qualitative analysis. Read about data analysis of the Project. The data analyses involved both quantitative and qualitative analysis. Pacific Research used descriptive statistics (e.g., means and frequencies) to summarize the quantitative data from the Likert scale-type items on many of the surveys. Open-ended questions from the surveys and interviews required qualitative analysis that entailed two steps. Data reduction. Survey responses were coded and data were classified based on categories that were developed before, during, and after the data collection phase. Drawing and verifying conclusions. Patterns in the data were used to generate tentative conclusions and then to confirm, refute, or refine these conclusions. Data from the quantitative and qualitative analyses are integrated in the Results section of this report to facilitate a more complete understanding of the perceptions of literacy program managers, teachers/tutors, adult learners, and fire departments project coordinators about the program.
The results of the findings include the reactions of those who participated in the pilot test and completed a survey or participated in an interview. Read selections from the evaluation results. The findings about general project implementation are based on surveys completed by literacy program managers, teachers/tutors, and fire department project coordinators. Most of the responses to the surveys are fairly positive, e.g., tended to be on the high end of a five-point scale where “1” meant “strongly disagree” and “5” meant “strongly agree.” When identifying strengths and areas of concern, the evaluator typically focused on those ratings that were relatively high or relatively low when compared to the ratings on a particular survey as a whole. The evaluation identified three general comments about project implementation. These general comments are followed by the specific results from the surveys. Many teachers/tutors did not think they had enough time to implement the readers and accompanying materials as they would have wanted to implement them. The program goals are viewed as important and clear and the project materials do a good job of clearly identifying the fire safety messages for students.
The pilot projects served a wider range of students than those for which it appears the project materials were best suited or easily adapted. Project materials are not a good match for this wide range of students. For students in Basic Literacy, teachers/tutors think project materials are best suited for students in beginning basic literacy and low intermediate literacy. For students studying English as a Second Language (ESL), teachers/tutors think project materials are best suited for students defined as low intermediate and high intermediate ESL.
Program Goals Literacy program managers rated the goals of the program as important, giving it a rating of 5.0 on a five-point scale, but gave a little lower rating to the statement that program goals are clear at 4.3.
Fire department project coordinators also rated the goals of the program as important and clear, giving both statements a rating of 4.9.
Fire Safety Messages Literacy program managers thought the materials clearly identify all the fire safety messages, giving it a rating of 4.4 on a five-point scale. Teachers/tutors reported that the materials contain clear and easily identifiable fire safety messages for students, giving it a rating of 4.5.
Fire department project coordinators thought the materials contain easily identifiable fire safety messages for students, giving it a rating of 4.7.
Project Materials Support for Organization Goals The end of project surveys asked literacy program managers and fire department project coordinators whether they were able to incorporate project materials into their respective programs without compromising their organization’s mission, either literacy or fire education. Literacy program managers thought they were able to incorporate pilot project materials into their literacy program without compromising the overall goal of improved literacy, giving it a rating of 4.5.
Fire department project coordinators also thought they were able to incorporate pilot project materials into the students’ literacy program without compromising the accuracy of the fire safety content, giving it a rating of 4.9.
Improved Understanding of Literacy and Fire Safety The end of project surveys asked literacy program managers and fire department project coordinators whether they have a better understanding of either fire education or literacy in America based on their participation in the program. Literacy program managers gave a rating of 5.0 to the statement of having a better understanding of fire safety and the fire safety problem in the United States.
Fire department project coordinators gave mean ratings of 4.4 to the statements of having a better understanding of literacy problem in United States and being able to reach a population that has been difficult to reach previously. However, project coordinators gave a relatively lower rating (4.0) to whether they have a better understanding of how to teach students with lower literacy skills about fire safety.
Future Participation Literacy program managers reported, based on their experiences in the pilot project, that they would be interested in continuing their participation in the project, giving the statement a rating of 4.6.
Fire department project coordinators, based on their experiences in the pilot project, also indicated they would be interested in continuing their participation in the project, giving it a rating of 4.4. However, while six of the seven projects “agreed somewhat” or “strongly agreed,” one project “disagreed somewhat.”
Time to implement Project Activities
Teachers/tutors gave a relatively low rating to the statement that they had enough time to implement all the activities in the pilot project, giving it a rating of 3.3.
Project Integration Teachers/tutors were asked to describe how they implemented the project according to a set of possible choices. Table 4 shows the number of teachers who reported implementing the possible choices. The largest numbers of teachers either set aside a block of solid time or set aside a portion of each lesson to work on the project. Table 4. How teachers/tutors integrated the Fire Safety Literacy Project
Reading Level of Students Literacy program managers gave their lowest ratings on the End of Project Survey to the match between the project materials/activities and the literacy level of the adult learners who participated in the project.
Responses from teachers/tutors support the results from the literacy program managers on the match between project materials and activities with the literacy level of the adult learners. Teachers/tutors were asked to describe the reading levels of the students who participated in the project and the reading levels of the students for whom the materials would work best.
The results show that projects served a wide range of Basic Literacy and ESL students. Perhaps more importantly, the teachers/tutors appear to suggest that the materials are probably better suited for a narrower range of students than the project served in the pilot test, especially for ESL students. For students in Basic Literacy, teachers/tutors think project materials are best suited for students in beginning basic literacy and low intermediate literacy. For students studying English as a Second Language, teachers/tutors think project materials are best suited for students defined as low intermediate and high intermediate ESL.
Fire Department Project Coordinator End of Project Survey The Fire Department Project Coordinator End of Project Survey contained five questions about the activities the Fire Department conducted for the literacy program, including classroom presentations, fire station visits, smoke alarm installations, and any “other” activities. Read the survey findings. Table 7 presents the mean ratings to three questions for three activities. The mean ratings are followed by the comments that literacy program managers gave to two other questions for the three activities. Table 7. Mean ratings of Fire Department coordinators about Fire Department activities.
Overall, the Fire Department project coordinators rated the fire station visits as easier to arrange than classroom presentations. In addition, project coordinators thought that each activity was an effective way for the Fire Department to convey fire safety messages and had no trouble working with teachers/tutors on any of the three activities.
Classroom Presentation. 1. The thing I liked most about this ACTIVITY was: The fire department project coordinators wrote five comments to the question about what they liked about classroom presentations. All five comments were different. Individual coordinators reported they enjoyed teaming with their literacy partners and that the activity was a learning experience for all involved. 2. The thing that posed the greatest challenge with this ACTIVITY was:
The greatest challenge to implementing classroom presentations was the issue of language. Two project coordinators reported that they were unable to communicate in Spanish and needed a translator or a bilingual staff member.
Fire Station Visit. 1. The thing I liked most about this ACTIVITY was: The two project coordinators who reported implementing this activity gave two different responses. One liked bringing students into the fire station as a way to introduce them to the community. The other coordinator liked meeting new people and helping them. 2. The thing that posed the greatest challenge with this ACTIVITY was:
The greatest challenge to implementing fire station visits was finding the time to devote to the project.
“Other” Activity. Five fire department project coordinators reported conducting “other” activities. The other activities included installation of smoke alarms (see below), fire extinguisher training, smoke/fire trailer demonstration, Christmas Party, safety day at the training academy, and a fire safety presentation at a tutor training session. 1. The thing I liked most about this ACTIVITY was: Four of the five project coordinators wrote comments to the question about what they liked about the “other” activity. All four comments focused on the enthusiasm and interaction of the adults with the trainers, especially for hands-on demonstrations. 2. The thing that posed the greatest challenge with this ACTIVITY was: The five project coordinators wrote four different comments to the question. They included arranging schedules, communicating to ESL students, and obtaining the commitment from adult learners to attend an afternoon Sunday event. Smoke Alarm Installation. The fire department project coordinators who reported installing smoke alarms installed a total of 113 smoke alarms in 73 homes. When compared to other ratings by fire department project coordinators, fire departments did not find it easy to arrange and implement smoke alarm installations with teachers/tutors and adult students, giving it a rating of 3.3 on the five point scale. However, the three project coordinators thought the adult learners were open to having them in their homes, giving it a rating of 4.2. 1. The thing I liked most about this ACTIVITY was: The thing they liked most about installing smoke alarms was that two project coordinators reported that they felt good that they were able to properly install high quality smoke alarms that would protect the people in the house. 2. The thing that posed the greatest challenge with this ACTIVITY was:
The greatest challenge to installing smoke alarms was communication because of language differences.
Fire Department Project Coordinator End of Project Interview The fire department Project Coordinator End of Project Interview contained two questions about the activities conducted by the fire department. 1. What was the role of the fire department in the Fire Safety Literacy Project? In which activities did the fire department participate? In which activities did fire department participation work well and why? In which activities did fire department participation not work well and why? Classroom Presentations. All seven project coordinators reported that the fire department participated in some kind of fire safety presentation. All seven project coordinators reported that the class presentations were effective. Three project coordinators thought it was effective because they were able to answer specific questions from adults. Two project coordinators thought it was effective because the presentations were informal. One project coordinator thought it was effective because presenters knew their role in the presentation. Fire Station Tours. Four project coordinators managers reported that the fire department conducted a Fire Station Tour and all four thought it was effective. Two project coordinators thought it was effective because the fire department staff answered the specific questions of the adult learners. 2. Did the fire department install smoke alarms as part of the Fire Safety Literacy Project? If so, what challenges, if any, did the fire department encounter when installing the smoke alarms with this population?
Four of the seven fire department project coordinators reported installing some smoke alarms. Three of the four reported that the main challenge was communication with the adult learners because of language differences.
ADULT LITERACY STUDENTS KNOWLEDGE ABOUT FIRE SAFETY AND WILLINGNESS TO CHANGE
The evaluation asked teachers/tutors as well as the adult learners on the surveys and during interviews what adult learners had learned about fire safety from the project. Read more about these findings All the pilot project sites also administered a fire safety knowledge assessment at the beginning of the program to determine what adult students knew about fire safety prior to participating in the program. The evaluation identified three general comments about what adult learners learned and what they knew about fire safety. Many teachers/tutors and adult learners thought adult students learned important fire safety messages from their participation in the program. Perhaps the most important thing adult students learned was what to do in case of a fire, such as following an escape plan, believing it to be especially important in terms of protecting their children and family in a fire.
The evaluation obtained valuable information to develop a valid and reliable Adult Student Fire Safety Assessment that can be used in a summative evaluation to measure the degree of fire safety knowledge gain for adult students who participate in the Fire Safety Literacy Project.
What Adult Students Learned About Fire Safety
The results about what students learned about fire safety are based on adult learner and teacher/tutor surveys on the two readers and the End of Project Interviews with adult learners. Read what adult students learned from the Fire Safety Literacy Project The interview asked adult learners what they had learned about fire safety. 1. What, if any, are some of the new lessons you learned about fire safety? What is the most important lesson that you learned? All eight adults interviewed reported that they had learned something from their participation in the pilot project. Five adult learners reported that they had learned how to make an escape plan, three adult learners stated they learned to make sure that they have a working smoke alarm, and two adults emphasized the importance of not overloading electrical sockets. Other interesting responses mentioned by an adult include posting the location of children on the outside of the house and not installing burglar bars.
The eight adults agreed about what was the most important thing they learned. Five adults reported that knowing what to do in case of a fire by following an escape plan was the most important thing they had learned. Three of adults linked this statement to protecting their children and family in a fire.
Student Fire Safety Knowledge Assessment The results about what students know about fire safety are based on the Student Fire Safety Assessment instrument administered to adult learners at the beginning of the project. The primary purpose of the student knowledge assessment was to develop a valid and reliable assessment that could be used to assess adult student knowledge gain in a summative study of the project after the formative phase of the evaluation is completed. Table 8 shows the results by item and according to the language in which the assessment was administered: English, Spanish, and Creole. Overall, the results show that 186 adult students had 53.6 percent of the items correct. In addition, the total percent of the items correct for tests administered in English and Spanish are virtually the same at 56 percent, while the percent correct for tests in Creole is substantially below the average at 36 percent. One possible reason for the lower percent correct in Creole is that the pilot project staff, who had the assessment translated to Creole, reported it was difficult to obtain a good translation.
The item analyses also show which items may be too easy or too difficult for students. These items may not provide a good measure of the fire safety knowledge gains that students would make as a result of participation in the program. Those items that were answered correctly by less than 30 percent and or more than 70 percent of the students will most likely be revised before a final version of the Student Fire Safety Knowledge Assessment is administered in a summative evaluation of the project.
FSLP EVALUATION: CONCLUSIONS For the most part, all project participants think that the Fire Safety Literacy Project is a good program and conveys important fire safety messages to adult learners. Several key themes emerged from the results, including: THE FIRE SAFETY LITERACY PROJECT IS RELEVANT: Almost all project participants think the program is important to adult learners and covers important fire safety topics. ADULT STUDENTS SEEMED TO LEARN IMPORTANT SAFETY INFORMATION. Almost all project participants believe that adult learners learned important safety information.
In conclusion, Pacific Research Associates believe that the Formative Evaluation of the Fire Safety Literacy Project provides the national partners with valuable information to revise the program.
Project Materials The Fire Safety Literacy Project materials, available in early summer 2005, will reflect the findings of the Formative Evaluation. Learn more about the Project materials. Download a pdf of the Pacific Research Associates Formative Evaluation Report. |
|
