- Printer-friendly version
- Login or register to post comments
Evaluation Plan
Program evaluation will be integrated into WKU's Quality Enhancement Plan[52], American Democracy Project[53], and American Humanics [54] programs, and consistent with the approaches advocated by the NSF[55][56][57]. The evaluation will consist primarily of two components, one focusing on curriculum and the other focusing on the results/outcomes of the Water and Wastewater Technician Training Institute (WTTI).
Evaluation of Curriculum: Content Validity Study
The curriculum of the WTTI is planned to consist of a combination of academic and technical education. The curriculum is intended to provide graduates with the competencies to meet industry expectations as operationalized through the certification standards of relevant state agencies and industry groups as well as the management skills to support the technical skills.
The curriculum, including the on-line modules, virtual laboratories, and hands-on experiences in labs and partnering operating plants, will be subjected to content validation utilizing industry standards for certification as the criterion. Subject Matter Experts (SMEs) identified through the Steering Committee industry representatives from certifying agencies and trade associations will play a critical role in the validation of the curriculum content. The content validation study is vital formative evaluation that will provide data to identify and inform any program content modifications.
Criterion Measures. Certification standards of relevant state agencies and industry groups will be analyzed to extract the underlying knowledge, skills, and abilities (KSAs) required to meet the standards. Some of the requirements may be extracted at the task level. KSAs essential to effective performance on the job that are not included in certification standards may also be included. The criterion analyses will be conducted by the evaluation team and verified through SME review.
Formative Evaluation of Program Content. The curriculum, including the on-line modules, virtual laboratories, and hands-on experiences in labs and partnering operating plants will be subjected to a content analysis to determine which KSAs are taught and the degree of emphasis given to each component of the curriculum.
A) Matrix Linking Industry Criterion Measures with Program Content. Criterion information (i.e., the X axis indicating the importance of KSAs for effective job performance and industry certification) and program information (i.e., the Y axis indicating the degree of emphasis of KSAs in the WWTTI) will be used to create a content matching matrix that will be used to link the content of the program to the KSAs needed for certification and success in the industry. This curriculum map will enable the determination of how well the curriculum matches the criteria and the identification of critical areas where the curriculum may be deficient and areas where there may be an excess in the curriculum.
B) 4-Year Degree Entry Requirements. A similar analysis will be conducted using the university associate degree requirements and the entrance requirements for a 4-year degree (e.g., requirements for entry in to the 4-year degree in Technology Management) as the criterion. The results from this analysis will help to ensure the curriculum meets university requirements for the associate degree and for admission into the (2+2) 4-year degree program.
The formative evaluation of the curriculum, including content analysis and validation, will provide important data to identify and inform any program content modifications. The content validity study of the curriculum will largely take place during the first year of the performance period as the curriculum content is developed and implemented. However, to ensure the content validity of the program over time, the curriculum will continue to be monitored as additions and modifications are made. Establishing that the curriculum content meets both industry standards and academic standards will help to ensure that WTTI graduates are able to effectively enter either the job market or continue their education to pursue a 4-year degree.
Evaluation of WTTI Program Outcomes Implementation Evaluation
Once the WTTI begins recruiting and admitting students, it is important to monitor the project to determine if the institute is being implemented as planned. Process measures will be used to assess whether various components of the project are operating effectively. Results of this implementation evaluation will provide data for determining if/what modifications need to be made to program components. Data will be collected and analyzed to address the recruitment, enrollment, and retention program components.
Recruitment. The target audience for recruitment into the WTTI consists of high school students and incumbent operators seeking career advancement potential. Data will be collected to determine the number of recruitment presentations in terms of the number of recipients and the number of schools or other entities visited. We will track the number of inquiries about the program (e.g., inquiries made on-line, by phone, or by mail) and the number of applicants to the program.
Enrollment and Retention. While recruitment numbers are important, the more critical statistics are the number of enrollees and graduates. Enrollee data tracked will include the number of enrollees, who they are (i.e., in terms of demographics such as age, race, gender, etc.), where they come from (e.g., in-state, the region of the state, out-of-state, etc.), how they learned about the program (e.g., a specific recruitment channel, word of mouth, industry practitioner, etc.), and why they enrolled in the program (e.g., to earn an associates degree, the 2+2 program, new career, career advancement, etc.). Student data such as these will enable the refinement of recruitment methods and ensure WTTI is reaching the target audience for the program. We will also track retention rate as it is vital to the success of the program that students persist through graduation. Discriminate analysis differentiating those who persist from those who drop out (e.g., utilizing data such as that identified above) should provide information to help address retention, if low retention rates prove to be a problem.
Year 3 Summative Evaluation
The third year of the performance period should witness the first graduating class of full-time students and either their entry into the job market or their transition to the 4-year program. While establishing the content validity of the WTTI curriculum is an important component of the program evaluation, it does not ensure graduates learned the material in the program, that graduates found employment, or that graduates were able to transfer that knowledge to perform effectively on the job. Data from outcome measures collected for the summative evaluation will enable a comparison of program outcomes to program objectives to determine the impact of the fully implemented program in terms of the quality of the outcome and whether or not the stated objectives have been achieved. Summative evaluation will address the following.
Graduation. Graduation data will be monitored to determine the graduation rate and whether enrollees are graduating within the projected time frame for full-time and part-time students, respectively.
Employment or Enrollment Upon Graduation. WTTI graduates will be tracked to determine what percentage find employment in the industry within 3-6 months of graduation or enter a 4-year program upon earning the associate degree.
Assessing WWTTI Program Effectiveness. An on-line survey instrument will be developed and administered to graduates of WTTI (upon graduation and again after 12-18 months on the job) to determine their perceptions of the effectiveness of WTTI. A similar survey will be developed and administered to employers of WTTI graduates and to industry constituents represented on the Standing Committees. The survey instrument will address issues such as the effectiveness of recruiting efforts, program curriculum, pathways to employment or 2+2 academic programs, employment opportunities, retention on the job, opportunities for career advancement, and the mentoring program. Data from graduates can be compared to data from employers and industry constituents to determine if there is agreement on these issues. Insight can sometimes be gained from examining any differences in perceptions of program effectiveness. This survey (or another data collection method) will also assess whether graduates are able to pass certification exams and are able to perform successfully on the job, and whether WTTI is effectively providing sufficient graduates to meet industry need for technicians.