Evaluation/Program Impacts
Logic Model
21st Century Community Learning Center Evaluation/Activity Types
Special Topics/Outcomes
Outcomes
100
What are appropriate comparison groups?
Groups must have similar group participants and non-participants.
100
What is the logic model?
It is a diagram that explains the connections between program goals, implementation, and outcomes.
100
What was the CCLC Evaluation?
An evaluation called by the No Child Left Behind Act. Funding was to be determined through accountability measures. The evaluation was conducted by the Mathematic Policy Research. It was a poor evaluation that would have resulted in a 40% reduction in federal funding to many 22st CClCs which provide support to many families in need.
100
Are youth over-scheduled in organized out-of-school activities?
No, there are more kids doing nothing. Only 6% are over- scheduled and still they are better off academically than kids who are not scheduled.
100
What are three different types of mechanisms that explain why ASP participation might promote academic achievement?
1. Gains in academic knowledge EXAMPLE: Math team, French club 2. Psychosocial processes EXAMPLE: access to teachers, counselors; support of other peers and growth experiences that allow the student to take initiative in team work and leader ship (i.e. president of ASB, becoming involved in a sports team, becoming a tutor) 3. Reduction in problem behavior -- Avoiding the use of recreational drugs (can interfere with learning and promoting positive development) -- Give counseling and aid to those children with depression or suicidal thoughts (provide support groups or therapy)
200
Why is this important to have comparison groups?
A comparison group highlights program impacts and changes in its participants.
200
Why is the logic model used?
The logic model is used because it directs attention to the process, timing, and sequence of change which influences outcomes.
200
1.What was the overall most positive activity type based on the “secret santa from hell” activity? 2.What was the overall least positive activity type?
1.Faith-based youth groups with high identity, initiative, emotional regulation, teamwork, positive relations, and adult networks and low nothing. 2. Academic clubs with no highs but low identity, initiative, emotional regulation, teamwork, positive relations, and adult networks.
200
What are the conclusions of youth being overscheduled?
Most youth are not over scheduled. We need to worry more about a big chunk of kids who are not doing anything. Need to cut back on TV.
200
Does the research show ASP is beneficial to academic performance or not?
The bulk of studies in organized activities show academic benefit -- Even in well-controlled, longitudinal studies, organized activity participants tend to have: • Better academic performance: GPA and achievement tests • Higher educational attainment: i.e. it increases the likelihood that these children will go to college and continue education • Higher occupational/ career success: i.e. organized activities in the after school program can help give students the tools they need to achieve necessary skills such as leadership, teamwork, and time-management skills (as well as conflict resolution skills)
300
Why is evaluation important?
(MAS) Management, Accountability, and Sustainability.
300
Why the logic model important?
It is important because it determines a programs goals, features, participants, and sustainability.
300
What were the results from the 21st CCLC Evaluation?
It was found that participants of the program had higher risk of lower achievement in test scores and more behavior problems.
300
Do extracurricular activities seem to protect against substance use?
Extracurricular activities help lower the rate of substance use, however, some sports groups and activities can lead to substance use such as alcohol. -- Certain sports have a “party culture” that may sometimes encourage the use of certain substances
300
What are common predictors of anti-social behavior?
o Low supervision o Poor parenting o Socio-economic status o The presence of peers (i.e. deviancy training) o Unstructured activity (i.e. watching tv, hanging out) o Dangerous neighborhoods, peer rejection o The use of after school time
400
What is formative/process evaluation?
Formative evaluation is conducted to check if programs are delivering the services intended. Process evaluation is conducted to see which program's functioning needs improvement. Going back and forth between these two types of evaluations can ultimately determine if a program is proud and ready to be evaluated on its impact.
400
What are the multiple facets of participation?
ENROLLMENT (if child is registered to be in program), ATTENDANCE (intensity: how frequently child attends; duration: period of participation), ENGAGEMENT (level of interest, effort, and enjoyment), NUMBER OF ACTIVITIES (total number activities enrolled in or the total amount of time spent in activities), BREATH (diversity of activity types a child participates in).
400
What were the problems surrounding the 21st CCLC Evaluation?
Baseline data was not collected and one could not randomly assign students to be or not to be in the program. With no baseline or comparison group and there was crossover of comparison groups it is hard to measure program impact.
400
What are 3 propositions of the over-scheduling hypothesis?
1. Adults pressure youth to participate to achieve long term educational/career goals 2. Organized activities dominate free time which cuts back on time with family, school work, and relaxing 3. Youth are at-risk of poor adjustment, have low achievement, psychological distress, and poor relationships with parents.
400
What are some ecological conditions under which programs are more likely to be beneficial?
1. Chronosystem: Maturity of the program and evaluation of the program over time. EXAMPLE: how the program has progressed in its maturity (how long as it been around, has the program been able to evaluate what works/what doesn’t work, how successful is the program in promoting positive development) 2. Mesosystem: neighborhood that the child lives in can affect the quality and development of the program and its benefits towards the child- (low SES schools may not have access to the same resources as other school especially if the ASP is school-based) -- Poor quality due to lack of resources can limit benefits
500
What is summative/outcome evaluation?
This evaluation looks at the program impacts. It helps determine whether a program is achieving its intended goals. It is an assessment of short term, long term, beyond academics, and impacts beyond the program.
500
Can you discuss the relations between the logic model elements?
Program goals and intended outcomes must be clear and measurable because stakeholders such as parents, participants, and funders want to know if the program is worthwhile. Program goals are also influenced by what the stakeholders want out of the program because they provide the resources for the program. These are interconnected because a program must have goals that are important to stakeholders to provide resources which will give the program time and sustainability. As a result program features must have a purpose to match goals and budget. It program goals prove to be of good quality program participants will increase.
500
What can we learn from the 21st CCLC Evaluation?
1. One study is not enough 2. Study must pass peer review scrutiny 3. The must be collaboration between evaluators and technical group 4. Scientist and evaluators must articulate the strengths and weaknesses of the study 5. Time and timing is critical, program needs to be proud before impact evaluation takes place.
500
Does research support or reject the 3 propositions of the over-scheduling hypothesis?
Reject 1. The main reason reported by youth is that they participate for enjoyment and excitement. 2. It does not take up free time because youth spend more time watching TV. 3. Only 6% at risk of poor adjustment, there are plenty more youth who do not do anything which is worse because organized activities are positively associated with well being.
500
What are some challenges to program development when considering various stakeholders and their perspective on target goals and outcomes for a program?
The problem is that each stakeholder has a different ideal of what a program should look like -- Remember class activity: police officer, administration (policy maker), teacher, student, parents • Student- wants an after school environment where they can have fun and enjoy time outside of school • Parent- wants children to be safe during the after school hours while making sure that their child continues to learn and finish assignments before coming home • Policy maker- wants to be able to provide the after school program with enough resources such as tutors, without straining the budget too much. EXAMPLE: hiring student/college volunteers to help tutor kids so that the schools don’t have to pay for more help (efficient ways for child to get help while conserving budget)
M
e
n
u