Assessing Short-Term Learning and Long-Term Impacts of Non-Formal Education Programs

Non-formal education programs, such as summer camps, offer opportunities for immersive learning and increased experiential science. Summer camps at the Chesapeake Bay National Estuarine Research Reserve in Virginia provide hands-on marine science experiences for 1st–8th grade students. This article assesses the short-term learning gains and long-term impacts of attending a marine science summer camp. Across all age groups, there is evidence of short-term learning, high numbers of new experiences for campers, and increased scientific confidence in many campers. The longitudinal study revealed that these impacts may last until at minimum college, influencing career choice, extracurricular activities, and course enrollment. INTRODUCTION Non-formal education is the use of organized educational activities outside of the established formal education system. Non-formal education typically includes programs in museums, youth facilities, after-school programs, and camps with specific learning goals (Luxembourg 2013). This type of education plays a valuable role in the Science, Technology, Engineering, and Math (STEM) field, affording students educational experiences for hands-on, collaborative learning, exploring their interests outside of a highly evaluated environment (Levay, Volmert, and Kendall-Taylor 2018). These experiences offer access to authentic learning opportunities, where participants obtain a deeper understanding of a topic by making it relevant through “real life” connections Sea Squirts students go birding in the teaching marsh. Courtesy of CBNERR Education Staff


INTRODUCTION
Non-formal education is the use of organized educational activities outside of the established formal education system. Non-formal education typically includes programs in museums, youth facilities, after-school programs, and camps with specific learning goals (Luxembourg 2013). This type of education plays a valuable role in the Science, Technology, Engineering, and Math (STEM) field, affording students educational experiences for hands-on, collaborative learning, exploring their interests outside of a highly evaluated environment (Levay, Volmert, and Kendall-Taylor 2018). These experiences offer access to authentic learning opportunities, where participants obtain a deeper understanding of a topic by making it relevant through "real life" connections Sea Squirts students go birding in the teaching marsh. Courtesy of CBNERR Education Staff (Roberts et al. 2018). The benefits are not limited to growth within the STEM field. Schools that combined formal and non-formal education components to connect the students with their surrounding environment reported better performance across subjects, reductions in behavioral issues in the classroom, increased engagement, and greater pride in accomplishments among their students (Lieberman and Hoody 1998). Participation in these programs increases scientific literacy and general interest in science, as well as strengthens skills in answering scientific questions (Dann and Schroeder 2015). Additionally, students are more likely to participate in stewardship actions, share what they have learned after participation, and consider careers in STEM after participating in non-formal education programs (Dann and Schroeder 2015;Foster and Shiel-Rolle 2011). The most effective type of non-formal experiences are those that engage students in active, hands-on learning over time (Marian and Jackson 2017).

ASSESSING LEARNING IN NON-FORMAL EDUCATION
The assessment process in formal education is more recognized than non-formal (Norland 2005). Historically, assessments in non-formal education were difficult to conduct due to low organizational capacity and a focus solely on visitor satisfaction (Norland 2005). Now, the goal of non-formal education assessment has shifted to better understanding learning and outcomes (Norland 2005). One of the ways to assess learning in these non-formal environments is through comparing pre-and post-test scores. The students answer content questions, as well as eliciting their attitudes and beliefs using Likert scale items (Dann and Schroeder 2015;Birinci Konur, Seyihoglu, Sezen, and Tekbiyik 2011).
Within environmental education, a discipline frequently conducted through non-formal education programs, connectedness to nature is an important variable to measure in determining impact. Connectedness to nature has a strong relationship to environmentally responsible behavior and can be used as a proxy for behavior in assessments (Frantz and Mayer 2014). To assess connectedness to nature and other attitude/belief changes, it may be necessary to adopt other assessment types such as journal entries, pre-and post-drawings, focus groups, and semistructured interviews (Brain and Tingey 2015;Birinci Konur et al. 2011;Christensen, Nielsen, Rogers, and Volkov 2005).

STUDY SITE
The Virginia Institute of Marine Science (VIMS), School of Marine Science of the College of William & Mary, is a multifaceted research and teaching facility that serves the Commonwealth of Virginia, and is among the largest marine research and education centers in the United States. VIMS has a three-part mission to conduct research in coastal ocean and estuarine science, educate students and citizens, and provide advisory service to policy makers, industry, and the public. VIMS provides these services to Virginia, the nation, and the world.
The Chesapeake Bay National Estuarine Research Reserve (CBNERR) is one of 29 protected areas that make up the National Estuarine Research Reserve System, established to promote informed management of the Nation's estuaries. A critical aspect of the Reserve's mission is to enhance public awareness and understanding of estuarine areas and provide suitable opportunities for public education and interpretation. For the past 10 years, CBNERR has been providing weeklong summer camps offered at VIMS for students entering grades 1-8. These free, hands-on camps enable students to learn about the Chesapeake Bay, and are tailored to meet the learning stages of each age group. Camps cover themes such as wetlands, environmental stewardship, general outdoor exploration, and marine careers.

HYPOTHESIS
The objectives of this study were to 1.) assess short-term learning through age-appropriate assessment techniques, and 2.) analyze long-term impacts of camp attendance through a longitudinal study. We hypothesized that attending a marine science summer camp would have positive effects on both short-term learning across all age groups and long-term impacts such as declaring an undergraduate major and extracurricular activities. Chesapeake Champions students test different cleanup methods in a simulated oil spill. Courtesy of Mallory Munden

Short-Term Learning
Content knowledge for Sea Squirts, a camp for rising 1st and 2nd grade students, was evaluated using five-question oral interviews. Interviews included one question covering a major theme for each day, and scores were assigned on a zero to two-point scale using an assessment map. The map helped to determine scores by looking for predetermined key words within the oral responses that were established by the two evaluators, and translated the depth of an answer to a numeric score. A score of zero indicated little to no understanding of the topic, a score of one indicated moderate understanding, and a score of two indicated complete or nearly complete understanding. Following the oral interviews, written assessments for attitudes and previous experiences were administered. Attitudes were assessed using a three-smiley scale, where campers circled if they felt positively, negatively, or impartial toward the statement ( Figure 1). Three smiley faces were used to ensure a clear choice between agreement, disagreement, and no feelings toward the statement, with a negative smiley face for negative feeling, a straight face for impartial feeling, and a smiley face for positive feeling. Previous experiences were evaluated using "yes"/"no" responses. Lastly, Sea Squirts campers participated in a picture assessment where they were given a prompt to "draw what you think the Chesapeake Bay looks like" on the first, and again on the last day of camp. Each drawing was scored by a single evaluator using a presence/absence score in 4 categories: plants (number of species), animals (number of species), people, and science. In the case of unclear depictions in the drawings, individual campers were asked for clarification. The sample size for Sea Squirts was was 16 (1st and 2nd grade) students.
In both Bay Buddies camps (3rd and 4th grade) and the one Chesapeake Champions camp (5th and 6th grade), content knowledge was evaluated using multiple choice and short answer questions. The Bay Buddies test consisted of six multiple choice and two short response questions, and included topics such as the importance of the Chesapeake Bay, animal biology, marine debris, water quality, marshes, and watersheds. The Chesapeake Champions test consisted of eight multiple choice and one short response question, and included topics such as the scientific method, climate change, conservation, animal biology, marine debris, water quality, and watersheds. In addition to content questions, attitudes were assessed on a standard Likert scale, and previous experiences were evaluated using "yes"/"no" responses. The sample sizes were 23 campers for Bay Buddies and 24 campers for Chesapeake Champions.
Content knowledge in Estuary Explorers (7th and 8th grade) was evaluated using four multiple choice and three short answer questions covering biology, water quality, wetland habitats, and climate change. The sample size was 16 campers. For this age group, assessments were conducted using electronic survey software due to time constraints for collection and analysis.
For all age groups, one test was administered upon arrival on the first day of camp, prior to any instruction (pre-test). A second test was administered four days later (post-test) on the final day of camp. Campers were not informed of scores on either assessment (assessments were kept anonymous through the use of pseudonyms the students selected on the first day). Students used the same pseudonym on each test to allow for growth comparisons in scores. Assessments were checked both before and after administration to ensure that all material was covered during the week and that they reflected the most important information.

Longitudinal Study
Another aspect of CBNERR's Camp Assessment was to conduct a longitudinal study of previous participants of the program. For the past eight years, similar camps have been offered and, to prepare for the end of project funding, we wanted to assess the impact of the camp several years after participation. An electronic survey was sent via email to 246 previous campers who now ranged from 9th grade to college students. The survey consisted of 10 questions, including information such as name and camps attended, undergraduate major (if applicable), current extracurricular involvement, extent of continued involvement with VIMS and CBNERR, and statements of impact through an openended response. Response to the survey was optional but encouraged using entry to a prize raffle and camp alumni stickers.

RESULTS AND DISCUSSION
Data for each age group was analyzed in three parts: content, new experiences, and attitudes. To analyze shortterm learning of scientific content, we calculated the average percent change between the pre-and post-test scores and performed a t-test to compare the averages with a significance criterion of 0.05. New experiences were analyzed by determining the percentage of campers in each age group that had at least one new experience over the course of the camp. For each camp, we also determined the most common new experience, and how many campers had not previously participated in that activity. For attitude changes, we performed a t-test to compare changes in average scores for each attitude item with a significance criterion of 0.05. Specifically, for the Sea Squirts camp, we also calculated changes in the categories (animals, plants, science, and human uses) included or removed from the pre-and post-camp drawings of the Chesapeake Bay. For the Estuary Explorers camp, we also analyzed changes in responses to intended behavior questions, such as whether or not the camper would recycle, using a t-test with a significance criterion of 0.05.
There was a statistically significant difference between the pre-and post-tests for each camp (p<0.001) ( Table 1). Across all camps, the overall average percent change in assessment scores was 107.8%. The camps for younger elementary students, Sea Squirts, and Bay Buddies had an average percent change of 77.6%. The camps for upper elementary and middle school students, Chesapeake Champions, and Estuary Explorers had an average percent change of 28.8%. Sea Squirts, Bay Buddies 1, Chesapeake Champions, and Estuary Explorers all had a large effect size, meaning that the difference between pre and post-test means is large and easily perceptible. Bay Buddies 2 had a medium effect size, meaning that the difference between pre-and post-test means is likely to be discerned without careful statistical analysis (Table 2).
For Sea Squirts, no statistically significant changes in drawings were observed. However, 25% of campers added one or more distinct categories to their second drawing, while 12.5% removed one or more categories in their second drawing. While this type of assessment was an interesting concept for evaluating younger learners, it proved difficult in practice due to the subjectivity of evaluating the drawings and creating the rubric, as well as the possibility that drawings reflected only the most recent camp-related experiences, and not the complete conceptual understanding of the Bay. Due to these challenges, we did not continue the drawing assessments in subsequent years.
All five camps provided a new experience to over two-thirds of the campers, with seining as the most common for Sea Squirts, buoy building and mapping for Bay Buddies, animal dissection for Chesapeake Champions, and trawling for Estuary Explorers ( Chesapeake Bay and interest in marine science significantly decreased (p= 0.02, p= 0.03). Since the children in this age group were just of age to begin school, they were likely learning about the Chesapeake Bay from a scientific perspective for the first time. Decreases in interest in marine science may have resulted from more clearly understanding the field of marine science and realizing that it was not what they were interested in. The decrease in feelings of personal impact suggests that this age group may have felt overwhelmed in how younger students are able to participate in helping the Bay, suggesting that we should place more emphasis on teaching practical solutions that feel manageable to younger children for future Sea Squirts camps. No positive, statistically significant changes in attitude were observed for the Sea Squirts camp. For Bay Buddies-Session One, campers were significantly more likely to feel confident in their marine science knowledge, that science was a part of their identity, and that they would consider a career in science. For Bay Buddies-Session Two, campers were significantly more likely to feel that their actions impacted the Chesapeake Bay. In the Chesapeake Champions camp, participants were significantly more likely to feel confident in their marine science knowledge (p<0.001) and that they were capable of answering scientific questions well (p<0.01). In the Estuary Explorers camp, there was a significant difference in students feeling that science was part of them (p= 0.03) and that they were interested in a career in science (p= 0.02). Estuary Explorers campers were also significantly more confident in their ability as scientists (p<0.001) and to answer scientific questions after camp (p<0.01). The Estuary Explorers camp exposes campers to scientists and their research, while the campers conduct their own research project making these results consistent with the goals of the camp. There was no significant difference in intended behavior before and after camp (p>0.05). The responses were high on both the preand post-tests, suggesting that the campers were already partaking in environmentally responsible actions and their importance before camp.

Longitudinal Study
The longitudinal study received 46 responses for a response rate of 19.11%. Of the seven respondents that are currently in college, 75% (N=6) are pursuing scientific majors. Of the 39 respondents currently in high school, 71.05% (N=27) are planning to pursue scientific majors with the pre-medical track being the most common ( Figure 2). When asked to state something that they remembered learning at camp, 74% (N=34) of respondents recalled a concrete fact such as "an adult oyster filters 50 gallons of water per day," while 22% (N=10) remembered learning a skill such as how to test water quality, and 2% (N=2) recalled a personal development such as learning to present in front of peers. Of the 46 respondents, the majority remain active with CBNERR through public outreach events (N=30), and volunteering as Junior Counselors for the camps that they once attended (N=12). Out of 46 responses, 17 participants reported not remaining active with CBNERR after camp. A follow-up question revealed that the most common reason for inactivity is distance from CBNERR. Qualitative data including scientific extracurricular activities and the impact of camp in the words of each camper was acquired and added to a database. The following is an example of a statement of impact from a previous camper: It made a subject that I found to be fun by itself even more real and accessible by PEOPLE, not just textbook figures or stoic lab coat clad robots. Summer camp made science very tangible, and when the week was over, it only made me want to be immersed into that environment more.
Common responses included "encouraging interest in science," "changing career path toward science," and a sense of "responsibility to the environment."

CONCLUSIONS
The assessment data suggest evidence of short-term learning, as every camp experienced a significant increase in scores. However, short-term attitudinal impacts will require more study due to both positive and negative changes observed. We encountered no obvious issues with the assessment instruments and found them to work well for our study, especially after excluding questions affecting the validity and reliability of the study such as using "all of the above" as an answer choice. In particular, the Sea Squirts assessment method appeared particularly effective in assessing content through interviews and interpreting answers using the assessment map, as well as using a smiley face scale to evaluate attitudes. One downfall of the interview format was the comfort level of campers in responding to counselors they had just met, which may have impacted test scores. Although the changes observed in the drawing assessments were not significant, they were still observable and of interest for further study. If conducted over subsequent camp years, increasingly clear patterns may arise.
The longitudinal study provides solid evidence for longterm positive impacts of attending a marine science summer camp. In 2016, the National Center for Education Statistics (NCES) stated that approximately 8.6% of college students are majoring in a scientific field, while our camp alumni sample had 75% in a scientific field (NCES 2016). We recognize, however, that this was a small sample size of students who had self-selected into a marine science summer program and would be strengthened with greater numbers of participants. In 2012, the National Science Foundation found that 39.2% of incoming college freshmen are intending to major in science fields, while our high school aged camp alumni had 71.1% planning to major in these fields (NSF 2014). These higher percentages could suggest that participation in a science camp influences the desire of students to pursue science majors. Since the majority of respondents recalled a hard fact, these data also suggest that learning in camp is long term, and remains with campers long after the program is complete. This information is useful for planning programs as it allows instructors to target key facts with the intention of engraining them. Overall, non-formal education programs similar to the CBNERR summer camps are valuable to both in increasing short-term learning about the environment, as well as provide long-term impacts on the student's future.