Collecting data
You need to collect data on your measure (or measures) in order to answer your question. Data collection can be resource-intensive, so choose your data sources and collection strategy wisely. Ask the following questions (adapted from page 53 of the Principal Investigator’s Guide) to guide you through the process:
What kind of information do you need? Do you need to know about opinions or attitudes? Knowledge levels? Details about implementation?
What is the best method to obtain that information? You have many choices for data collection—surveys, focus groups, interviews, and more.
What is the best strategy for collecting that information via that method for this project? From whom do you need to collect information? Where is the best place to reach them via the method(s) you’ve chosen? When should this happen? If there is a facilitator, whom should that be?
DATA COLLECTION EXAMPLES
How many questions participants get right in a Jeopardy! game at the end of the program
A staff member uses an observation form to record the number of times participants form collaborative groups
A staff member writes field notes based on the interactions they observe
Quantitative or qualitative
To put it simply, quantitative data means numbers, and qualitative data means anything else—usually text or spoken words, but could also include things like images, sounds, or behaviors. Traditional library assessments tend to be quantitative in nature, like the number of items borrowed or the number of program participants. Collecting qualitative data by listening to or observing people can better capture individual experiences, thoughts, and attitudes. Your assessment may include both—a survey can have both quantitative and qualitative questions, and an observation protocol may involve both counting the number of times something happens and listening to someone’s comments. You can also quantitatively analyze qualitative data (see the section on analyzing databelow).
Data collection instruments
An instrument is a tool you use in the collection process—for example, a survey, interview guide, or observation record. You can find examples of instruments for informal learning in the Additional Resources for this section. However, connected learning programs are often unique, with one-of-a-kind elements that may need one-of-a-kind assessments. You may wish to modify someone else’s instrument to fit your own assessment needs, or even create one that is entirely original.
Testing the process
Before diving headlong into data collection, test the entire process. For a small and simple assessment, this could simply be asking a co-worker to review your plan. A larger effort should be tested more thoroughly, ideally with a sample that resembles who or what you will be assessing. A large survey, for example, would benefit from being tested with a handful of teens first. As you conduct your tests, consider the following questions:
Does the instrument you are using give you the kind of data you need to answer your question? Do you need to make changes to wording, format, time or place?
Are there ways you can automate or simplify data collection? For instance, you could provide computers at the library for people to fill out a survey online, rather than handing them a sheet of paper to fill out .
Do you have the capacity to handle the data collection you’re planning? If an interview takes much longer than expected, for example, you may need to either shorten your interview instrument, add more staff to the effort, or increase the amount of time your assessment will run.
Do you need to collect any baseline data so you can make before and after comparisons?
Is there a way that data collection can happen seamlessly (for instance, the teens provide you with the data you need as part of the program), and not intrusive (such as taking time from the program to have the teens fill out a feedback form)?
PRIVACY AND CONSENT
Be sure to respect your participants’ privacy and autonomy as you are collecting data (and any time outside stakeholders are involved). This is particularly important when you are interacting with minors. Project Outcome is a good starting place for learning about issues of privacy and consent in library assessment (requires free registration).
Ways to collect data
The following table includes some—though certainly not all!—methods to collect data for your assessment. Each method has potential strengths and weaknesses and can be used to assess a wide variety of factors. Choose one (or more) that fits your question, your program, and your capacity.
ANALYZING ARTIFACTS
You can gain a great deal of insight from the items learners create through your initiatives, whether they are musical performances, poetry, or 3-D models. Knowledge, skill, development, attitudes, behaviors—these can all be observed in learners’ creations. You can also ask students to write about the learning experience or their creative process, and analyze their responses.
Good for:
Demonstrating achievement first-hand
Potential drawbacks:
There can be a high level of subjectivity; be careful not to read too much into a creation (particularly a non-textual one) without input from the creator
People may not like being watched
COLLECTING PASSIVE FEEDBACK
Along with on-demand surveys and questionnaires, you can collect passive feedback through methods such as comment cards.
Good for:
Collecting data unobtrusively
Potential drawbacks:
Responses may come only from people with extreme or passionate opinions—not from people “in the middle”
MAKING OBSERVATIONS
Using an observation guide or protocol, you can collect data by simply observing what is happening during a program or in a space and taking detailed “field notes.” This can be done unobtrusively in the background, or in a more engaged manner. If observing an individual directly (perhaps as they use a resource or do an activity) you can ask them to use a “think-aloud” strategy to help you understand why they do what they do.
Good for:
Uncovering unexpected outcomes
Directly observing what is happening, not just relying on participants’ self-reports
Potential drawbacks:
People often behave differently when they know they are being observed
Effective observations require a great deal of attention and focus
People may not like being watched
TESTING KNOWLEDGE OR SKILLS
Formal tests of knowledge or skills feel too much like school for a connected learning library setting, but you can find small, creative ways to “test” a teen’s progress without making it feel like an exam, such as integrating a competitive quiz towards the end of a program or presenting challenges for participants to work on.
Good for:
Directly assessing what a teen knows or can do
Potential drawbacks:
They can feel too formal and school-like
They can feel like too much pressure, especially for youth who are hanging out or messing around
Other methods are described in different modules; surveys and focus groups are covered in the Community Mapping module, and the Capacity module discusses interviews and town halls.
Last updated