This week on the podcast, we are going to talk about a problem you may not even know you have. Does your organization collect survey data? Many nonprofits conduct surveys in one form or another. This could be as complex as a program evaluation, pre- and post participant questionnaire, or it can be a simple satisfaction survey.
Regardless of how complex a tool you may use, they all serve the same purpose- surveys provide valuable feedback for nonprofits. Internally, they help you make decisions, they help you determine whether your programs are having the impact you intend to have and whether stakeholders are satisfied with their engagement. Externally, survey data demonstrates the difference your nonprofit makes in the community and helps stakeholders, such as funders, decide whether they want to engage with your nonprofit or not.
Given the importance of this data then, we need to ensure that the surveys you design and how you conduct them facilitates the collection of accurate data. So, the problem we’ll be tackling today is, how can you and your nonprofit design and conduct effective surveys?
Sneak Peek at the Episode…
⦿ [2:36] Even though it seems like we answer questions in a split second, as it turns out, the process of answering questions is a bit more involved… Rather, research tells us this process is broken down into four steps: interpreting the question, generating an opinion, formatting the response, and editing.
⦿ [4:37] This is where the trouble with surveys can occur. If your surveys have these variations, then the data you are collecting and using for important decisions is not accurate.
⦿ [7:43] One way to solve for this is to randomly sort the questions on your survey tool such that every respondent gets a random order of questions.
⦿ [12:16] And so if your survey respondents all select answers on the extreme ends of the scales, that is going to skew your data. So how can you prevent this from happening?
⦿ [15:24] If you think about something as simple as even a satisfaction survey, if you ask respondents to identify their gender or their race, there are stereotypes that may be associated with these, and thus they may answer according to these stereotypes and you won’t get an accurate picture of their feedback.
⦿ [18:20] While I did mention some ways to combat each specific survey effect, there are broad strokes we can take to improve surveys which will help us avoid these survey effects as well.
⦿ [21:17] Vague language leaves room for interpretation and that means each respondent could interpret the question differently which results in variation in responses.
Put it into Practice
⦿ What survey response effects could your surveys and evaluations be susceptible to?
⦿ Where do you use vague language that can be more specific?
⦿ Who might serve on a focus group for your program evaluations?
Rate, Review, and Follow
Are you subscribed to our podcast? If you’re not, I invite you to do that today. I’m adding a bunch of new, information-packed training episodes into the mix and I don’t want you to miss any of it! Click here to subscribe in Apple Podcasts!
And, if you got a lot of value out of this episode, I would be so grateful for a review on Apple Podcasts. Reviews help others find the Nonprofit Science Podcast and helps me support others just like you. Simply click here, open the show up in Apple Podcasts, select “Ratings and Reviews” and “Write a Review” to let me know what your favorite part of this podcast is. Thank you so much!
Links Mentioned in this Episode:
⦿ Article: Designing a questionnaire
⦿ Article: The impact of administration mode on response effects in survey measurement
⦿ Article: A comparison of question order effects on item-by-item and grid formats: visual layout matters
⦿ Article: Response order effects in the Youth Tobacco Survey: Results of a split-ballot experiment
⦿ Article: Choir of believers? Experimental and longitudinal evidence on survey participation, response bias, and public service motivation
⦿ Episode 21: How can we get more women in nonprofit leadership positions
