Hello folks,
Were putting together a series of questions the experience of joining and participating in the SC community, particularly the challenges around communication, community guidance and availability of body of knowledge resources.
The purpose of this set of questions is to get data sets that would inform work that is being done around onboarding in the community cultivation channel. For more info on the work being done aroudn this check out this link (https://roamresearch.com/#/app/SourceCred/page/t_rZiNZBc)
Please comment, discuss and contribute to this set of questions so that we have a well formulated survey that would yield relevant data.
Here goes:
QUESTIONS FOR COMMUNITY
User profile
What is your area of interest in SC?
Area of Expertise - Dev, Design, UI/UX, content, Generalist (develop further)
How long have you been involved?
Degree of engagement - Wall flower, participate in discussion, contribute,
How often (1 is daily, 10 is monthly)
Onboarding process
How would you rate the current onboarding process (1 is terrible, 5 is awesome)
what is the biggest challenge you faced before you got comfortable contributing/interacting with the community
How long did it take you to :
Start participating in weekly calls or discussion?
Start contributing to one or more projects areas in SC?
What would have made your life easier during onboarding? What type of resource or assistance did you miss?
Body of knowledge/Content
How would you rate the current organisation of body of knowledge (1 is terrible, 5 is awesome)
What challenges have you faced while looking for required content/documentation?
What part of the body of knowledge had the largest value addition to you? What was your favorite segment of the over all body of knowledge?
How would you like the organisation of content to be different?
Communication channels
How intuitive has organisation of communication channels been? (1 is terrible, 5 is awesome)
What communication channel do you find yourself spending the most time in? Top three?
Community guidance
How would you rate the current quality of community assistance (1 is terrible, 5 is awesome)?
Was it easy to find who was in-charge of different projects?
Was community guidance/assistance easily available? Was it helpful?
How would you suggest that the community improves engagement?
Ravi, this is phenomenal! Thanks for taking the time to formulate these questions and champion the survey, this is a great place to build from.
Here’s some of the things I’ve learned from some research about formulating surveys:
Identify the answers we’re looking for and develop questions out of those. This helps make sure the data and questions will be relevant.
Be simple, direct, and specific. Use language that has as little room for interpretation as possible, and is clearly and widely understood.
Ask only one thing at a time. Break up more complex questions into multiple smaller, more direct questions.
Keep questions as neutral as possible to avoid “leading questions”.
Use interval questions to get nuanced data (1-5 scale).
Use more “extreme” language to get more potent answers, like “most”, “worst”, “biggest”, etc.
So what is the data we’re trying to get? We want to know whether people: know what’s expected of them, have the resources they need to engage, feel supported by and engaged with the community, have the opportunity to do what excites them, feel like they belong here. (Loosely tied to the mountain climbing onboarding philosophy.)
Here is my reworking of the survey based on your original questions and the things I learned from my research, mixed with my current working onboarding philosophy:
Please answer these questions honestly in regards to your experience. The feedback will help us to improve our community operations. Thanks!
User Profile
What is your expertise as a contributor?
What part of SourceCred do you find the most engaging?
When did you get involved in SourceCred? (less than a month ago, within the last 6 months, within the last year, more than a year ago.)
How often do you engage with the project right now? (1 basically never, 2 rarely, 3 check in from time to time, 4 frequently around, 5 can’t stay away)
In what ways do you engage with SourceCred? Choose all that apply: wallflower, text conversations in Discord, attending calls, posting thoughts in Discourse, working on a project/initiative, leading a team/championing an initiative, piece-meal work on my own, other _____.
How connected do you feel to the vision of SourceCred? (1 not at all, 2 I’ve got other priorities here, 3 it resonates, 4 it really excites me, 5 it’s what I’m ABOUT)
Expectations
I have at least a basic understanding of SourceCred. (Agree/Disagree 1-5)
I have useful skills I want to contribute. (Agree/Disagree 1-5)
I know how to start contributing. (Agree/Disagree 1-5)
It was easy to start contributing to SourceCred. (Agree/Disagree 1-5)
When I contribute, I know what’s expected of me. (Agree/Disagree 1-5)
What is the most frustrating part about trying to understand the scope of SC?
What is the biggest barrier you faced before you could start contributing?
What do you need the most to be able to contribute effectively?
Resources
I know what resources/tools are available to me as I learn about/interact with SourceCred. (Agree/Disagree 1-5)
I can easily find the documentation I need. (Agree/Disagree 1-5)
The documentation is useful and relevant. (Agree/Disagree 1-5)
It’s easy to share ideas and have conversation in the Discord. (Agree/Disagree 1-5)
It’s easy to get answers to my questions when I ask them. (Agree/Disagree 1-5)
I know who leads what in SourceCred. (Agree/Disagree 1-5)
I have the tools I need to work effectively in SourceCred. (Agree/Disagree 1-5)
I know how much cred/grain I have. (Agree/Disagree 1-5)
What tool, support, or resource would make the biggest impact in your ability to join or start a project in SourceCred?
Connection
I feel comfortable in the SourceCred community. (Agree/Disagree 1-5)
Someone in SourceCred cares about my experience. (Agree/Disagree 1-5)
People in SourceCred are willing to help me. (Agree/Disagree 1-5)
Others in SourceCred recognize the value I’m bringing to the project. (Agree/Disagree 1-5)
I can use my skills to make SourceCred better. (Agree/Disagree 1-5)
How did the experience of onboarding into SourceCred make you feel?
What experience and/or resource would have improved your onboarding the most?
How do you feel when you spend time with the SourceCred community?
What would (or what does) make you feel like you belong in the SourceCred community?
Open to any suggestions around how to rephrase questions to make them more potent, how to condense this list more, or any other feedback!
Excellent… I love the additions.
@louis and I did have the concern that there were too many questions, but the 1-5 rating reduces the cognitive burden
That’s awesome. And idk about you guys but I know I would start being lazy to answer more than 10 questions at a time. So maybe we could start with a set of 10-15 questions on a specific topic, then share another one a week later or so.
One way to split it could be first asking about expectations and resources for contribution. i.e. If people:
know what’s expected of them, have the resources they need to engage
Questionnaire structure would be:
User profile Qs
Expectations Qs
Resources Qs
Second about community assistance, belonging, connection, vision…
Alright, so how about this for the first questionnaire, oriented towards onboarding and connection with the community.
SourceCred Community Onboarding Survey
User Profile
When did you get involved in SourceCred? (less than a month ago, within the last 6 months, within the last year, more than a year ago.)
How often do you engage with the project right now? (1 basically never, 2 rarely, 3 check in from time to time, 4 frequently around, 5 can’t stay away)
Is the vision of the project clear to you (1-5 scale)
What part of SourceCred do you find the most engaging?
Onboarding
How did the experience of onboarding into SourceCred make you feel?
What resource helped you the most: documentation, chats, people?
What experience and/or resource would have improved your onboarding the most?
Connection
How connected do you feel to the vision of SourceCred? (1 not at all, 2 I’ve got other priorities here, 3 it resonates, 4 it really excites me, 5 it’s what I’m ABOUT)
Someone in SourceCred cares about my experience. (Agree/Disagree 1-5)
People in SourceCred are willing to help me. (Agree/Disagree 1-5)
Others in SourceCred recognize the value I’m bringing to the project. (Agree/Disagree 1-5)
How do you feel when you spend time with the SourceCred community?
If that makes sense I could go and put that into a Typeform.
The second questionnaire would be oriented towards expectations and contributions. We can use the questions above as well as the feedback we get from the first survey.
Cheers!
edit: does anyone have a paid subscription to Typeform?
for consistency sakes lets do “1 - less than a month, 2 - within the last 3 months, 3 - within the last 6 months, 4 - within the last year, 5 - more than a year ago”
@rpp63@louis@LB Wow, watching the progression of these draft iterations is very inspiring. The gain in clarity and concision across each version is really something, great job yall! A couple of minor thoughts: 1) how long will it take someone to digest and answer these questions in total? My guess is that at more than 2 minutes total, you’ll see a sharp drop off in responses. I think the length is ok where it is now, but thinking in terms of “time-cost” is something I though I’d just mention. 2) One glaring conceptual hole seems to be this: stated simple, SourceCred automates compensation. None of these questions address that… There is no question along the lines of: “are you interested in earning grain”, “how much grain do you expect to earn”, “would you put in more time if you earned more grain”, etc. Now, maybe this is intentional, the operating assumption in all this seems to be that you are targeting “casual” community members who are motivated by social participation more so than earning grain. Maybe this is good, I just wanted to make the assumption explicit.
and in the long run “earning cred/grain” will take more place. My suggestion for the questionnaire :
Replace Question 4
with “Earning grain in the next 2 years is: (not important / few bucks / enough to reinvest or play with / to approach a kind of wage/salary)”
keep the questionnaire as it is for those who responded in question 4 (not important or few bucks) but make a second track with light changes in the questions for those who responded (enough to reinvest… / wage )
the second track would have have the replacement of the questions 9 & 10 from track1 (inputs welcome from @DeltaFreq or @s_ben with the maker trial or some hypotheses we want to test)
9. Regarding the payout proposition, you will work if the slow/fast ratio is : fast only OR 25(slow)/75(fast) OR 50/50 OR 75(slow)/25(fast)
10. Regarding metrics, what are the most important ones (or the rationale) for you to make a contribution?
@DeltaFreq I totally agree that product/purpose related questions are key. I wonder if they deserve their own standalone survey.
Maybe we can add a question in the first, onboarding related, survey about purpose for joining community. That might give us an idea about high level motivations of the community, while a more detailed product related survey could dig in a little more.
I must admit, it didn’t even occur to me
What if we have some grain attached to completing a survey? That could allow us to expend the set of questions to give us actual metrics.
I do think we need to have a solid handle of what we hope to get from this survey, and then validate the questions one more time.
@LB@louis@benoxmo@DeltaFreq
That could be a great motivation indeed. One way to do that is to have respondents signal that hey have answered the questionnaire in the Didathing channel
Edit: this would work with Cred, not grain. My bad!
Edit2: with Ravi we were thinking that maybe people could leave an Eth address at the end of the questionnaire and receive some grain
Note: the free plan was limited to 10 questions (unfortunately, I’ve already used the trial). So I had to cut down 2 questions and couldn’t provide as much context I would have wanted to.
Maybe I should give a try to google form instead so we have more room for this. Let me know what you guys think.
Note 2: I can change anything re design, colors, grammar, syntax…
You did a really good job of paring down so many questions into the 10 allowed by typeform! Most of my feedback is about sharpening up the questions and rephrasing them slightly to make them more potent. Otherwise, I think you did a good job and I’m super grateful y’all have been working on this. It’s something we’ve needed for a long time, and we’ll only get better at collecting community feedback as time goes on! Release early and often as they say.
Direct Feedback by Question:
How long
None
How frequently
Nitpick comment: Here, maybe add “once a quarter or less” or an additional option for “almost never”
“What part of SourceCred do you find most engaging? What drives you? ;-)”
I feel like what I find the most engaging and what drives me are two different questions. I’d recommend staying simple and sticking to the original question: “What part of SourceCred do you find most engaging?”
Although, I decided to answer both and it was pretty interesting, so based on the kinds of answers you get in this feedback test maybe it’s worth asking both! However, if people give answers that are too varied or vague, we’ll want to simplify.
How did your first steps into SourceCred make you feel?
Somehow this question doesn’t feel right, but I can’t quite put my finger on why. What is the heart of this question? Is it to find out what people’s emotions were like in their first week? Or is it to understand what they liked/disliked about onboarding? How long of a time frame is “first steps”? a week? a month? Let’s hone in on what kind of answers we’re looking for from this question, and get more specific with how we ask it.
What resource helped you to find useful information making your first steps?
a. website documentation
b. Discord chat
c. Discourse forum
The available answers don’t feel quite expansive enough to me. I think I first really got my footing in understanding the community by going to the first ever community calls, because I’m a very in-person learner. But that wasn’t an option, so I went with the closest answer “discord chat”.
I’d recommend either reworking this question to figure out what answer we’re really looking for, or make it a text field response.
Potential rewording: “As you were getting involved with SourceCred, what resource was the most useful to you?”
What experience and/or resource would have made our onboarding easier and more enjoyable?
I want to sharpen this question just a little:
“What’s the one experience or resource you most wished for, but couldn’t get/find while you were getting involved with SourceCred?”
How connected do you feel to the vision of sourcecred? (5=very connected)
I’d rephrase this slightly to be an I statement: “I feel connected to the vision of SourceCred (1 Disagree, 5 Agree)”
Does someone in SourceCred care about your experience?
I’d change this subtly, right now it feels confusing to answer 1-5 to the question.
Here’s how I’d rephrase it: “I feel like someone in SourceCred cares about my experience (1 Disagree, 5 Agree)”
Are people in the SourceCred chats and forums willing to help you?
Again, I’d shift this to the first person, and I’d also simplify it a little. What if people in the forums are willing to help me, but not in the chats?
Here’s how I’d rephrase it: “It’s easy to get my questions about SourceCred answered when I ask them. (1 Disagree, 5 Agree)”
Do other community members recognize the value you are bringing to the project?
This question feels like it doesn’t quite fit with the rest of them for some reason. I think I’d replace it with this question instead:
“I know I can use my skills to make SourceCred better. (1 Disagree, 5 Agree)”
Additional Thoughts:
The 0-5 questions:
I think we should switch it from 012345 which is six options, to 12345 or 1234567. The reason is that it’s important to give people the chance to be in the middle or kind of neutral. Like a 3 on a 1-5 disagree/agree scale, or a 4 on a 1-7 strongly disagree/strongly agree scale.
Whether you use 1-5 or 1-7 is up to whether you want to give people the option to respond strongly agree/disagree.
The order:
I’d order these slightly differently for a few reasons: 1. I’d put the questions about how long and how frequently at the end, sometimes people answer more honestly that way, 2. I’d put the 1-5 questions first, and the text answer questions second
The goal:
I just want to circle back around to what the ultimate goal of these questions are. What is the biggest thing you’re really trying to suss out with this survey? And double check that all the questions support that goal.
In the end, these are just my thoughts, you are ultimately the designer and it’s up to you how you format things. Thanks for your work and I hope my feedback was helpful.
Going to follow LBs format of going through this question by question, I’m also including what the purpose of the question seems to be and what we can possibly learn from the response:
How long
None
How frequently
None
Q1&2 is going to be for general community engagement
“What part of SourceCred do you find most engaging? What drives you?
I think this question needs to be more targeted. How do we want to use the information gathered here, and how are we planning on using the metadata? For example could ask where people spend the most amount of time, but are we talking about communication channel ie discord, discourse etc, discord channel?
This is a good question to learn the following: What is the prefered means to user engagement, what is the best way to reach the company
How did your first steps into SourceCred make you feel?
Again we can focus this question more. This an important question, maybe we can combine Q8-9 and split this into multiple questions:
4.a. How quickly were you about to start participating in the community in a comfortable manner (or something like this)
4.b. How easy was it to get acquainted with the communication and work flow of SC community (or something like this) 4.a & 4.b combined could be a placeholder metric for how intuitive/userfriendly the engagement process currently is
What resource helped you to find useful information making your first steps? a. website documentation b. Discord chat c. Discourse forum
I liked LBs potetial reworking.
Why don’t we offer options, and they can pick as many as they like: Discord channel (allow for a text box to enter any particulars, if that is something we can do on typeform), Discourse forum, Website, SC documentation, community calls (text box to mention any specifics)
This metric will allow us to concentrate on the communication channels prefered by the community
What experience and/or resource would have made our onboarding easier and more enjoyable?I Narrowing it down to one experience would be good “What’s the one experience or resource you most wished for, but couldn’t get/find while you were getting involved with SourceCred?” Thoughts about the onboarding process
How connected do you feel to the vision of sourcecred? (5=very connected)
This might be a place where we can create user groups by asking their motivation to joint he community: “Why did you join the SC community (choose any) : Appreciate the vision/mission/goal, learn about product, earn grain/cred etc…” Major motivations for the community
Does someone in SourceCred care about your experience?
In my option Q8-9 can be combined : I received enough community support and engagement to allow me to participate in a meaningful way. (1 - disagree, 5 agree) User engagement experience with the community
Are people in the SourceCred chats and forums willing to help you?
Do other community members recognize the value you are bringing to the project?
I’m good with LBs suggestion - “I know I can use my skills to make SourceCred better. (1 Disagree, 5 Agree)” This adds to the quality of user engagement metric
Ok, so I think we’ve done enough iterations.
These are the 10 questions that I believe we should have in the typeform:
When did you first get involved with SourceCred? (Less than a month, Within the last 3 months, Within the last 6 months, Within the last year, More than a year ago)
How often do you engage with the project right now? (Almost never, Rarely - once a quarter, Occationally - once a month, Frequently - once a week, Can’t stay away - multiple days a week)
Where do you spend the most time engaging with the community? (Discord {enter channel name}, Discourse, Website, Community chats {enter events/calls})
How quickly were you able to start, comfortably, participating in the community? (Right away, within a few days, with a couple of weeks, within a month, never got comfortable)
What was the biggest hindrance to participation? ( I didn’t know who to speak with, I wasn’t sure where to ask questions, I wasn’t able to clearly follow community progress, I wasn’t able to quickly find relevant material)
Which resource helped you to find useful information making your first steps? (Discord channel {allow for a text box to enter any particulars}, Discourse forum, Website, SC documentation, community calls {text box to mention any specifics})
What experience and/or resource would have made our onboarding easier and more enjoyable?
Why did you join the SC community? (Appreciate the vision/mission/goal, learn about product, earn grain/cred, explore the implementation SC instance in your project)
How much do you feel people in SourceCred chats and forums are willing to help you? (1 little, 5 a lot)
Do you feel the community recognize and rewards the value you bring to the project? ( 1- I feel appreciated,5 - no one cares)
There are two things that I want to pose to the group:
Q3 & Q5 appear similar but Q3 tells the preferred method of onboarding while Q5 explore community engagement, Will the community be incisive enouhg to see this difference? And will their resposes reflect this difference? Or should we just consolidate?
Choice for Q8 really need to be developed well, otherwise we will not be able to extract as much info as we need from the question, so please feel free to develop a list of 5 options
@rpp63 > suggestions for Q8 in terms of possible answers:
to have a remote job you could make a living out of
to contribute to an opensource project
to experiment a decentralized way to get things done
for the cutting edge tech (the graph theory behind or the web3 interactions)
for the diversity of people and how inclusive they are
and regarding Q3&Q5 : it’s in my mind both related to the onboarding process.
Q3 is indeed about the method whereas Q5 is about the barrier
for Q3 : I would change in the question “community” with “the SourceCred project”. There is a better fit with all the answers espcially with the “website one”
for Q5 : if you change in the question “to participation” with “to make your first contribution”, I find it more incisive.
at some point. Good opportunity to make connections as well!
I think this question needs to be more targeted. How do we want to use the information gathered here, and how are we planning on using the metadata? For example could ask where people spend the most amount of time, but are we talking about communication channel ie discord, discourse etc, discord channel?
This is a good question to learn the following: What is the prefered means to user engagement, what is the best way to reach the company