Building SourceCred for the Wider World (Pt. 1: Background)

[eeli]

Context / Inspiration — What got me thinking…

Thanks to our benevolent corporate sugar daddy PL, we are in a very different position resource-wise than many communities who want/need some community credit primitive. In our community we’re essentially using SourceCred to define how to get the goodies from corporate sugar daddy. However, our situation is an anomaly, or at least I hope it is, because I believe we should be building for communities that need SourceCred to get off the ground and to scale—to make self-organizing collectives that were not possible possible—to enable and unlock new forms of human organization through a brilliant organizational primitive.

So how can we better serve resource constrained communities that don’t look or think like us?

Excitement around the Creditor is a major step towards allowing communities to align SourceCred with what they value. Even within our community, SourceCred has worked not because plugins are great out of the box, but because it has the capacity to transcend the plugins. SourceCred can dance elegantly with our values from week to week, month to month. This power is what we should be leaning into as much as possible. But where is this power coming from?

Currently, the only way for communities to specify what they value outside of plugins are the props and did-a-thing special channels in Discord. I haven’t done a formal analysis, but my strong hunch is that without these channels, our cred scores would be majorly misaligned as they would fail to capture all the great contributions that cannot be captured well by plugins. Only a fraction of what we value can be captured by plugins, and it can only be captured so well.

My point here is that the the foundation of SourceCred is soft community consensus (props/did-a-thing) that transcends any hard tokenization of it (plugins). The plugins can be improved with time, allowing us to more precisely quantify contributions in a more automated, scalable way. The floor is the “soft-cred”, and the sky is filled with clouds of great plugins, or “hard-cred”.

So how specifically can we empower communities to align SourceCred with what they value?

Recap: If we’ve learned anything, it’s that communities (including ourselves) need a way to align their SourceCred instance with their values beyond the plugins. We’ve seen that props and did-a-thing can help serve that purpose.

Now, let’s see how these two channels can be better. Right now, they reward popular contributions, not necessarily ones that align with our values. We have been fortunate to be a small community of good faith actors thus far, allowing popularity to be a good signal of value. However, this mechanism is fragile as the community grows and noise begins to dilute signal.

To be hyper clear, we need something that can offer:

1. quality signaling about what’s valuable
2. quality signaling about how valuable things are, and
3. aggregation of information about non-plugin contributions.

How do props and did-a-thing rank?

1. Works if Discord populated mostly by good faith, knowledgeable actors.
2. Same as (1) but can also suffer from popularity dynamics. What’s most valuable might not be the most popular. What’s most popular might not be the most valuable. More specifically, it has the property that Alice might receive less cred than Bob would for the same contribution. We can do better.
3. Great

Ultimately, this boils down to two problems we need to solve:

1. Drowning out noise for trusted, quality signal.
2. Fair quantification of value such that the same contribution gets credited equally independent of who did it.

This was a birds-eye overview of the situation. In Pt 2. I’ll explore how I think we can go about this. But first, please let me know your thoughts so I can include everyone’s wonderful feedback.

[panchomiguel]

There’s a lot of threads to pull at in this post, both pragmatic and idealistic.

I’ve been pondering this difference about SourceCred for community building vs. SourceCred for established communities. There’s definitely something important here wrt the norms we as a community are actively establishing (the soft community consensus as I understand it).

I think it’s super valuable to keep all of these questions and analysis in mind while we shape the Creditor.

[prose11]

Hello! I come to SourceCred via MakerDAO, where I am questioning the same things as you mention for improving our implementation of SourceCred. I read part 2 as well and it seems you have running into some of the same problems I am with trying to figure out how to actually pay people based on their contributions, not how popular or favorable a message their post sends.

There are obvious differences for Maker, as we only run SourceCred on our Discourse forum and thus many issues of not being compensated for work done can be linked back to not expanding our platform use of SourceCred. However, even if we were to expand our running of SourceCred to the extent the team here is doing, the issues posted above still stand.

One main issue for the flow of Cred in our platform is that most users are not actively thinking about the cred instances that get generated based on their actions. A little confusing, because ultimately that’s how seamless we would like the algorithm to be. I began to wonder how easily the system could be gamed at Maker and while it seems everyone is still acting in good faith, the answer is pretty easily. I will not specify the user of the example I am about to give, as to not encourage people using this potential exploit over at Maker, but the problem could easily be applied to just about anyone on the platform:

After receiving a “like” from a highly valued community member after a post I made (about a week after I heavily started investigating SourceCred) I started to wonder if it was possible to “target” likes from people with high Cred scores on our forum and how easy that would be. Thanks to the Discourse platform, it was very easy for me to see which “high profile” users had already liked my posts. I noticed a glaring absence from one of the contributors best paid by SourceCred on Maker’s platform. So I went to their profile and navigated to “Likes” to see if there was a particular subject I could post about if I were trying to game the system for higher Cred. This user had significantly more likes received than given, and when I looked at their likes, it was clear that nearly half of them occurred when a user tagged and complimented the high profile user’s work. As this seems like polite social discourse, it would not be surprising if the same could be said (perhaps to a slightly lessor degree) for almost all of the top Cred earners in MakerDAO. In other words, it would be incredibly easy to target likes and create Cred flow by being complimentary and highlighting good work.

While the above is rather specific to MakerDAO (our use of only one platform and the general ignorance surrounding how Cred is calculated), it is not hard to see how this could apply to any platform running SourceCred. Along the lines of this thread, there’s a popularity and social expectation problem where popular users will be well “liked” and distribute a fair amount of their “likes” to people that compliment them for their work. When this is genuine, it does not pose too much of an issue IMO. People complimenting work draw more attention to it, allowing more Cred to flow to the post and they are rewarded in kind for doing so. However, if the entire community were more aware of how SourceCred is functioning, and the economic incentive was great enough (as we hope it to be, so it can replace some hourly/salary decisions for the DAO), users could gain much more cred for being complimentary than spending their time debating ideas that were brought forth. I think this points to a bigger potential issue with relying solely on plugins to distribute grain. More optimistic and complimentary work will be over-liked by the community (and thus over-payed) while contributions that are much more critical of systems in place (and therefore potentially much more important to community health) could end up under-valued.

I’m intrigued by your use-case examples in part 2 @eeli and think that finding a way to flow Cred from soft sources will be incredibly important to being able to implement SourceCred at scale. I was wondering out-loud to @s_ben about the possibility of the algorithm valuing different posts based on their housing in different sections of Discourse, that way forum moderators could move posts to their proper category and help curb the ability to game the system. I think more so than just having weights, having the ability to favor types of posts for communities will be extremely important, at least until Boosting is functioning at scale and the community can help drive the process automatically.

I’m hoping this was somewhat helpful as a perspective of someone grappling with how to make SourceCred more effective in their own community. For my part, I’m also looking for ideas and suggestions to test within Maker for driving more meaningful contributions. At present, our problem seems to be having too many smart people, spread too thin, while not being properly compensated for their effort. I believe SourceCred is the best answer we have available to fixing that problem and with some input and advice I hope we get to the point where the SourceCred community can point to MakerDAO as a living organization embodying the ethos and power of SourceCred. Thanks for the time, feel free to contact me here or on your discord (MakerDAO’s works too if you’re curious ) !

[s_ben]

You raise some good points @prose11.

@prose11

One main issue for the flow of Cred in our platform is that most users are not actively thinking about the cred instances that get generated based on their actions. A little confusing, because ultimately that’s how seamless we would like the algorithm to be.

Conflicted on this personally. On the one hand, if people are not thinking too much about how they engage the system, they’re also less likely to be gaming it. Also, it’s less effort/adjustment on their part. This becomes a bigger factor if many communities are using SourceCred. When a contributor goes to a new community, they’ll have to learn that new community’s “Crediquette” to not accidentally screw up that communities’ scores. On the other hand, contributors engaging more intentionally, with knowledge of the algorithm and operating in good faith, may be able to “steer” the system more effectively. Being forced to learn a community’s Crediquette might also be an effective way to make an instance more sticky (people value what they work for) and filter new contributors for value fit–curious if @LB, @joiecousins or others working on community cultivation have thoughts here.

In watching communities use SourceCred, my observation, based on the Maker survey, talking to random contributors, and just my generally observing patterns of behavior, is that most contributors aren’t altering their behavior that much. Though a deeper analysis could definitely prove me wrong.

@prose11

I began to wonder how easily the system could be gamed at Maker and while it seems everyone is still acting in good faith, the answer is pretty easily. I will not specify the user of the example I am about to give, as to not encourage people using this potential exploit over at Maker, but the problem could easily be applied to just about anyone on the platform:

After receiving a “like” from a highly valued community member after a post I made (about a week after I heavily started investigating SourceCred) I started to wonder if it was possible to “target” likes from people with high Cred scores on our forum and how easy that would be. Thanks to the Discourse platform, it was very easy for me to see which “high profile” users had already liked my posts. I noticed a glaring absence from one of the contributors best paid by SourceCred on Maker’s platform. So I went to their profile and navigated to “Likes” to see if there was a particular subject I could post about if I were trying to game the system for higher Cred. This user had significantly more likes received than given, and when I looked at their likes, it was clear that nearly half of them occurred when a user tagged and complimented the high profile user’s work. As this seems like polite social discourse, it would not be surprising if the same could be said (perhaps to a slightly lessor degree) for almost all of the top Cred earners in MakerDAO. In other words, it would be incredibly easy to target likes and create Cred flow by being complimentary and highlighting good work.

Fascinating example. Thank you for talking about this example openly. It makes addressing these things easier. There will certainly be people that engage with these platforms sometimes very differently than the mean. This could indeed create opportunities for gaming. I’m not terribly worried about this currently, because the amounts being distributed are not high enough for most to invest in gaming. Also, one nice thing about likes, is that a single like is not so consequential. Just like on Twitter, I like tweets freely, but Retweet much less. I see Retweets as deployments of my scarce ‘social capital’, and put more thought into them.

It is also (in the higher-trust communities I’m following) fairly easy to see someone blatantly gaming. For instance, in Maker, every week when I run the scores, I scrutinize the new’ish people earning Cred. It only takes a quick glance at their activity in Discord to see anything unusual. I also look at the top Cred-generating posts, and look at each one. If there was a big spike on a post that didn’t seem that valuable, I would investigate further. Now, this will change when rewards are increased. I think basically we have to just move at the speed of trust. If we move too fast in increasing payouts, we’ll get gamed and will need to go back to the drawing board and come up with new features, process, etc.

@prose11

However, if the entire community were more aware of how SourceCred is functioning, and the economic incentive was great enough (as we hope it to be, so it can replace some hourly/salary decisions for the DAO), users could gain much more cred for being complimentary than spending their time debating ideas that were brought forth.

Ass kissing Cred No way to eliminate that entirely from human nature, but one fun idea (which could totally be implemented), is to quantify ass kissing and reduce Cred for it.

@prose11

I think this points to a bigger potential issue with relying solely on plugins to distribute grain. More optimistic and complimentary work will be over-liked by the community (and thus over-payed) while contributions that are much more critical of systems in place (and therefore potentially much more important to community health) could end up under-valued.

This is certainly a danger. Networks generally tend towards winner take all modes. One failure mode would be to have a “Cred elite” with enough power to crush any and all dissent, to the detriment of the long-term health of the community. This is something that has been much discussed, and taken into consideration. But this issue will likely arise again and need to be addressed further as we move into larger communities with lower trust levels. I will say that one nice feature of SourceCred currently, is that it’s not a binary the way traditional employment is. In a company, if you’re critical of the company’s direction, you run the risk of being fired. Your pay goes from 1->0. In SourceCred, barring the holder of the Treasury’s keys deciding to censor your payment (a large violation of the social contract unless that possibility was made clear up front to new contributors), your Cred score may go down, but it won’t go to zero. If you find some supporters to like your posts, even if they have lower Cred scores, you will continue to get paid, albeit at a discount. And because the algorithm is retroactive, if your critiques or unpopular ideas end up being valuable and winning, your Cred score should go up proportionally to reward you for taking on that risk.

@prose11

At present, our problem seems to be having too many smart people, spread too thin, while not being properly compensated for their effort. I believe SourceCred is the best answer we have available to fixing that problem and with some input and advice I hope we get to the point where the SourceCred community can point to MakerDAO as a living organization embodying the ethos and power of SourceCred.

As do I! You raise some key issues in your analysis, which is great to see. We definitely have a long way to go, but I think we’re chipping away in the right direction, and we always welcome critique and new ideas:)

[prose11]

Super insightful as always @s_ben !

@s_ben

In watching communities use SourceCred, my observation, based on the Maker survey, talking to random contributors, and just my generally observing patterns of behavior, is that most contributors aren’t altering their behavior that much. Though a deeper analysis could definitely prove me wrong.

I get the same feeling and I think it’s a net positive. I think initially I was too concerned with making sure the payouts were functioning “correctly,” not realizing that the algorithm is quite powerful, and even if people are using it in different ways it can roughly approximate how much to value a like based on all the other likes coming from the various nodes. I’m still thinking that getting the word out about SourceCred (from the prospective of encouraging people to like and comment on work their find valuable) will help even further, but my initial worries about different use styles being the origin of misbalanced compensation are far in the past.

@s_ben

Now, this will change when rewards are increased. I think basically we have to just move at the speed of trust . If we move too fast in increasing payouts, we’ll get gamed and will need to go back to the drawing board and come up with new features, process, etc.

This is huge and I’m really happy to see the phrase “moving at the speed of trust” so heavily leaned on in the SourceCred Community. I think a personal flaw I see in myself is when I like something I want it to blow up to the full potential I envision for it, and get frustrated when it doesn’t seem to be doing so. With that said, it’s clear some major developments are being made over at SourceCred and I think having this improvements to point to will allow more trust, both over at Maker and wherever SourceCred is implemented.

@s_ben

If you find some supporters to like your posts, even if they have lower Cred scores, you will continue to get paid, albeit at a discount. And because the algorithm is retroactive, if your critiques or unpopular ideas end up being valuable and winning, your Cred score should go up proportionally to reward you for taking on that risk.

This is another thing I objectively undervalued when I first started to dig into SourceCred. My perception was that while older ideas could become liked and referenced later on, I was thinking there wasn’t a great chance for that happening. In just these few weeks since I posted this, not only have I noticed myself going back into old topics and liking ideas (albeit with the knowledge of the algorithm), I have also noticed that my older posts are still receiving likes at Maker, and even getting link backs when new discussions are formed. That experience has sold me on the idea that dissent can still prove profitable with SourceCred, even if it is likely to be initially undervalued.

Thank you for sticking with me as I try to poke holes in the thing I want to succeed haha. I’m sure I’ll run into other brainstorms that cause me to question some functionality, but for the time being I’m spending a lot more brain-power trying to find ways to scale up our reliance on SourceCred at Maker. Thank you for being so welcoming in your community and spending the time to walk me through what I’m sure you have gone over countless times internally.