Cred Rebalancing: A Props-Oriented "CredSpective"

Hello all, your friendly Temporary Benevolent Dictator here. It’s been a while since the last major change to the SourceCred weights, and I think it’s about time we shake things up a bit.

Throughout SourceCred’s history, we’ve only had data about on-platform contributions. The algorithm can see posts on the forum and changes to the code, but has struggled to see emotional labor, or logistical contributions, or all of the work that goes into meetings. This has resulted in a long-term bias in the Cred scores: the top slots have usually been occupied by developers, who naturally do most of their work on platforms. However, other kinds of contributions have been under-recognized.

However, in recent months usage of the #props, #didathing, and #meeting-notes channels in Discord has gone way up. As a result, we now have data about off-platform contributions. Using this data, we can correct some of these biases, by re-balancing our Cred towards the props, didathing, and meeting-notes channels. I think of it as a “change in CredSpective”.

If we’re going to re-balance the Cred, we can choose to do so conservatively, trying to move the weights slightly towards being more level. Or we can make an aggressive change, bringing the weights a lot higher and creating a much larger retroactive reward for undervalued labor of the past several months.

My intention is to do a big shift, and to err on the side of over-correcting by being too generous with these channels. I think it’ll be good to see some volatility in the Cred scores; SourceCred is still young, so we shouldn’t be afraid to experiment with new settings. It’s OK if we over-correct a bit; we can always retroactively rebalance things later. Think of it as a reminder not to take our Cred scores too seriously; there’s still a lot of room to improve them!

With all that said, here are the new weights I’m proposing. (You’ll note I took the opportunity to clean up a few other Discord issues, like turning on role-based Cred minting, and rebalancing the emoji weights.)

Channel Weights

Weight Current New
channel weights
#meeting-notes 1x 8x
#props 1x 15x
#didathing 1x 12x
#announcements 1x 0.1x
#random-chitchat 1x 0.25x
#questions 1x 3x
#tech-support 1x 3x
role weights
no role 1x 0x
@ community 1x 1x
@ contributors 1x 2x
@ core 1x 3x
emoji weights
:sourcecred: 16x 3x
:100: 8x 1x
:+1: 4x 1x
:-1: 1x 0x
all other emoji 1x 1x

Now, for the most exciting part. Here’s the new all-time Cred with the new settings:

Name % Before % After Change
decentralion 27.7% 20.4% −26.5%
lbstrobbe 6.4% 7.4% 16.6%
wchargin 12.9% 7.3% −43.1%
KuraFire 2.8% 6.2% 116.7%
s-ben 6.2% 6.1% −1.0%
hammad 4.6% 5.7% 22.7%
topocount 2.8% 4.1% 43.5%
Thena 1.8% 4.0% 121.0%
bex 2.2% 3.6% 66.9%
beanow 6.1% 3.5% −42.2%
joiecousins 1.3% 3.5% 161.7%
panchomiguel 1.9% 3.5% 81.2%
sandpiper 1.7% 3.1% 89.7%
benoxmo 0.6% 1.1% 85.7%
dependabot 1.8% 1.0% −41.9%
burrrata 1.7% 0.9% −45.5%
yalor 0.7% 0.9% 22.7%
mzargham 1.3% 0.8% −39.3%
eeli 0.5% 0.8% 57.1%
youngkidwarrior 0.6% 0.8% 34.5%
Jojo 0.3% 0.7% 156.1%
cortanav 0.3% 0.7% 139.8%
evan 0.8% 0.7% −15.5%
ian 0.2% 0.6% 178.2%
brianlitwin 1.0% 0.6% −38.8%
vsoch 1.0% 0.6% −46.3%
kate-huntoon 0.2% 0.5% 181.4%
greenkeeper 0.7% 0.4% −49.2%
crisog 0.1% 0.3% 175.0%
peth 0.2% 0.3% 59.5%
Past 4 Weeks
Name % Before % After Change
Thena 10.2% 11.5% 12.6%
KuraFire 8.6% 10.1% 18.1%
lbstrobbe 9.2% 9.9% 7.6%
decentralion 9.5% 8.5% −10.6%
hammad 6.4% 8.4% 31.4%
joiecousins 4.4% 6.1% 39.8%
s-ben 5.2% 5.3% 1.4%
Jojo 3.9% 4.7% 20.2%
topocount 5.1% 4.5% −11.5%
sandpiper 3.4% 3.3% −2.8%
eeli 3.1% 2.7% −11.9%
kate-huntoon 1.3% 1.9% 49.6%
kstone 1.3% 1.9% 42.9%
peth 1.4% 1.8% 32.7%
panchomiguel 3.4% 1.7% −48.3%
Willow 1.5% 1.3% −9.1%
bex 1.1% 1.0% −13.2%
wchargin 1.5% 0.9% −37.7%
Felix 1.3% 0.9% −29.9%
dependabot 2.1% 0.6% −69.1%
benoxmo 0.6% 0.6% 14.5%
prose11 1.0% 0.6% −36.1%
BrianBelhumeur 0.9% 0.6% −31.9%
saintmedusa 0.6% 0.5% −18.3%
owocki 0.3% 0.5% 82.6%
cryptouf 0.2% 0.5% 122.6%
monstrosity 0.7% 0.5% −31.7%
magwalk 0.6% 0.4% −26.3%
Felipe-Duarte 0.6% 0.4% −28.6%
youngkidwarrior 0.5% 0.4% −15.9%
Past 12 Weeks
Name % Before % After Change
decentralion 12.2% 11.0% −10.1%
Thena 9.1% 10.7% 17.4%
KuraFire 7.9% 9.7% 21.8%
lbstrobbe 7.6% 8.0% 5.3%
topocount 6.9% 6.4% −8.1%
hammad 5.1% 5.7% 12.5%
panchomiguel 6.4% 5.4% −14.5%
joiecousins 4.0% 5.4% 33.9%
s-ben 5.2% 5.1% −2.5%
bex 3.1% 3.5% 12.7%
sandpiper 2.9% 3.3% 13.1%
Jojo 1.7% 2.2% 31.3%
eeli 2.7% 2.2% −20.4%
yalor 1.2% 1.7% 41.2%
kate-huntoon 1.0% 1.4% 44.2%
cortanav 0.8% 1.2% 60.2%
benoxmo 0.8% 1.1% 38.1%
dependabot 3.0% 1.0% −66.8%
wchargin 1.6% 1.0% −38.1%
youngkidwarrior 1.2% 0.8% −31.4%
kstone 0.5% 0.7% 41.2%
peth 0.5% 0.7% 37.0%
Simple-Poll 0.7% 0.6% −11.3%
Felix 0.7% 0.5% −27.0%
Willow 0.5% 0.5% −4.4%
BrianBelhumeur 0.6% 0.4% −27.5%
amico 0.4% 0.4% 6.6%
crisog 0.3% 0.4% 25.8%
pablomendez-95 0.3% 0.4% 33.9%
lanski 0.3% 0.3% 32.3%
Past 52 Weeks
Name % Before % After Change
decentralion 21.2% 16.0% −24.7%
lbstrobbe 8.9% 8.7% −2.2%
KuraFire 4.4% 7.5% 73.0%
hammad 7.1% 7.0% −2.0%
s-ben 7.7% 6.7% −13.0%
topocount 4.3% 5.0% 14.6%
Thena 2.8% 4.9% 76.5%
bex 3.3% 4.4% 35.2%
joiecousins 2.1% 4.3% 109.0%
panchomiguel 2.9% 4.3% 46.2%
sandpiper 2.5% 3.8% 51.5%
wchargin 3.9% 2.5% −35.8%
beanow 4.7% 2.3% −50.1%
benoxmo 0.9% 1.4% 48.3%
dependabot 2.6% 1.2% −53.4%
yalor 1.1% 1.0% −2.0%
eeli 0.8% 0.9% 25.5%
youngkidwarrior 0.9% 0.9% 7.4%
Jojo 0.4% 0.9% 104.5%
cortanav 0.4% 0.8% 91.4%
ian 0.3% 0.7% 158.0%
evan 0.9% 0.7% −25.0%
kate-huntoon 0.3% 0.6% 124.7%
mzargham 1.3% 0.6% −55.9%
crisog 0.2% 0.4% 119.6%
peth 0.3% 0.4% 27.3%
Simple-Poll 0.3% 0.4% 32.1%
brianlitwin 0.8% 0.4% −54.1%
vsoch 0.8% 0.3% −55.5%
amico 0.5% 0.3% −35.8%

This is a big shift in Cred! Looking at the weights, we can see why. The weights apply multiplicatively, which means a :sourcecred: emoji from a Core team member on the props channel is now worth 3 * 15 * 3 = 135 Cred. This is super high! It’s equivalent to 8 merged pull requests, or 33 Discourse likes. These weights will likely come down in the future, for example when we start to use the Creditor instead of Discord channels to keep track of props. (Or, if we keep high props weights, we’ll want to go back and retroactively-props old contributions from before the props channel!)

But for now, we can enjoy some freshly re-biased Cred.

Next Steps

As TBD, I’ve have final say over any weight changes. However, we should start working towards a more decentralized process. As such, I’d like to see whether we have ‘rough consensus’ for these changes. Please share your thoughts: do you think these changes go too far? Or perhaps they don’t go too far enough? If there are no major objections, I intend to roll these out by end of the week, so they’ll be live in our next Grain distribution.

8 Likes

Interesting… A few things:

  • The channel weights for props, didathing, and meeting-notes are SUPER high. I understand the idea is to shake things up dramatically though. The main downside to this I see is that discourse is now likely to be really undervalued vs discord going forward. Maybe this is fine since we don’t use discourse that much vs discord :man_shrugging:

  • I like the emoji weights changes, I think that will make discord less susceptible to abuse, and better at capturing value accurately. Even the 2x super emojis were causing some issues at 1hive for awhile.

  • I also approve of permissioned roles for minting cred. This was the main thing that stopped abuse in its tracks at 1hive. Although, we did minting using cred based roles rather than hand picked roles. Maybe this can be taken into account in a future iteration.

Overall, I can’t really complain about the changes as is. They do seem to improve the distribution of cred for the most part. I hope this doesn’t disincentivize dev work going forward. Although this is a balancing act anyway which we will likely have to continue iteration on. Perhaps we could implement budgets on our plugins? (Afaik these haven’t been implemented on the SourceCred self instance?)

3 Likes

Thanks @decentralion I really appreciate all the analysis.

I think we can bump weights on discourse when we find better, more regular uses for it. I’m inclined to start directing support traffic here after thinking though the organization a bit further, since it’s more searchable and a potentially happy medium between the accessibility of discord and the permanence of github issues

3 Likes

Re: discourse and dev work being downplayed

I think this will cause a cultural shift towards double-tapping discourse and dev work in props/did-a-thing being more acceptable crediquette, which means that it will be subject to the same value signals that care work relies on. This seems good, as it will incentivize devs to make that sort of “manual” value capturing more robust and usable.


I support the changes. I don’t want us to get too flippant about cred being experimental, because some of us rely on the income. But this seems to generally support most of us that do, and I do agree with leaning into our org as an experiment in ways that are nourishing.

7 Likes

Some loose thoughts/observations, which by no means are conclusions. All these points are an effort to understand more:

  • You lay out the problem very clearly here, but I’m unsure why this specific strategy is the best option. For instance, is this the best way to recognize off-platform contributions? If so, why? What other solutions might exist? Why this over those specifically? And most importantly, does this change actually achieve the goals stated? If so, how do we know?

  • I’m curious to hear more of your take on the downsides. With any rebiasing, there’s two sides to it, and countless more side effects.

  • Is this fair for historical contributors? For instance, I see that William’s cred is down over 40%, but wouldn’t he have acted differently under the proposed cred system? Wouldn’t everyone have?

Cred on posts scales linearly with reactions right now from my understanding, so 3 :sourcecred: emoji’s from core team would be 405 cred, which is roughly 24 PRs. This certainly holds devs accountable, but I can’t help but scratch my head here.

There’s certainly a strong check on dev power here (good), and I like @blueridger’s point that this forces devs to connect with and be accountable with the larger community (good). At the same time, I really don’t want to have to sell what I’ve been working on in order to receive some balanced recognition. Cred-wise, it feels like there’s no option to quietly go along doing one’s work here.

7 Likes

As @blueridger stated, this will most likely cause a shift in dev work being manually pointed out by the contributors, via didathing and props channels because, as @eeli points out, three :sourcecred: can in the right circumstances be roughly equivalent to 24 PRs, which even for relatively minor PRs is a load of dev work.

I understand that this manual recognition work is already being asked of those who provide contributions that are less dev-oriented, so in that sense it doesn’t seem unreasonable to ask the same of devs. However, as the current focus on the creditor attests, a long-term goal of SourceCred is automatically value-capturing contributions, and this policy shift seems like it would encourage a reverse trend towards manual value evaluations of on-platform contributions.

It might be beneficial to think of the issue in two parts: 1) rebalancing current cred scores to more accurately reflect off-platform contributions in the past, and 2) creating a policy going forward that will maintain a balance that’s more fair than it has been historically. I suggest being more aggressive in point 1, and less aggressive on point 2 to correct the oversights of the past of on/off-platform value capturing, and continue with a policy going forward that help to maintain that corrected balance for future contributions.

4 Likes

Welcome to the world of a community cultivator! We say that we believe in “dogfooding” but honestly the experiences of our product that the devs have been dogfooding and that devs have actually served to non-devs are totally different. I see this as just a dedicated commitment to dogfooding.

5 Likes

@blueridger I’m someone who likes to “Let my work speak for itself.” I’ll likely adapt to this cultural shift, but I still really don’t enjoy shilling my contributions. I’m more concerned for more introverted developers/contributors though. What allowances can we make for them in this project, beyond hoping that they have an advocate that will props them?

I guess the big question we need to work to answer is: How can we instill the practice of elevating under-recognized contributions?

2 Likes

Why does thumbs down mint Cred?

It currently does but won’t after the change, since it will be a zero-multiplier.

I hear you (and @eeli) that having to manually advocate for props or recognition is uncomfortable, especially for introverted types. However, as @BrianBelhumeur pointed out, this is already the expectation for non-dev work. It’s not easy to have to ask for recognition, or rely on other people seeing your contributions.

5 Likes

I agree, and I don’t want to simply dismiss the problem of bias towards work that earns “hard cred” from plugins. This is something we’ve got to solve, and will make everyone better off if done well.

That being said, I don’t see pushing devs into the same problem that our community cultivators have as the solution here. I would love to see a solution where community cultivators and care workers don’t have to constantly advocate on their own behalf.

I don’t have the answers here, and I don’t mean to be critical without being constructive. I guess I’m just advocating that we shift the conversation towards bringing care workers and community cultivators out of the state they’re currently in.

Yes, but those solutions are all longer term, and do not recognize that our work as cultivators right now are not being properly rewarded. To be honest, I don’t think it’s too much to ask that devs tolerate this system until something better can be implemented. Yes, I want the Creditor made, and think it’ll solve many of the issues, but right now our system under-recognizes non-dev work.

5 Likes

It may not be a solution by itself, but it’ll certainly incentivize the devs to prioritize finding solutions. In that way, I think it’s quite elegant.

5 Likes

If devs wanna channel their critical thinking and problem solving energy in solidarity with our care workers, check out my new post:

3 Likes

Also, this change literally results in a positive all-time cred delta for all active devs, and positive 12-week cred delta for most active devs. So folks could likely do exactly what they are already doing and still get fair compensation.

2 Likes

I like the general direction of this proposal, but there are a few things that seem off in terms of the new cred amounts.

Main thing would be that its heavily weighting to contributors that joined after we established props / didathing, so older contributors are disadvantaged since those channels didn’t always exist. I think this is a good situation where time scoped weight config would be really valuable (i.e. these weight changes only apply to activity in the last 6 months instead of changing all the historical cred as well). @Beanow’s cred score especially stands out to me as being too low relative to cred scores of newer contributors.

I feel like we would need to dig through where and how the cred is actually flowing to make a proper informed decision about this as a community because it definitely feels like it needs some more fine tuning to iron out some inconsistencies.

In addition, I think this could also introduce a lot of gaming (intentional or unintentional) since the cred flows are now primary dictated by reactions on discord / humans and therefore starts heading much closer to Nosedive / popularity contest territory where the social dynamics and people’s personalities start to affect cred scores and crediquitte becomes extremely important but also extra awkward with the current additive way of calculating cred based on reactions.

IMO our current method of minting more cred as more emoji reactions come in isn’t ideal and is something we should work on refining if we want to have high signal / low noise cred flows. Right now I always have to play this internal mind game to see if I should react to something or not based on how many reactions it already has since I don’t want it to get over-credited even though I like the contribution. I’ve seen even in MetaGame that contributions that invoke more emotions often get a lot more reactions than ones that are more mundane but still super valuable (e.g. someone making a really funny meme or charismatic post that hypes people up vs someone who did the grunt work of coordinating dozens of people or cleaning up backlog of old issues / forum posts, etc). There’s also the bias of people adding reactions with the same emojis as the previous person who reacted.

@blueridger and I were discussing an alternative approach where instead of counting the amount of reactions, we have a way for people to indicate how much they think that contribution is worth and the resulting weight would be the “mean” instead of the “sum”. For example, we could have a bot that automatically adds 4 reactions to every post in props/didathing that represents “common (1)”, “rare (3)” “epic (5)” and “legendary (8)” contributions and people just react with the one they think applies and the resulting weight would just be the average of the individual reaction weights. I think something like this would greatly increase the signal/noise ratio for props/didathing channels and is likely closer to what I envision weighting nodes in the creditor to look like.

Also, this seems to be inflating the total cred a lot as well, would be nice to understand what the implications of that are, if any. There’s the extra dynamic with grain payouts as well with the Recent policy which is already favouring more recent contributors, so shifting the cred heavily in favour of recent contributors compounds that effect. I would be interested in seeing what projected grain payouts would look like with these new cred scores and if we would want to adjust the distribution strategy there as well.

8 Likes

Appreciate the question. I support this change and am quite happy with it, for two reasons:

  • I’m happy to see more cred flowing to people who aren’t developers. This has been a nominal goal of SourceCred from day zero, but for a long time we only had dev contributions. That’s clearly no longer the case, so it’s great to see the top two spots no longer occupied by @decentralion and me.

  • The meta-point of treating the cred scores as fuzzy and volatile is important. We report cred scores as scalars, but those scalars collapse a whole world of both uncertainty and complex structure: i.e., cred really stands for something more than a one-dimensional number. Shaking things up, in a way that’s respectful of people’s income streams, helps reinforce this point.

If my goal were to maximize my short-term cred, then sure, I might have acted differently under the proposed system. But I’d also be acting pretty differently under the current system! (For instance, I also supported the last change to significantly reduce my grain distributions.) Even as a purely self-interested agent, I should really aim to maximize the value of my cred, which incorporates both how much cred I have and the success of the project (and thus the per-unit value of cred). It’s clear to me that helping the project succeed is much higher leverage, so I tend to just do what I think is best and let the cred flow where it may.

5 Likes

Great points. It’s easy for someone who’s work is recognized (me) to say let’s hold off. At the end of the day supporting care work comes down to action and not cheap talk, and I’m willing to step into a system that I don’t find perfect if that’s what it takes. I don’t have any conclusions here and don’t mean to sound adversarial, I’m just raising some concerns/questions in the hopes that we can balance everyone’s concerns/priorities/comfort

If this is treated as a temporary fix while the creditor is being worked out, then my concerns entirely dissolve for the reasons @wchargin cites below. I just want to see what the consensus is around this point. Is this perceived as the future of SourceCred, or a temporary patch while we figure out a better way to treat community cultivators / care work?

2 Likes

A question just popped into my mind:

How does this specifically help community cultivators / care workers? If the response of this is for devs to start using these channels, won’t they just start getting crazy props cred too?

And another:

How will we measure whether we’re rewarding care/community work sufficiently? We all agree we’re way out of balance now, but how will we know when we’re back on track? What measures are we implicitly using here?

1 Like