Cred Rebalancing: A Props-Oriented "CredSpective"

21

Interesting…

Overall, like the changes. I like going big, in part because I think we’ll likely do this again before we converge on something more stable long-term. The longer we go without big changes, the harder they’ll be to make later.

I understand the concern of devs that they now have to ‘shill’ their contributions. Part of the magic of SourceCred is that you can just contribute, and the algorithm will automagically reward you. However, I think this will be better for everyone in the long run, including devs, for a few reasons:

  • If it’s not a problem for you, it’s not a problem. Everyone has this bias, including myself. When devs have to do the work of explaining the why, they will understand the problem better and be incentivized to build tools that make that easier. They could even automate this away entirely, making other domains rewarded automagically. If it’s not their problem, it’s likely this continues to stall, as it has for over a year now, and frustrations of cultivation team and others build :grimacing:
  • Better communication. Many non-technical contributors are completely unaware of most code contributions. By having to communicate what they did and why, devs will educate others and keep them up to speed. Doing this atomically, here and there, in one or two sentence posts, could be less of a burden than writing up longer ‘reports’ periodically. Similar to devs I see on Twitter tweeting out things they’re doing for fun and clout. This could also have unexpected benefits for devs, such as being seen more, which is key to fostering emergent strategies. Devs will likely get love for their contributions (thank you @topocount for making my life soooo much easier with this feature !! :heart: :sourcecred: :heart: :sourcecred: :heart: :kissing_cat: !! ) , as well as valuable feedback as to what the community wants. Which leads to another benefit…
  • Proto “boosting”. I imagine opening up this communication channel between devs and the wider community will cause devs to be better in tune with community needs. They’ll get showered with :heart: s and :sourcecred: emojis for doing work the community appreciates, gaining a share of Cred and Grain other participants earned in other domains. While this is not like boosting in important ways (it’s smaller-scale, organic, and backward looking (only paying for work already done), it could provide some interesting data and insights we can use when designing larger-scale cryptoeconomic mechanisms.

I do have some concerns and ideas though.

Concerns

  • Subjective Cred: A big part of why SourceCred is fair–at least on the platforms it supports–is that Cred is tied to objective artifacts. People have already expressed concerns that #pros and didathing are already seeing signs of being ‘popularity contests’. This could increase this effect, over rewarding extroverts, self-promoters and gamers. We will need to create new plugins or other value capturing tools for non-dev teams to recieve the benefit of intersubjectivity. Norms around describing concrete actions could help, but not be as robust as say, Pull Requests on GitHub… Another problem, that @META_DREAMER points out already occurs in MetaGame, is that important but non-sexy contributions (e.g. code refactoring) won’t get the Cred they deserve. Indeed, the first thing Maker wants to look into with our latest feature (Cred minting on categories and tags in Discourse), is incentivize the creation of MIPS and Signal Request posts, which are more tedious governance documents that contributors don’t create enough of. This could go in the opposite direction, making necessary but unsexy work even less desirable.

  • Runaway Cred inflation:

If :sourcecred: emojis are worth this much, we could see a flurry of :sourcecred: s, inflating the total Cred supply more than we desire. I already find it hard to gauge what a :sourcecred: translates to. I can’t really reasonably judge how much a contribution is worth relative to other contributions. Newcomers will be even more unaware. I like @befitsandpiper’s suggestion of Cred budgets for the Discord to put a limit on this. Is there a reason we’re not doing this already? This would also allow us to place a floor on the Cred distributed to GitHub minted Cred. We could, say, guarantee that even if you can’t be bothered to ‘shill’ your work, you’re still guaranteed some amount of Cred per the existing system.

  • Older contributors are way down. Presumably, as @META_DREAMER points out, because their contributions were made before #props and didathing came into use. I think this makes sense from a strategic perspective. We have an influx of new contributors, and I think it’s a good idea to invest in them. However, a central promise of SourceCred is that it fairly rewards contributors over time. If you make a contribution that is later found to be valuable, you get more Cred. By devaluing past contributions, it does the opposite of that. This could affect the long-term incentives. Will current contributors contribute less, knowing their past contributions could get downgraded in the future, even if they’re as valuable or more to the project? I agree with @META_DREAMER that time-weighted Cred makes a lot of sense here. I can imagine this coming up repeatedly a lot in communities using SC, for a number of reasons. Large changes in Cred weightings, changes in contribution patterns, forking of large, mature projects in which 99% of the Cred would go to the forked project, etc.

Overall, excited about this change. Would rather see us dive into the hard work of valuing unseen contributions now than later when it’s even harder (or impossible potentially, leaving a big part of SC’s mission behind). I do think there are some valid concerns though, and that we need the tools and will to build and course correct as problems crop up. E.g. curbing Black Mirror popularity contests, runaway Cred inflation or gross over/under valuing.

6 Likes
22

Partial response:
Looking for direct responses please

Firstly - Dandelion you where the TBD crown well and this is evidence of that - I thank you.

1. Why do roles increase cred? Until roles become attainable - or their attainable-ness is made visible in the community - I think it may be counterintuitive to assign them more weight. It gives people with the most power more capital in a system where attaining leadership is not well figured out.

2. random-chitchat is a highly emergent space and as such should be weighted higher and certainly not depreciated - i believe we can continue to train ourselves (like we do throughout SC) to not assign cred to nonesense things or truly random conversations but also this low-stakes platform allows for real gems to transpire that then stand out via their cred assignment - it is a very useful space

Clarifying question: are other channels not mentioned because they don’t have weights or because weight changes are not being proposed to them?

Sidenote: @blueridger the format of these weight charts - this is the info that would be great to be able to access via bot on discord.

3 Likes
23

There’s a couple of things that concern me about this proposal.

I’m concerned the proposal:

  • … improves non-technical contributor Cred by coincidence rather than design.
  • … might substantially promote centralization of Cred.
  • … introduces additional gate-keeping to the community.
  • … will cause stronger bias towards recent contributions over time.
  • … causes a double-whammy focus on recent contributions, as the grain distribution also shifted towards it.

Now to be frank, reaching out and raising objections at all from a -40% position kinda sucks. And I’m feeling forced to include disclaimers for my position here even though I intend to be mostly objective.

So here goes: I do agree developers such as myself in particular had a historical unfair advantage because of better platform support. As far as I’m concerned, fixing that bias is important, fair and welcome. By extension I’m expecting such a re-balance to mean I will lose some Cred, I’ve got no issues there.

However my theory is these new parameters benefit recent like-minted Cred, and this appears to correct Cred earned for non-technical contributions by coincidence, since they have recent likes.

This has to do with Cred inflation. With activity-minted Cred (like on GitHub), if the size of the community is constant you would expect a constant Cred inflation rate, because new contributions come in over time. This is fine, because in relative terms things should line up with the relative work you’ve put in.

You’d expect if the community grows in size linearly, the Cred inflation rate for activity-minted Cred would also increase linearly. The relative shares of past contributors could shrink from this, and this is fine because it roughly lines up with the larger community possibly getting more work done than in the past.

However for a linearly growing community, I’d expect like/reaction-minted Cred to increase Cred inflation exponentially. For +10 contributors, that’s +10 people’s worth more did-a-thing messages coming in, but +10x10! more Cred minted from reactions on did-a-thing compared to before.

If I’m right, this problem would exist regardless of this weight change, and ideas like sourcing Cred for likes from a pool instead have been around for some time to counter this. But this is why I think increasing the weights for like-minted Cred simply exacerbates a disproportional focus on recent contributions. And it’s by sheer coincidence that recent Discord users also has better contributor diversity.

Anecdotal evidence for this I think is comparing my change to @mzargham’s change in Cred. If my undeserved GitHub Cred benefit represents ~42% of my all-time Cred. Then it makes no sense that mzargham’s would be ~39%. If anything I feel like he should be gaining Cred for historically undervalued work that isn’t on GitHub. What we do have in common though is a low amount of recent contributions. And the “center of gravity” of when we earned Cred is roughly the same time period (last year-ish). Suggesting it’s tied to recentness more than the platform.

Note that this would give any old-guard contributor a perverse incentive to stifle the growth of the community as well.

My other great concern is with centralization and gate-keeping.

Fundamentally, a big reason you’d want to use SourceCred at all is to eliminate a lot of gate-keeping, such as in traditional employment/freelance situations. So in my mind reintroducing any form of gate-keeping requires a strong argument or lead to valuable experiments.

I think filtering no-role reactions could be a valuable experiment to learn about whether this improves Cred quality and offers any Sybil attack resistance.

But I fail to see why having increasing weights for different tiers in the social hierarchy:

  • Is even necessary at all.
  • Is not equivalent to manually approved reward-tiers (like employment is).

Back in my days (lol), the core/contributor/community roles weren’t formalized. But even having an informal status of core contributor and being 3rd on the “leaderboard” in my experience gave me a snowballing potential over others. It comes with more visibility, authority, easier access to other influential people (at CredCon for example) and easier access to off-the-record information. Those benefits compounded my ability to earn even more Cred.

Certainly I didn’t feel like core contributors need additional help to secure a fair amount of Cred for their work. At the same time, my case showed a new contributor was able to work their way up to core without knowing anyone on the team before. So (at least for a developer) the previous situation delivered on the promise of reduced gate-keeping and being meritocratic.

So as a counter-proposal why not have a role “crediquette-diploma” with 1x, and grant it to anyone we’re confident is not a Sybil account and has a basic understanding that their reactions meaningfully change Cred calculations. With 0x by default. In order to experiment and find out if it warrants the heavy measure of gate-keeping for this.

Finally I have a vague intuition that an escalating recent Cred bias (my theory above), may cause more centralization of Cred as well. If older contributions’ relative Cred decays faster-than-natural, it implies old Cred that was flowing to unpopular nodes lose traction faster-than-natural as well. I worry this could penalize people whenever they’re not strongly connected to the recent high Cred people.

8 Likes
24

As someone who’s a bit more critical of this proposal, I feel it’s my responsibility to offer alternatives that we can work from.

Back in December, I wrote this post addressing this issue, with a proposed means of solving it, which I’d like to bring to light here.

In that post, I also used props and did-a-thing as means of recognizing “soft cred” (i.e. non-plugin cred). However, the main difference is that I sought to create rough weights for non-plugin contributions, such that they could be essentially “plugin-ified”.

For instance, leading a meeting, self-care, or providing guidance could have some rough cred minting, just like GitHub commits, emoji reactions, or Discourse likes. This would involve coming to consensus about these weighting. This seems to massively reduces the problem of popularity contests, abuse, personality type biasing, and unnecessary conflict about recognition.

In addition, it also allows us to cleanly reweight these types of contributions. For instance, if we want to tweak meeting cred later, we can do it, because we’ve tagged/identified it directly. The proposal as is will struggle with this, because it doesn’t recognize different types of props (only the popularity of it), will make retrospective action much harder.

Ironically, the situation we’re in is evidence of this. We are having a tremendously difficult time figuring out how to reward community cultivation and care work because we have not identified it in the system.

3 Likes
25

I love the astronaut imagine. EVERY. TIME. I love it.

26

+10 on the idea that we need to recognize this contributions in the graph more explicitly to better flow cred for care work / non dev work. If we increase the weights in props/didathing, we should prioritize implementing a better / more fair way of valuing those posts, like how @blueridger has described in their post.

Additionally, there’s a blindspot we have in SC since we’ve been a relatively slow growing community: the current props/didathing setup causes cred inflation and extreme variance in the “cred to contribution value” ratio as the community grows. MetaGame has grown a lot lately and there’s a lot of things that we should proactively fix / learn from, e.g:

Taking a qualitative/average instead of quantitative/sum approach to counting emoji reactions will result in higher signal / quality cred flows.

We can setup multiple different discord channels for different types of contributions (e.g. “did-a-care-work-thing” and “did-a-product-thing” and “did-a-ops-thing”) to better value certain types of work by design rather than coincidence. I’m imagining these entries could become “tagged” nodes in the creditor as well for Cred historians to link them / categorize them further.

And IMO we should try executing on this ASAP and also not hold back on the cred updates from going through.

4 Likes
27

Thanks everyone for the thoughtful and civil discussion. I’ve decided that we will be moving ahead with the spirit of this rebalancing, with some adjustments to mitigate Cred inflation. In light of all the questions raised here, I’d like to more clearly articulate why this change is important, and why we’re making it.

The SourceCred community is a fusion of two rather distinct perspectives, and groups of people. We might call these the “Coders” and the “Cultivators”.

Since SourceCred’s inception, coders have enjoyed many advantages compared to cultivators. They have structural advantages; they know GitHub. They have economic advantages; they can more easily afford to work in exchange for uncertain payments. They have cultural advantages; their labor is seen as intrinsically legitimate in a culture where cultivators’ work is, by default, invisible.

They also have, and have always had, a dramatic bias in their favor within the SourceCred algorithm. We have always minted Cred automatically based on GitHub activity; in the beginning, GitHub was the only source of Cred. As such, by default, the bulk of coder labor automatically earns Cred. This is a stark contrast to cultivator labor; by default, it earns no Cred at all.

The #props channel has started to change this dynamic. For the first time in the project, we have a way of tracking most contributions which is not dramatically biased towards coders. It’s not particularly biased towards cultivators, either. Props is a more neutral platform. Anyone contributions can be seen by it.

What about Discourse? Discourse is also a neutral platform. However, it is niche: most contributions, whether Coder or Cultivator, don't correspond cleanly to Discourse posts. As such, we can't really scale up Discourse as a main source of Cred for contributors. In the past, setting very high weights on Discourse has been distracting at best, and disruptive at worst.

In principle, we could have implemented #props on top of Discourse instead of Discord. If we’d done so, then I’d be ramping up Discourse weights instead. At this point, however, we’re better off looking forward towards the Creditor rather than migrating #props to Discourse.

My weight change proposal, in substance, shifts the balance of Cred minting power out of the coder-native platform of GitHub, and into the neutral commons of our Discord. As we’ve seen, this is uncomfortable for coders, as they can no longer count on getting compensated automatically.

Getting valued by default feels like a “right”. Of course my labor should get seen! Of course I shouldn’t need to shill it to others! Isn’t that the whole point of SourceCred? Part of the promise of SourceCred is our vision of a world where people can just do great work and get rewarded for it, without needing to shill their labor or play politics.

Taking this away from coders feels like breaking a promise.

However, right now, to our cultivators, that promise literally is broken. It is not functioning. And the closest thing we have to working for them, imperfect as it is, is the #props channel.

Some coders have said, in effect: Sure, the system isn’t working for cultivators. However, it is working for me. Why are you breaking my experience? How does that benefit cultivators, really? Why don’t we just keep things working as they are, but also build a new system that works for cultivators?

The simple answer is that the coders are the ones with the power to modify and change the system. If coders are uncomfortable, they can directly build something better. Frankly, it’s good if devs are uncomfortable when the system doesn’t work. This is the essence of dogfooding. This is the path by which we genuinely build a system that works better for everyone.

This change does create new biases. It creates a bias towards currently active contributors, because the #props channel hasn’t been around for long. It’s biased towards popular contributors, who are more likely to rack up emoji reactions.

Looking at the situation holistically, the trade in biases is worth it. The set of people losing Cred due to the recency bias mostly benefitted from pro-coder bias, so I see this as a clear improvement in equity. However, we are going to keep our new biases in mind, and work to correct them as we improve SourceCred.

This discussion has thrown into light many of the problems with props. It’s also kicked off a lot of great suggestions on how we can improve it. This is great! Let’s keep this creative energy going, and bring it into our next collective arc, namely building the Creditor.

I believe the Creditor will be the solution to the root problems we’re seeing here. It will enable anyone to play the role of “Cred Historian”, finding valuable and under-appreciated contributions from any part of the project and boosting them. And the Cred Historians will get compensated for that labor, meaning that coders who just wanna code–or cultivators who just wanna cultivate–can leave the shilling to the Cred Historians. Once the Creditor is operational, we’ll turn off (or greatly reduce) Cred minting in all the existing platforms. So don’t worry that the future of SourceCred is hyper-props-ization. It’ll actually be much better.

Adjustments

Plugin Budget Weights

The Cred-per-person minted by #props is proportional to numPropsPerPerson * numReactionsPerProps, meaning that the Cred grows quadratically as usage of props increases. As of this change, there is already enormous amounts of Cred being minted in the Discord plugin, and if props usage intensifies, it would likely completely swamp all historical Cred.

Here is a bar chart showing Cred minting by plugin across time, for both the current mainline weights (left) and the new proposed weights (right).

image
image2082×794 73.6 KB

Just with a glance at that chart, you can see that under the original proposal, recent Discord Cred is a giant fraction of total Cred. But it’s hard to tell how much. So, the chart below re-normalizes to show percentage of total Cred being created. (Thanks @wchargin for this suggestion). If a blue bar has a height of “5” in the chart, that means the Discord plugin produced 5% of the project’s total lifetime Cred in that particular week.

image
image2020×748 58.6 KB

In this chart, we can see that there were two weeks in January in which the Discord plugin was minting close to 10% of project total Cred per week. To put this in context, those two weeks in January minted about as much Cred as the first two years of the project, from Feb 2018 - Feb 2020.

As discussed in the post above, I’m OK with introducing a significant (but ultimately temporary) bias towards recent contributors. However, it doesn’t serve us if the Cred inflation goes completely out of control. As such, I’m going to set a plugin budget (effective retroactively) that no plugin can mint more than 5,000 Cred per week. Why 5,000? If you look on the left chart, you’ll see a single giant spike of minting in early 2020, when we minted 5,000 Cred following CredCon. We used the initiatives plugin, which is kind of like a proto-Creditor. So we’ll treat that as a “high water mark” for how much Cred we mint per week.

Thanks to @Beanow for surfacing this concern.

Discord channel weights

@Jolie_Ze suggests leaving the random-chitchat channel weight unchanged, and just trusting folks to use the emojis intentionally. I’m think biasing towards trust is a good habit, and there haven’t been any particular issues there. So I’m going to leave the random-chitchat channel weight unchanged at 1x (the default for all channels).

Non-bugfix: Role-based weights

I’m leaving the Discord role-based weight configs in place. I hear the concerns that this represents gatekeeping, and that how roles are assigned isn’t legible. However, I also think the roles are being reasonably well maintained (even if illegibly), and do correspond to community trust. This is a valuable defense against Eternal September style issues. I believe the solution is not to remove the roles, but to set up a transparent and accountable process for maintaining them.

Next Steps

I’m going to implement these changes within 7 days, so the updated Cred will be in effect, either this upcoming Grain distribution, or the one immediately after.

Here are regenerated Cred change tables after the fixes:

All time
Name % Before % After Change
decentralion 27.4% 21.2% −22.7%
wchargin 12.6% 7.9% −37.3%
lbstrobbe 6.5% 7.3% 12.4%
s-ben 6.1% 6.0% −1.8%
KuraFire 3.0% 5.6% 88.8%
hammad 4.7% 5.5% 18.0%
topocount 2.9% 3.9% 36.2%
beanow 6.1% 3.8% −37.1%
Thena 2.1% 3.5% 71.4%
bex 2.1% 3.4% 60.0%
sandpiper 1.7% 3.3% 96.9%
panchomiguel 1.9% 3.2% 67.7%
joiecousins 1.4% 3.0% 117.0%
benoxmo 0.6% 1.1% 84.1%
burrrata 1.7% 1.1% −36.1%
dependabot 1.7% 1.1% −37.2%
mzargham 1.3% 0.9% −28.6%
eeli 0.5% 0.8% 52.9%
youngkidwarrior 0.6% 0.8% 34.9%
Jojo 0.4% 0.8% 83.8%
yalor 0.7% 0.7% 6.5%
evan 0.8% 0.7% −14.4%
brianlitwin 1.0% 0.6% −33.5%
ian 0.2% 0.6% 193.2%
vsoch 1.0% 0.6% −42.7%
cortanav 0.3% 0.6% 97.6%
kate-huntoon 0.2% 0.5% 200.3%
greenkeeper 0.7% 0.4% −45.7%
Simple-Poll 0.2% 0.4% 102.1%
peth 0.2% 0.3% 71.4%
4 Weeks
Name % Before % After Change
Thena 11.5% 10.7% −6.8%
KuraFire 7.8% 8.6% 10.5%
lbstrobbe 9.2% 8.3% −9.8%
decentralion 9.0% 7.9% −12.1%
hammad 5.7% 6.5% 15.6%
topocount 5.3% 6.0% 14.0%
Jojo 4.9% 5.8% 19.6%
s-ben 5.2% 4.3% −17.9%
joiecousins 4.0% 4.2% 5.0%
sandpiper 3.1% 3.5% 12.6%
eeli 3.1% 3.4% 11.2%
wchargin 1.7% 3.1% 81.9%
peth 1.0% 2.1% 104.9%
Willow 2.2% 2.0% −12.4%
kate-huntoon 1.0% 1.9% 82.3%
panchomiguel 2.8% 1.8% −37.0%
Felix 1.5% 1.6% 12.6%
tua 0.4% 1.1% 170.3%
bex 1.2% 1.0% −16.0%
echojuliet 0.0% 1.0% 1954.5%
BrianBelhumeur 1.0% 1.0% −4.8%
dependabot 1.7% 0.9% −48.2%
prose11 1.1% 0.8% −22.1%
kstone 1.0% 0.8% −22.5%
benoxmo 0.7% 0.7% 7.1%
Marcie 0.7% 0.6% −5.6%
owocki 0.2% 0.6% 197.8%
youngkidwarrior 0.6% 0.6% 1.2%
Jolie-Ze 0.5% 0.4% −29.7%
eeli-discourse 0.6% 0.4% −38.2%
12 weeks
Name % Before % After Change
decentralion 11.7% 10.8% −7.9%
Thena 9.7% 10.2% 5.4%
lbstrobbe 8.0% 8.5% 5.5%
KuraFire 7.7% 8.4% 8.5%
topocount 6.8% 6.5% −4.3%
panchomiguel 5.8% 5.0% −13.0%
hammad 4.9% 4.9% −0.2%
s-ben 5.1% 4.7% −7.5%
joiecousins 3.9% 4.4% 14.3%
bex 2.9% 3.6% 21.4%
sandpiper 2.9% 3.2% 11.5%
Jojo 2.3% 2.8% 20.1%
eeli 2.8% 2.7% −2.6%
kate-huntoon 0.9% 1.8% 96.1%
wchargin 1.7% 1.7% 0.5%
yalor 1.1% 1.4% 32.8%
benoxmo 0.8% 1.3% 51.8%
dependabot 2.8% 1.1% −60.1%
cortanav 0.7% 1.0% 43.2%
Simple-Poll 0.6% 0.9% 51.3%
Felix 0.9% 0.8% −1.8%
peth 0.4% 0.8% 78.0%
youngkidwarrior 1.2% 0.7% −38.2%
Willow 0.9% 0.7% −22.0%
tua 0.4% 0.7% 74.8%
BrianBelhumeur 0.7% 0.6% −9.5%
pablomendez-95 0.2% 0.5% 124.0%
lanski 0.2% 0.5% 118.1%
prose11 0.5% 0.5% −12.0%
echojuliet 0.1% 0.4% 454.7%
52 weeks
Name % Before % After Change
decentralion 20.9% 16.5% −21.1%
lbstrobbe 9.0% 8.6% −3.9%
KuraFire 4.5% 7.0% 56.4%
hammad 7.1% 6.9% −2.2%
s-ben 7.5% 6.6% −12.8%
topocount 4.4% 4.9% 12.9%
Thena 3.1% 4.4% 42.0%
bex 3.2% 4.2% 34.3%
sandpiper 2.5% 4.1% 63.2%
panchomiguel 2.8% 4.0% 40.5%
joiecousins 2.1% 3.7% 79.8%
wchargin 3.8% 2.9% −23.7%
beanow 4.7% 2.5% −46.3%
benoxmo 0.9% 1.4% 52.6%
dependabot 2.5% 1.3% −47.8%
eeli 0.8% 1.0% 26.7%
youngkidwarrior 0.9% 1.0% 11.8%
Jojo 0.7% 1.0% 52.3%
yalor 1.0% 0.9% −11.7%
ian 0.3% 0.8% 180.3%
cortanav 0.4% 0.7% 63.7%
kate-huntoon 0.3% 0.7% 148.9%
evan 0.9% 0.6% −28.6%
mzargham 1.2% 0.6% −50.2%
Simple-Poll 0.3% 0.5% 67.5%
peth 0.3% 0.4% 42.0%
burrrata 0.8% 0.4% −46.9%
crisog 0.2% 0.4% 131.4%
brianlitwin 0.8% 0.4% −52.4%
doctorrobinson 0.4% 0.4% −2.2%

Thanks to @wchargin and @s_ben for comments on drafts of this post.

7 Likes
28

I like the revised version a lot better! Few questions though:

  1. When the budget for a given week is reached, what happens to additional cred? Does the 5000 just get spread thinner or does the plugin stop minting cred after 5000 cred has been minted?

  2. Do we expect to keep a cap like this for the foreseeable future? e.g. If we have 10 new active contributors join over the next month or two, will cred still be capped at 5k and therefore introduce a negative bias against recent cred?

  3. Can we get a chart of what the revised cred minting over time looks like?

  4. You mention that we would turn off or reduce cred minting once the creditor is there. Would that change also be retroactive, or would we introduce time-scoped weight changes at that point?

4 Likes
29

So lately, there’s been some talk about double-dipping and changes to crediquette that this weight change will result in. It seems taken for granted that this weight change is drastic enough that the crediquette will change to developers using the props and did-a-thing channels for work on github, since the github plugin cred will be trivial. If that’s true, then the current proposal actually still favors the devs, because they will still be getting github cred on top of their did-a-thing usage. If the idea is to shift crediquette in this way, why are we giving github plugin cred at all? The only reason I can see is retroactivity, which is breaking anyways.

30

I would argue for keeping some “activity minted” Cred on GitHub. There are just going to be far too many PRs, many of which small, unsexy ones, that don’t get enough Cred. And I doubt devs want to shill every PR, or that the community wants to be spammed with every PR. Also, it keeps somewhat of a check on popularity contests, gaming, and subjective biases. My understanding is that the minting on GitHub is low enough that devs can’t just rely on that and get meaningful income, creating a strong-enough incentive to post in chat. Open to that understanding being challenged though if others think the minting is too high.

2 Likes
31

This might be true, but it’s impossible to say at this point. In a props/did-a-thing heavy system, everything will come down to the behavior in those channels. If devs post boring work that doesn’t attract much reaction, then this wouldn’t be the case. If they posted every PR and every review and did get reactions, then yeah it would get out of control.

Ultimately it comes down to the popularity, and I’d argue it favors glamor and popularity more so than dev work, given that the reactions still far outweigh cred from dev work.

That being said, I agree that if we’re really going after fairness, we might as well lean in and remove GH given your points. The fundamental issue is that one class has their contributions tracked and another does not. If we move away from this props-heavy approach, devs will be back in advantage they were before because their contributions on GH were tracked when cultivation work never was during this period.

Ultimately it seems like the only long run solution is to quantify weights on community and care work, but if we’re going to go props-first, we should do it right.

32

There’s some improvement there, but TL;DR I’m still rather concerned with this. Because of:

  • Parameter choice
  • Gatekeeping, centralization and bottlenecking, without enough justification
  • Switching from one invisible group to the next

First about the plugin budget cap. I’d like to echo @META_DREAMER’s question there.

When the budget for a given week is reached, what happens to additional cred? Does the 5000 just get spread thinner or does the plugin stop minting cred after 5000 cred has been minted?

Assuming it gets spread thin, I think this is only effective at curbing this escalating recency bias when it’s saturated most of the time. So using the 5k Cred from the CredCon anomaly seems like a very high threshold. It’s also an approach that’s sensitive to the overall weights, as that influences the “typical” absolute Cred inflation rate. For example, with the baseline chart ~1k Cred would saturate Discord most of the time this year. With the proposed-1 chart that would be ~2-3k Cred. So every weight change we’d have to fix this parameter as well.

Also, I think the goal here should be to preserve “the signal” when looking at these limiter functions. A ‘good contribution’ during a busy time shouldn’t be worth less cred than that same contribution would have had during a quiet time. A hard cap doesn’t do a great job in scenarios like these.

If we could do a two pass system, it would be interesting to see what the median reaction weight per post is per time window and mint Cred relative to this. That would offer a way to make like-minted Cred fairly constant and easier to set parameters for. And would award Cred that makes sense for the community size at that time. (e.g. a 10 out of 20 people react would be a stronger signal than 10 out of 200).

I’m not sure my suggestion was clear. What I’m proposing is:

Weight Current Proposed Counter proposal
role weights
no role 1x 0x 0x
lower barrier role (optional) 1x 0x 0x / 1x
@ community 1x 1x 1x
@ contributors 1x 2x 1x
@ core 1x 3x 1x

Although even for this, I think a strong argument is (perpetually) necessary to justify increased gatekeeping.

The existing hierarchy (community/contributor/core) doesn’t need to be removed. They can be used for access-control and social hierarchy as they are. The new role would be an easier to obtain one, between no role and @ community. But only if necessary to lower the gatekeeping threshold.

Sure enough, having an important role should mean increased trust (given they have increased authority and executive power). But I still expect them to be humans with human-like discernment and biases, with limited time on their hands. Transparency and accountability (while important) can’t resolve this.

In my mind, part of SourceCred’s overall premise is to use algorithms to support a set of values (using whatever governance), while simultaneously insulating from personal bias enough it consistently recognizes contributions that fit those values, in a way that can scale to outperform traditional models.

Given that we’re also relegating ‘automatic Cred’ to obscurity with these weight changes, we’re looking to be leaning heavily on the social recognition now. So I’m alarmed by a centralization of the Cred-minting power.

About that “limited time on their hands” this is not just gatekeeping / centralization, but also a review bottleneck. You’re losing out on substantial Cred when core doesn’t have the bandwidth to evaluate or simply missed your contributions. So ‘everyone’ is going to want core to review ‘everything’. Which I’m not expecting to scale properly, so it would amplify their biases because they are forced to decide what they’ll spend time on reviewing.

Sure Eternal September is a real concern, plenty examples exist in open source projects alone. However at the same time SourceCred’s culture isn’t finalized, it’s still in need of being challenged, refined, etc. Echo chambers and amplifying biases with financial consequences are equally a thing to fear for the culture’s health imo.

Overall it just seems like a regression to me. It’s a heavy measure that needs substantially more justification than a “while we’re at it” / “go big or go home”. If we absolutely need this centralization crutch, I’d like to see a much stronger clarification.

If this was a vote, this issue is why I’d vote against.

I’d also just like to point out we’re intentionally switching, from one undervalued group to a new one. The proposal is an uncomfortable choice between which bias is the lesser evil.

I’m not so worried about devs now having an “equally bad” experience of having to self-promote. I’m worried for those who struggle to do this. Social anxieties, conflicting moral values about shilling, or the higher noise and stress levels of Discord (full disclosure: this describes me atm). I think the #props channel alone will not be enough to support them. This could apply to the creditor as well.

Their situation before wasn’t great either. Only if they were comfortable working on GitHub (Discourse weights are currently not that high) would they have an alternative source of Cred. But by making socially-earned Cred the only viable option, I’d predict the new structural bias will be between the socially adept vs the quiet.

So for the future I really hope that we’ll find a well balanced mix of systems (or silver bullet!) for a wide audience, rather than hop from one invisible group to another.

Yes, and :]

Unsexy ones were already at a disadvantage because the sexy ones are easier to double-dip with. Having GitHub Cred at a reduced weight has been a useful baseline so they’re not left empty handed, but the unsexy problem probably needs special attention.

It’s an industry-wide deeply rooted issue (whether proprietary or open, paid or volunteering). It’s much easier to just churn features all day, even if they’re unfinished and buggy (assuming it gets approved), because there’s only so many people that read the code to call out poor hygiene, but a lot of people that can be hyped about a new feature.

Sadly even with a half-arsed job, features win the popularity contest hands down, because the audience is orders of magnitude larger.

It could even be a type of attack, where you try to get away with as many experimental features as the community will let you, and leave someone else to clean it up (with the opportunity cost of not doing sexy things during that time).

But a baseline Cred doesn’t solve it completely, and the unfairness of not having such a baseline outside of GitHub is clear.

5 Likes
33

I gotta say I’ve been agreeing too, like, why change a system that works for a certain kind of labor? An equally unfairly valued kind of labor.

The simple answer is that the coders are the ones with the power to modify and change the system. If coders are uncomfortable, they can directly build something better. Frankly, it’s good if devs are uncomfortable when the system doesn’t work. This is the essence of dogfooding. This is the path by which we genuinely build a system that works better for everyone.

Damn. This really is y’alls game. I’m grateful that @decentralion has a broad enough view of us all to be able to navigate this nuance of power and value. It’s basically incentivizing coders to make a better system for everybody… right? or for non-dev contributions*?

*(In the future, let’s move away from dev and non-dev and toward dev Contributions vs non-dev Contributions)

One incentive I see coming out of this is for coders to shape their dev-related contributions to the minting matrix around the needs of non-dev work. I can only see this happening by learning more about what non-dev work is in SourceCred and how to make it more accessible. But basically, coders learning from folks who engage in (mostly) non-dev contributions what they do and what they need. The incentive being… that if they solve the non-dev contribution problem then they can return to their pre-prosposal GitHub contribution weights? Wait, I’m confused. Unless… is the point that coders are put in an uncomfortable position in order to create change?

Sidenote: Cred historians sound awesome and if we’re going to have them I ask that their role be described and perhaps we do some training for that function.

34

I think that we actually do mean to talk about contributors here, not just contributions.

The guideline to focus on contributions is great when discussing the valuation of individual contributions, or even classes of contributions like “pull requests”. But what we’re talking about here is more cultural. It’s about power dynamics. There are people who have been historically well compensated and who have the unique superpower to make direct changes to the systems that pay us all. And there are people whose work has been undervalued—not just contributions, even the existence of the work—and to whom the system is as opaque as it is to anyone on the street. This problem involves humans, emotions, and identities. To tackle it effectively, we must consider all those factors.

At the end of the day, “contributions, not contributors” guides us toward respectful and constructive discourse. In this context, we need to depart from the rule’s letter, but the good news is, I think, that we’ve done a good job of following its spirit. I’m among the people who have been called out by name, and I felt included, not targeted. Let’s continue to have healthy discussions on difficult topics: we have to!

3 Likes
35

But what we’re talking about here is more cultural. It’s about power dynamics. There are people who have been historically well compensated and who have the unique superpower to make direct changes to the systems that pay us all.

I’m among the people who have been called out by name, and I felt included, not targeted

This delineation and identification is very affirming and validating to me as a non-developer. Thanks for the clarity.

36

Hey all, just closing the loop here: the Cred weight changes have now officially landed as described above (see implementation).

To answer questions that have come up along the way:

Once the budget gets reached, the Cred gets diluted to stay within budget. For example, if a given week had 10,000 “raw” Cred worth of props, then every reaction would have a uniform 1/2 weight multiplier.

I expect to keep a cap like this in place for automatically plugin-minted Cred essentially forever. I’d like us to move the bulk of Cred minting into a system with more robust governance and incentive design, e.g. the Creditor. Under activity-minted Cred, there’s basically an incentive for the whole community to game the metric by inflating the activity level, which means it isn’t very robust.

If we take a very long time to implement those better minting mechanisms, then the 5k weekly plugin budget will start to create frustration and a pressure to change the system. I think this is a feature, not a bug.

Here you go:

image
image1876×774 60.4 KB

You can find the source data and all the before-after comparisons (for the finalized changed) in this notebook, although be aware its a bit of a mess. There’s a lot more data analysis there that I used in exploration but didn’t publish in this thread.

I think we’d make that change retroactively. Once our “Cred Historians” have flowed Cred back to the historical contributions we wouldn’t really need activity minting in the past anymore. But we’ll cross that bridge once we come to it.

This is a reasonable question, and I considered this approach. My sense is that this change I originally proposed is big enough to suit my intent, and escalating in response to pushback didn’t seem very responsible.

Feel free to make a separate thread proposing this change, along with the analysis and rationale. If it gets broad buy-in from the community I’ll look favorably on ratifying it.

2 Likes
37

Respectfully, I don’t think that’s an appropriate followup.

From a TBD role a change has been proposed, feedback was requested, and by TBD privilege it has been implemented. Responding to that, I’m asking for clarifications and expressing concerns.

Rallying support for an alternative is a last resort measure should I’ve lost confidence and feel the need to try and overrule or bypass a TBD choice. I don’t see reason to do that. I have faith it’s all in good confidence and there’s time to discuss.

Besides, I feel like my other questions wouldn’t be heard if I moved just my alternative idea to a new thread, while they’re perfectly on topic here :]

As such I’ll repeat and summarize my previous position.

I think this is a heavy change including:

  • Disproportional recency bias.
  • Parameters I can’t explain (such as 5k per plugin when it doesn’t saturate the cap).
  • Great concerns about gatekeeping, centralization and bottlenecks.
  • Risk of excluding a new group.

But primarily the increased weights for higher social roles are concerning to me.

An irony I hadn’t pointed out yet:

Besides dogfooding, this is about power dynamics. Those who already have superpowers shouldn’t also have a Cred advantage. But we’re giving the roles such as core who have a different set of superpowers an extra Cred advantage in the same PR?

  • Why wouldn’t this cause the same issue all over again?
  • Are we preemptively doing this or has an actual Sybil / Eternal Summer issue occurred?
  • Are increased authority and moderation options not sufficient in our near-term trust level?
  • Do you think the 1x-2x-3x role weights had the urgency to need including in this change or could it have been discussed more.

I’ll stop myself there for the moment.

1 Like
38

I think the missing perspective here is that being personally responsible for every proposed change to Cred is a lot of work. While it’s not obvious from the outside, in the week after posting this thread I stayed up until 10-11pm multiple nights doing the data analysis and parameter exploration to try to understand folks’ concerns and make appropriate changes. I would guess I’ve spent somewhere between 20-40 hours of work on this over the course of the thread.

In response, I’ve gotten a lot of pushback from both sides of the aisle. I’ve been told this proposal goes too far, and also that this change should have already happened a long time ago. I’ve spent a lot of energy engaging with all of these concerns, and trying to update the proposal to suit the needs of our community. I’ve also reached the point where I’m burned out on this decision, and don’t have additional spoons to sink into it. At that point, I judged the proposal good-enough to be an improvement that’s worth making, ratified, and merged it.

I feel like you’re telling me: “you cannot stop working on this until my concerns have been addressed”. However, you’re not offering to do any of the emotional, analytical, or political labor to justify your change, and bring the community onboard.

Putting all of this work on one person is unsustainable, especially as the community grows larger and these changes become more complex. Also, having the same person act as proposer and decider is not robust, because it means that when the proposer gets burned out (as I did), there’s more likely to be a rush decision. Therefore, we need to decentralize this process. I intend to decentralize this by stepping back from proposing any further changes to Cred.

Your primary concern actually is off topic here. This change is focused on making Props a much larger part of the Cred, as a way of improving equity for cultivaiton folks. The new weights on Discord roles are an incidental safeguard. If you’re concerned about the way this concentrates power and sets up unaccountable ways of awarding that power, a separate thread will be a much better place to dive in. I suspect that others, like myself, are exhausted of this thread, so your concerns are getting less attention here than they would if they stood on their own.

So, here are my takeaways:

  1. This proposal has been ratified and merged. I do not have the bandwidth to make any further changes to it. However, that does not mean the decision is “final”, since it’s easily (retroactively) reversed by a later change in weights.
  2. We need to decentralize responsibility for making changes to Cred. To put this in practice: I am not going to propose any further changes to the weights, except in case of emergency. If we do a good job of avoiding Cred emergencies, then this will be my last Cred-change proposal as TBD.
  3. Everyone else in the community is explicitly invited to propose their own changes to our weights/config. It’s the responsibility of the proposer(s) to analyze the change, respond to criticism, and build consensus. Eventually, we will develop a formal governance system for ratifying such changes. For the time being, I will ratify decisions using my TBD hat. My intention is to ratify changes with broad support, and make contentious decisions sparingly.
7 Likes
39

Thank you for sharing that @decentralion.

I’m sorry if I made you feel unseen in your work. As well as for pressing the matter when you’re at your limit. I definitely know how much effort these things take. I didn’t know you were at your limit.

That wasn’t what I had in mind, but I understand it comes across like that. Since I did press and did talk about TBD responsibility. I could have expressed that a lot better.

My feeling was I had 2 options:

  • Engage in political struggle to push for my concerns regardless of your stance.
  • Raise concerns about the current change and leave it up to your discretion.

I got worked up, because I felt like I was pushed aside by being told to go take the first option. Aside from already being heated from worries about the content of the change.

This wording pains me a bit. So far I’ve also put about 10-16 hours in this thread and had 2 sleepless nights over it. But I feel like there’s the suggestion I’m less invested because I’m only critiquing.

However I do think there’s some truth. I’m not offering to take on the burden of bringing changes myself. My last attempt led to an ineffective too many captains situation and was the main contributor I got burned out working on SourceCred. There’s still trauma here that makes me very apprehensive to be any kind of political torch-bearer ever again.

Saying that, it’d be a double standard to expect from anyone else to do that on their own. So I can only agree with looking into governance models that distributes this burden.

6 Likes