Thanks everyone for participating in this discussion. On a meta-level, Iām glad to see this conversation happening, uncomfortable parts and all. The difficulty around deciding what āfairā is for a system like this, and negotiating changes to it in a way that lets people still feel respected, enfranchised, and heard, is one of the biggest challenges for us (and for anyone who uses SourceCred). As @anon60584824 mentioned, the āhuman bitsā of deploying a system like SourceCred are what makes it really challenging.
I really appreciate that in this thread, everyone has made a clear effort to respect and care for āthe human bitsā of other SourceCred contributors: working to understand othersā perspectives and intentions with good faith, hearing when feelings have been hurt and apologizing, and generally being supportive and honest with each other. Our ability to thrive is going to depend on our ability to support each other in these ways, just as much as it will depend on our ability to implement increasingly clever algorithms.
With that said: I think a lot of the challenges that have been surfaced in this thread reflect the immaturity of the SourceCred algorithm and parameter system. The promise of SourceCredās graph-based approach is that we can mint cred at the level of things we really care about ā goals like deployability (as supported by having a docker container and GitHub actions) or documentation (as supported by having a bevy of Getting Started guides on the Discourse). Once we have a supernode that represents the high level goal, we can tune the cred flow for the high level goal, and that will propagate out to everyone who worked on it.
However, we havenāt finished building the system that lets us do this. @Beanow is hard at work on the initiatives plugin, and afterwards we need to refactor the Graph to give us better APIs for controlling cred minting. It will probably be somewhere between āweeksā and ātwo monthsā before the more sophisticated, goal-based cred minting process is online.
While weāre waiting for that better system, we have an extremely crude alpha version, which mints cred solely on the amount and type of activity that SourceCred observes. As such, we mint cred for every Discourse post, for every GitHub pull request, and so forth. We can tune the cred minting (and thus the relative cred weights) by changing those overall weights.
This is a really crude system, kind of like hitting the cred distribution with a cudgel. Any weight we set for Discourse posts will either be too low for super in-depth guides and detailed discussion of the system, or too high for a placeholder post retroactively initializing an initiative. Any weight we set for GitHub pulls will either be too low for an involved refactor which increases Discourse performance, or it will be too high for a tiny PR to remove a deprecated file.
Ideally, we would already have next-gen system online, and I wouldnāt need to make crude and contentious changes to the weights. But, weāre not there yet, and I have an ongoing responsibility to tune the system to try to accomplish a few goals:
- Have the overall cred do a reasonable job of representing the value that all contributors have been providing
- Have the incremental cred create reasonable incentives for future work, so that as people are implicitly guided by the scores, theyāre also contributing healthily to the community
When I first turned on the system, we had more than a year of GitHub contribution history, and very little activity on the Discourse. When I gave the system what I considered āreasonable weightsā, it responded by giving almost all of the cred to myself and @wchargin, due to our very long history of code contributions. However, I felt it was important to cultivate the nascent Discourse community, and give meaningful cred to people who had started contributing in non-code ways.
So I deliberately chose weights that were unreasonable: a Discourse thread had 8x more weight than a GitHub pull request; a Discourse post had 32x more weight than a GitHub comment. Looked at directly, these weights didnāt really make sense. However, choosing these weights made for the cred distribution I felt was healthiest for the project at the time. I donāt regret setting the weights this way, but I should have anticipated and more clearly communicated that they would need to once Discourse activity levels approached our GitHub activity levels.
Now, several months later, we have a thriving Discourse community that is accomplishing a ton: exploring and explaining how SourceCred works, coming up with norms and patterns for our community, dog-fooding new features like Artifacts and Initiatives. This is great! But it also means that, because I chose unbalanced weights earlier, our overall cred is becoming very unbalanced. Weāve reached the point where the cred earned from a single week focused on Discourse can outpace the cred earned from several months of work on GitHub.
While we are still working with a set of crude levers to tune the cred, I still need to use those levers to do the best I can to keep the overall system in alignment. In this case, this means a change to make the Discourse and GitHub weights more consistent. If we had the tools to do so, Iād prefer to change the weights prospectively, not retroactively, and āstand byā our past weights (perhaps up to the most recent week or two). Unfortunately, we just donāt have that capability yetāand the result is a fairly disruptive change to cred.
The good news is, just as we can retro-actively update the cred now, we will retroactively update the cred with the new tools we develop in the coming months. Therefore, you can think of this current cred distribution as a temporary āworking copyā that weāre going to keep improving. And the way Grain distribution works always tries to keep alignment between the most recent Cred and the lifetime Grain flows; therefore, people with too little cred right now will be āmade wholeā once we launch the improved system.
Weāll also launch a new UI which allows exploring cred at the level of initiatives, artifacts, and other supernodes. So rather than SourceCred explaining cred in terms of raw activity, like so:
It will be able to communicate my cred in terms of specific initiatives or artifacts Iām connected to (like setting up the core graph module and algorithm, or my leadership roles within the project as a whole). I think that will help these discussion a lot, by making it easy to see why other people have Cred, in terms of the specific initiatives and artifacts they supported.
This wonāt be a perfect system either, and weāll keep on having friction as we work on SourceCred. But I hope by improving the algorithm, we weāll be able to make more intentional changes to the system that wonāt be as frustrating or lead to so much churn in the cred.
In the mean time, weāre still very much in CredSperiment mode, so we need to accept there will be some turbulence in deploying our still-experimental and alpha quality system. Thanks for being along for this ride, yāall!