Based on the feedback here, I’ve come up with a new set of weights:
| Node Type | Old Weight | New Weight |
|---|---|---|
| Discourse Topic | 8x | 8x |
| Discourse Post | 2x | 2x |
| GitHub Pull Request | 1x | 16x |
| GitHub Issue | 1x | 4x |
| GitHub Comment | 1/16x | 1x |
| GitHub Review | 1x | 8x |
These new weights incorporate two major points of feedback:
- It feels cleaner to leave Discourse weights as they are, and only boost GitHub weights
- GitHub PRs should be valued more highly than issues
I also gave a relative boost to reviews, as code review is very important.
I’ve also taken a stab at my suggestion of changing the lens, by building a “prototype UI” that analyzes the weights through the lens of seeing what contributions we value, instead of the lens of which people gain cred. The results were illuminating.
Here’s a set of four bar charts showing the cred-by-activity-type under the old weights (left) and the new weights (right). It also shows the total cred across all time (top) and just the cred for the last full week (bottom).
Here are the things that really stand out to me:
- Under the old weights, all GitHub activity got less than 5% of the recent cred. This is crazily low given how important development is to the project right now
- Under the new weights, GitHub activity gets 20% of the recent cred. This still feels really low to me, and suggests we might need still higher weights on GitHub
- Under the new weights, GitHub activity has grown to about 75% of the total cred. This makes sense to me, because the GitHub history has been going on much longer, and is disproportionately important.
I propose that for now, we focus on getting the “latest cred” totals right first. The reason is this puts us on a tighter feedback loop: each week we have a pretty clear memory of what happened in the week, and how much we value it, so we can more easily come to agreement on what cred “makes sense”. Once we’ve gotten good at latest cred, we can orient on the harder problem of all time cred.
Looking at it through this lens, we still have room for improvement – rewarding development activity with only 20% of the latest cred is too low, especially considering how blocked we are on development. However, since 20% is much more reasonable than 5%, this weights change looks like an improvement.
Acting as TBD, I’m going to merge the new weights, and distribute corresponding payouts, so we can keep to our normal payout schedule (+/- 12 hours). However, I’m happy for this discussion to keep going. The CredSperiment is… an experiment! And we can change the weights every week if we want to. (Eventually we will settle down and Cred will become more stable, but that’s still a while out.)
Of particular note–as we improve the underlying infrastructure (i.e. move away from activity cred), we’ll get ever more powerful tools for configuring SourceCred in a way that rewards the contributions that we need and appreciate the most.
Thanks to @wchargin, who made major contributions towards enabling cred analysis notebooks. The data analysis in this post was done in a prototype cred analysis notebook. 