Changing the CredSperiment weights

Currently, SourceCred puts a very high weight on Discourse activity relative to GitHub activity; for example, the weight on a Discourse topic is 8x higher than the weight for a pull request.

Node Type Weight
Discourse Topic 8x
Discourse Post 2x
GitHub Pull Request 1x
GitHub Issue 1x
GitHub Comment 1/16x
GitHub Review 1x

At the time I chose these weights, we had much less Discourse activity than GitHub activity, and I deliberately pumped the Discourse weights so that Discourse contributors would be meaningfully present in the overall distribution. Now that we’ve had a few months of increasing Discourse activity, I think we should re-balance Discourse and GitHub to be closer together.

We can see some need for this in the extreme results from the past week, in which @burrrata did a ton of work around organizing the forums and retroactively creating initiatives and artifacts–which also corresponded to creating a lot of Discourse threads:

@burrrata should definitely be appreciated and cred-ited for their work last week! However, for them to earn (say) 1/5th of @wchargin’s lifetime cred in a single week is quite extreme.

Overall, this issue will be structurally fixed by the supernode system, as we move to having cred minted from artifacts rather than minted based on activity levels. While we’re still working on that, however, I will be changing the weights to make our cred better reflect the reality of the project.

Here are the new weights I’m proposing:

Node Type Old Weight New Weight
Discourse Topic 8x 2x
Discourse Post 2x 1x
GitHub Pull Request 1x 2x
GitHub Issue 1x 2x
GitHub Comment 1/16x 1x
GitHub Review 1x 2x

I tried to be more consistent between GitHub and Discourse:

  • things that are raw posts or comments have 1x weight
  • things that start threads that will contain more posts or comments have 2x weight.

Here a snapshot of the cred that results:

One thing we can immediately see is that “peak GitHub cred” (that @wchargin and I generated together when we were pair-programming) and “peak Discourse cred” (from @burrrata’s work this past week) are fairly similar. This makes sense to me intuitively.

2 Likes

If you’d like to diff the full cred instance before and after the new weights, check out:

(In the future, we’ll have fancy cred analysis notebooks to dive into changes like this even more clearly.)

Overall, re-balancing the CredSperiment weights seems like a healthy thing for the community.

A few things to consider:

  • The newly proposed weight changes simultaneously reduce Discourse weights while increasing GitHub weights. This creates a drastic shift in Cred. Seems like if we want to adjust the Cred weights we should either decrease the Discourse weights or increase the GitHub weights, but not both.
  • It would be nice if we integrated a creation and review process for Discourse Artifacts (similar to GitHub PRs and reviews). This would enhance the contributor experience/rewards while also improving contribution quality. It would also make opportunities to earn Cred from raw Discourse activity at parity with GitHub. As is, there’s many more opportunities to earn Cred form raw activity on GitHub.
  • Writing code and then submitting a PR on GitHub often takes more effort than creating Issues. The current and new configurations give Issues and PRs the same weight. Why is this?
  • In the past (and I think present) some contributors have been paid reliable salaries and provided a safety net of healthcare and stuff while also receiving Cred. As Cred is translating to Grain this creates a bonus payout on top of past/current salaries. We want to recognize and reward all contributions to the platform, and building out the MVP of SourceCred was an incredibly valuable and awesome thing. At the same time, there’s a different risk/reward profile for those who are able to contribute while being supported to do so vs those who contribute on their own with no guarantees of immediate rewards, basic life support, or future Cred values (as Cred is currently very volatile and retroactively updated). Now that people are getting serious about contributing to SourceCred, Cred is being retroactively updated to favor past contributions, and Cred is potentially going to be used for voting and/or Grain minting/boosting (which could result in further consolidation of Cred) it’s important to consider these dynamics.
  • The way the SouceCred system currently works there’s no sense of time. We can’t say that weights were like X, then Y, then Z. We just update the weights and recompute the entire system from scratch. This means that at any point Cred can change, sometimes drastically. This is part of the design so no surprise. Cred changes, Grain does not. That being said, the proposed change drastically alters contributor Cred. Personally I would go from 13.6% to 4.5%. That’s a 3X decrease. @s_ben’s contributions would be valued almost as much as @vsoch. Is that an accurate representation of contributions, or is GitHub over-weighted? Looking at the weight changes overall, almost everyone’s Cred would go down except for a few developers. This is good because we want to encourage more developers to contribute. This is bad because part of the SourceCred culture is to value lots of contributions outside of development. The newly proposed weights skew highly towards development at the cost of other contributions. This recreates the value distribution biases that we’ve been complaining about in other projects for months on the forum lol
  • Currently there’s a binary distribution between platforms: code and non-code. This is mostly due to the infrastructure we’re using rather than cultural values. When we have Supernodes this will all be much easier (hopefully). Currently, however, GitHub is for work on the code while Discourse is a catch-all for everything else. This catch-all includes art, business development, documentation, protocol design discussions, fundraising and strategy discussions, community calls, Initiatives, Artifacts, and much more. We’re using Discourse to manage our project level operations, design and dogfood SourceCred mechanisms, and onboard new contributors. Seems important. Building the code for the protocol to work is also super important! We need to find a balance between the two.

I don’t have answers to all these things. TBH I’m still wrapping my head around how the SourceCred protocol works on a technical level and how that relates to Cred flows. As such, I’m not (yet) qualified to have a super strong opinions on how exactly weights should be configured. I do, however, have an intuition that the appropriate Cred weighting (at this time) is probably somewhere between the old version and the proposed new version. On top of that there’s some larger ideas here that are important to explore so that we can setup SourceCred for positive sum growth. Curious to hear what everyone else thinks as well as if this should/could be posted to the public forum :slight_smile:

hey @burrrata you made a lot of really strong points here, and I want to give some feedback / opinion as well!

I’m in total agreement here - adjustments should be more conservative. Maybe someone can comment on the reason that we would want to have the more drastic change?

+1. Pull requests are more meaty than opening an issue.

Wow really? It would be amazing if this model actually worked - I just assume it takes the “open source” model of doing it in your free time (alongside a job that offers such benefits).

Sorry to be the target of the “not fair” example, I don’t feel great about this :confused: The first idea you bring up is with respect to temporal data. Given that the formula changes, it would make most sense (if there is need to compare time points) to normalize based on some minimum / absolute values. So if everyone’s cred goes relatively down, this would still scale to some average value. I haven’t thought it through in detail, but something like that would make sense. Am I the “contributor chart widget” or “month’s of contributor time?” I hope you realize I’ve contributed (as a developer) to sourcecred far beyond the “chart widget” (which is largely done by @beanow and I just turned it into an action). I was heavily involved with a lot of the container work, discussion on issues and review. I feel a bit attacked here, especially because I’ve been also doing developer work for months, so I also feel the need to defend myself.

I’m curious - is SourceCred can be simplified to an algorithm, and you basically need to do operations over APIs and generate metrics /plot, why aren’t there different implementations to make it easier to use? If there were a Python implementation, it would be hugely easy for data scientists / researchers to use, for example. Docker helps a bit, but only for those that understand containers. I’m wondering if it would make sense to first create a language agnostic “Sourcecred algorithm” document to build other tools from.

1 Like

@burrrata maybe in the future you should think about whether or not you want to target individual people in your examples, especially if there is some negative implication (e.g., a “lesser” contribution is now rated the same.) It’s really hurtful to the person you are targeting, and I don’t see how it could possibly help the community (but rather hurts it).

First, I want to apologize that the post I shared was not clearer. The post I shared was meant to highlight some points that needed further thought and exploration. It was not at all meant to be a definitive statement.

Second, I by no mean intended to diminish the importance of your contributions. I’m not super active on the GitHub front so I’m not aware of everything going on in that domain. I saw the GitHub action repo (which is super cool btw!), but not much else. This led me to believe that GitHub was overly weighted vs other contributions. That’s the only point I was trying to make. There was no judgement about the quality or value of contributions, but that it seemed like a single contribution onGitHub was being valued more than ongoing contributions on Discourse (which I now know is inaccurate).

Third, sorry again for the confusion. I really value all your contributions. The entire goal of the post was to highlight how it’s important to make the Cred weights such that contributors (such as yourself!) are recognized and rewarded, but it seemed like the newly proposed Cred weights might consolidate Cred. There’s lots of people contributing a lot of value to the project in a lot of ways and it’s really important that we setup SourceCred to recognize and reward all those contributions. This way everyone feels valued and appreciated and we can create value together in a positive sum game :slight_smile:

1 Like

Agree! I’m happy to move forward positively, and collaboratively. I’m fairly new to the community, so I am also getting the hang of things.

1 Like

You raise some interesting questions @burrrata. As I said on the Spontanious Community Call (thx for the great writeup on that btw!), I’m OK with the new weights. They make sense to me, even if my share has gone down (for now).

I need to think more on some of these issues, but just thought I’d share my 50,000 ft view of some things:

  1. Cred scores =! lifetime value contributed, yet. While that’s the basic goal, I sense that they are being used as a lever to adjust incentives until Incentives and Boosting are live (@decentralion correct me if I’m wrong). This was why when @burrrata joined the project and quickly had more cred than me (even though I’d been contributing for months), I was a little surprised but OK with it. I didn’t think @burrrata had already generated more value than me. I thought they were being “boosted” to incentivize them. Which it appears to have done:) Similarly it feels like like @vsoch may be being boosted? Or not, I also only saw the widget and didn’t realize all the other work on containers, etc. This is just a a problem generally we’ll see again. It’s impossible for any one person, even the project lead, to have visibility into everything, especially as the project grows. I think that for now, we do need a TBD (Temporary Benevolent Dictator) that can make decisions quickly and steer the plane as it’s being assembled as it flies though a mountain pass in a storm. I’m also keen on seeing incentives and boosting implemented so we can start using those to steer. I will also point out that, since I’m the only one that has sold Grain (~$3.5k worth (plus $2k for an earlier contract with @protocol to write an article)), I have realized infinity more dollars than any other contributor. It’s important to realize that there is non-zero counterpary risk here, dependent on a number of factors outside of our control. I absolutely trust all parties are acting in good faith here, and believe Grain will be valuable (probably more than the price I’m selling at!), but my experience in VC-backed startups and crypto have taught me that counterparty risk is a deep, complex issue, and I factor that into my decisions.
  2. Decentralized governance is hard. We’re exploring a lot of new territory here. As I’ve said before, we’re grappling with some big, unsolved issues in decentralized governance. While I’m happy we’re focusing on this, and glad we’re not just going the easier route of relying on a typical startup model, I worry about scope. Are we going to reinvent reputation systems, solve the open source funding problem and solve decentralized governance? Probably not all three. However, I think it’s very valuable to wrestle with these fundamental issues as we dogfood, as they will inevitably arise again and again in other projects that adopt SourceCred. It will also inform the design of incentives and boosting (key to making this work at a project level and acrue value to Grain), and help us come up with an alternative “story”/model to present to investors, who might have other ideas anyway btw; it’s not 2017 anymore, investment in the space has dwindled somewhat and drifted towards traditional equity models, and if we want to shop a new token model, it had better be solid. I think we are very fortunate to have @protocol as a seed investor/partner here. They’ve given us a lot of trust and freedom to experiment with new models, and seem value aligned. But as we start raising capital from other investors, we should be ready for hard questions, negotiation and compromise.

So @mzargham built a python implementation early on in sourcecred/research, which he used when exploring designs. However I believe it’s fallen out of sync with the main repo, which has seen big changes.

Python is the only language I’m half fluent in actually, and have had lots of ideas for tinkering with the scores. Just need to find that time…

Yeah…Discourse may be overvalued except that we’re kind of relying on it for all non-code work. Until we have incentives, boosting, and other platforms for strategy, biz dev, documentation, etc. etc. etc., it’s difficult to value it. We also want to be compelling to developers, who are scarce…Don’t have strong opinions, just wanted to input some info into the process.

The core value prop of SourceCred is that it recognizes and rewards contributions. That involves creating a contribution graph that gives contributors reputation that can then be correlated to rewards and potentially governance. Traditionally problems like funding, governance, and reputation are separated and that’s part of the reason why they’re so hard to solve: they’re not actually completely separate things. They all connect to each other. SourceCred acknowledges the interdependence of things and connects them in a way that is incentive aligned for positive sum value creation.

While we’re still in the CredSperiment, however, it’s important that TBD (Temporary Benevolent Dictator) helps to direct and organize things.

If we choose to deploy a bonding curve to handle our token sales there will be no negotiations, only discussions and presentations to share the value prop of the protocol :slight_smile:

So cool! I had no idea this was a thing. Created an Initiative for this as part of the Retroactive Cred Activation.

1 Like

Thanks for pushing back @burrrata :slight_smile: You make a good point about the interdependence of funding, governance, and reputation. What excites me most is that SourceCred is exploring new territory in the multidimensional “solution space” here. We should not confine ourselves along traditional lines prematurely.

1 Like

I can easily make time to work on the Python module, I mostly just need background about how the algorithm works (I’d rather not try and reverse engineer from code).

Why would I need to be boosted? Perhaps this reflects that a tool isn’t doing proper justice to show what the contributions actually are. I did indeed come up with sourcecred-action, but I also contributed a few hundred lines to the main repository for the automated builds, along with the Dockerfile:

And participated in discussions of course around that, and setting up the registries. Why do I need to defend myself? I’ve never encountered anything like this in open source. The fact that I have to justify my contributions for a tool that does exactly that is very ironic. :confused:

I’d like to please ask everyone to stop referencing me as “the example that doesn’t seem fair / was boosted.” It’s just not appropriate to keep unfairly targeting me like that. Let’s please focus the discussion on general, constructive points for SourceCred. You can still say “This change in weights makes me unhappy because of X,Y,Z” without targeting individuals.

2 Likes

SourceCred works really well IMO within a single repo, but once you merge contributions across even repos, using the same programming language, the scores lose meaning. Activity is just different in nature. Therefore we need a higher level “incentives compiler” to value across projects. For now, that is done by @decentralion via manual mode (manually changing weights for contributions).

Have you seen @mzargham’s article on pagerank in SourceCred?

I don’t pretend to understand how the algorithm works on a deep level, but this was a good primer for me.

Apologies for the unfair comparison earlier. Boosting is just the general term we’ve been using for a long time for valuing contributions and/or contributors more in the graph though. It’s not intended to be a negative term (exactly the opposite! we boost things that are undervalued!). I don’t think we should change the term boosting. Just be more careful in language generally.

One use of boosting that we’ve discussed is for ‘recruitment’, because there’s currently no way of paying someone a real wage from the get go, without paying them a salary outside of the system (which could skew incentives). Building up enough cred to pay the bills currently could take longer than is feasible for many developers. Boosting would be a way for people to express, with skin in the game, that they want to bet on the future value of a contributor. They will actually get a share of that contributor’s cred too, so the booster is taking on risk and reward. This is only at the concept stage for now, but could be a good way to get contributors we couldn’t otherwise.

No worries, thanks for the clarification. And thanks for the link! I haven’t read that yet, and your timing is spot on because I’m taking a quick programming break.

I don’t totally get boosting, but I figure I’ll catch on. Just to be explicitly clear about my incentives, there is absolutely nothing in there monetary related - I contribute to projects that I think are important, and / or fun. I found this community through @beanow and SFOSC, and I took interest because SourceCred can provide tooling to make communities more fun and transparent.

1 Like

Glad to have you aboard!

I think SourceCred can (and will) mean many things to many people. A working reputation system is very powerful. Personally what excites me most is the money aspect. Being able to make a living working on open source. But it’s still an experiment that can go in many directions.

For more on boosting, the Initiative for it is a good overview.

One of the features that I think will make comparing activity across repos feasible.

If you want to learn more about Boosting I recommend the Boosting: a prediction market on ideas thread. FTR, however, it’s still in the design phase and the mechanisms to make it possible have not yet been created. The Cred Boosting Initiative should have the most recent status and dependencies to make Boosting a reality.

Also, for a high level overview of SourceCred the SourceCred in 5 minutes thread is a great place to start :slight_smile:

Thanks to you both! The take home for the algorithms post isn’t anything about PageRank (you can implement this in a few lines with networkx) but the human bits (much harder!)

To be clear, this isn’t a statement about the PageRank algorithm, it is a statement about algorithms, and in particular algorithms used to reduce complex human efforts into quantitive boxes so we can compare them. There is no absolute objective measure of value. Every metric is objective conditioned on subjective choices of what to value. We make these subjective value judgements everyday, and there is nothing wrong with that, but when it comes to encoded our subjective value judgements into code, and implicitly or explicitly imposing them on others, it is our obligation as algorithm designs to be mindful of those subjective choices, and the impacts they can and will have on others.

And I completely agree, part of implementing any algorithm involves human bias (whether intended or unintended) and a great deal of randomization. Even running “the same” software on different OS can lead to different results (e.g., see this paper) and I’d bet you the same would be true for SourceCred, whether run on different OS or even with different languages. I suspect that would get messy too if money is involved, and you get a different answer in a different context.

Indeed, those messy humans…when I wrote my article on SourceCred and DAOs, I was depressed how intractable many of these problems are. Not really any working solutions yet, though I think SourceCred approach is starting to work already:)

Have you seen the podcast interview with @mzargham? Recored a while ago but we haven’t put it out on social media yet. Goes into some of these issues.