SourceCred for documentation

Edit: while I was writing this Max, the WasabiWallet doco maintainer was writing a post giving more details,

WasabiWallet is experimenting with paying contributors 0.1 BTC a month based on contributions to their docs. This was inspired by the Contribution Game they ran last month, which gave away 1 BTC to contributors to the WasabiWallet repo based on number of lines added/deleted, which resulted in a lot of needed love for the docs. They’re looking into SourceCred to score the contributions on the docs moving forward.

image

Since I work as a technical writer in cryptocurrency DAO (Decred) and know SourceCred a bit, I thought I’d share my initial thoughts on how this might work, and see if anyone else has ideas.

First impressions:

  • I think it should work fairly well out of the box, with default settings. When I run SourceCred on the docs repo I’ve been paid to work in the last year, dcrdocs, the scores seem to reflect reality as I understand it pretty well. Below is a chart of the timeline cred (hosted here).

  • The activity is “bursty”. You can see a big spike when the docs were first fleshed out. Then a lull…then some more sustained activity, basically when I join (the green line) and start fleshing out holes in the existing documentation, documenting new features as they came out (many wouldn’t have gotten documented at all if I hadn’t joined), and doing general maintenance and polishing. There is a concern this burstiness may skew rewards, as slow months will cause higher payouts for contributions, regardless of quality. In this post I trip out on the idea of applying the difficulty adjustment in Bitcoin’s PoW algorithm to cred, but that is a bit high concept. A more practical solution could be paying a certain amount per unit of cred, as the amount of cred will go up and down with activity. The $/cred could be based on, for instance, the average cred produced in the last few months.
  • While at a high level, the cred scores make sense, I do wonder if lines added/deleted might be better here? While not a robust metric for code, lines added/deleted might be a more robust proxy for documentation? I know SourceCred’s plugin architecture allows you to plug in any heuristic (lines added/deleted, various pattern matching, etc.). Would it be possible to have lines added/deleted be an input to calculating cred scores?
  • Manual mode (the ability to adjust weights of contributions manually) may be useful here. Especially since SourceCred has mainly been trained on code, and documentaiton is fairly new territory. It is possible contributors’ are incentivized to game their scores, or just go for a lot of low-hanging fruit, which while valuable, should not be scored as high as high-hanging fruit. Key here I think would to just be transparent about the adjustment, as @nopara73 was with the Contribution Game, posting adjustments to GitHub,

Love that this is being considered for docs btw. Would love to get paid this way!

2 Likes

I too am concerned about this

@burrrata, I think the general sense is that, for most situations, there will be a “human in the loop” somewhere. When I bill for documentation work in Decred, it is actually evaluated using a process you could call reputation-based peer review. Other people in my domain (in this case other people that contribute to the docs) review my invoice, and if they think it’s too high, they can contest it (anonymously). I think this is smart, as the person that has the most information about how to value something is typically the person doing the work.

1 Like

Oh that’s really interesting. I like the idea of reviewers being able to anonymously contest things. Currently the way Aragon DAOs are set up it’s all public, which leads to a lot of people dragging their feet to avoid hurt feelings or confrontation. Having an anonymous way to provide feedback and/or vote would be great. What’s the best way to learn about that aspect of Decred specifically?

The exact process isn’t formally documented anywhere (that I’m aware of). Partially because it’s been fairly informal up until recently (a point of centralization deemed to be necessary). To further Decentralize and Automate moving forward though, they are currently formalizing and automating this process in the Contractor Management System (the system I submit my invoices to). So far, this system has been more about automating the basic invoice review and processing and payments functionality. But the current building is focused on a voting scheme whereby contractors can vote other contractors in and out of the DAO (should they abuse their invoicing, for example). Details can be found here in a proposal that funded the construction of the system, https://proposals.decred.org/proposals/fa38a3593d9a3f6cb2478a24c25114f5097c572f6dadf24c78bb521ed10992a4

The part of the process related to peer review is being discussed currently, as well as the problems related to contractors billing across different domains (who are you peers?).

As for the contractor model generally, here are some basic resources,

1 Like

THIS IS AWESOME. I really really like how when you bring someone into the org, your reputation is tied to theirs, thus creating skin in the game for all parties involved. This system is currently at the proposal stage and has not been built yet right?

Proposal passed a while back, funding it. So far work has concentrated on automating the current system for reviewing invoices, process, payments, etc., but work is beginning soon on the voting part. This is though basically just automating the informal system that’s already in place. E.g. you have to find two people in your domain to nominate you to become a contractor. What new besides the automation is the appeal process if your contractor status is revoked. I suppose a person you vouch for’s reputation is tied to yours in the same sense it is at a company.

One interesting thing about SourceScred is that if you bring someone on, their reputation is literally flowing to you and vice versa. A “bonus” for bringing on someone that works out has been discussed. But actually when I imagine how it might work, I think this would happen organically, as the person you brought on would be more inclined to work with you and “pay you back”.