Smart(er) cred minting: supply/demand modeling

The CredSperiment
1

So, based on a couple of recent discussions in yesterdays’ office hours, and forum posts on:

Putting all those thoughts together, I came up with the idea that, maybe our cred minting system would be smart(er) if it modeled the traditional supply and demand theory.

Ballpark figures

To find some ballpark figures for our scenarios. In a previous startup I worked at we applied for a loan through a government innovation program. Part of their requirements was a €3000/m ceiling for full-time employees before taxes. I feel this is a nice figure because it’s not terribly competitive for say a senior software dev, but this was a requirement regardless of whether you’re CTO or a delivery guy and the cap kind of implies room for taking equity in the startup, similar to the retrospectively adjusting attribute of SourceCred. As well as implies what’s considered reasonable to comfortably cover living expenses.

Now if we want to peg @decentralion’s current absolute cred to represent full-time work capped at this €3k/m ceiling, (assuming a freelance style, handling fees and taxes after payout), I wind up with:

1993 cred, between 2018-02-04 - 2019-09-22 (just short of 20 months).
€3000 * 20 = €60.000
€60.000 / 1993 cred = ~€30,11/cred

For simplicity rounding this to €30 = 1 cred ratio, and applied to our current cred minting system:

  • Forum topic: 8 cred = €240
  • Forum post: 2 cred = €60
  • Github issue: 1 cred = €30
  • Github PR: 1 cred = €30
  • PR review: 1 cred = €30
  • Github comment: 1/16 cred = €1,88

Of course this is cred minting, so the topic / post / issue / PR creates this amount of new value, which will flow to several people. So you don’t get this amount 1:1 as the author.

Still, clearly this is well off the mark for the kind of value I feel they actually create for the project. Even if you take cred “reevaluations” out of the equation and consider this as purely an up-front incentive.

Entertaining those values for a bit

Suppose we leave those cred minting incentives as-is, and we get a whopping 10 full-time contributors additional “development bandwidth” thanks to the experiment. And their time is split according to those relative incentives.

  • Forum topic: 61% incentive = 6.1 FTE
  • Forum post: 15% incentive = 1.5 FTE
  • Github issue: 8% incentive = 0.8 FTE
  • Github PR: 8% incentive = 0.8 FTE
  • PR review: 8% incentive = 0.8 FTE
  • Github comment: 0.4% incentive = negligible

Assuming all contributions are of similar quality to what they’re now (e.g. no bike-shedding or cred farming). Having 6.1 + 1.5 FTE devoted to activity on discourse and 3x 0.8 FTE on github, I picture a scenario where this will create:

  • Impressive amounts of theory crafting, ideas, discussion, perhaps research, articles, rants, scenarios, experience write-ups, new use-cases and so on.
  • Perhaps a natural inclination / self organization towards news and summary threads to distill all this information (as forum topics are a pretty good format for this and heavily incentivized).
  • The excess bandwidth on the forums might make this the place for Q&A, on-boarding new users and coordinating other tasks, like design work and promoting the project.
  • An overabundance of material on the forums does not translate efficiently to code improvements, due to lack of developer bandwidth to follow up on it.
  • Development lagging behind, also means breaking any iterative cycles of ideas > build > hands-on experience > refine ideas.

The longer this continues, the more skewed and wasteful this becomes. And this “wasteful” part I think you can rephrase as the demand for people helping out in the forums is just saturated. While demand for coding is not.

How supply & demand fits the ideas

So the idea of giving more cred to the first 3 PR reviews and decreasing from there. As well as giving less cred to PRs that are already merged / closed. Can be considered a “demand” of 3 reviews on open PRs. The same for diminishing returns on PR comments. There’s a certain amount of “demand” for feedback on PRs which can be saturated.

Setting milestones / priorities / areas of improvements / voting on roadmaps and attaching cred to it or boosting it with mana, is just another way of describing: we have a demand for X.

Other ideas like, surveys to gauge feature value or using download numbers / usage stats to inject new cred, can also be considered updating demand numbers for a feature, or demand for the software as a whole.

It also reflects what I would normally do when trying to suggest ways to help the project out. I would highlight the kind of work where supply doesn’t meet demand, the work that would really help out but few people are currently working on.

Likewise, I value contributions that address a real need for which there’s not much supply (for instance someone implementing a proof-of-concept themselves and making a write-up of their findings) a huge amount more than even the most thoughtful feature requests, not only because there’s a lot more effort involved, but also because it’s a lot harder to find people willing and able to do this.

Implementation thoughts

My feeling right now is that any implementations just needs to be an improvement over the current cred minting system of node weight == new cred. A highly accurate supply / demand tracker is way overkill I think.

However we need:

  • A way of tracking demand.
  • A way to classify what contributions are “supply” to a demand.
  • A way to track how much supply there is for a demand.
  • A formula to translate this supply:demand ratio to an amount of cred to mint on the next “supply” contribution.
  • Room for moderation systems to rectify inaccuracies and abuse.

For now I suggest, manually writing demand rules and try to find some sane default parameters. Similar to how weights are currently manual with defaults.

Choosing classes of “demand” in a way that’s simple to detect supply for. Such as being a particular node type, having a particular github label, being referenced in a github release or “summary thread” on discourse, or a common pattern (comments on open issues vs closed issues, reviews on open PRs vs closed/merged PRs).

Applying the supply/demand system sequentially and ordered by time. In order to have deterministic results that can be explained in a UI. Such as “this comment got 1 cred, because it matched the ‘comments on an open issue’ rule and had an unmet demand of 10 more comments at the time of posting” or “this comment got 0.1 cred, because it matched the ‘comments on an open issue’ rule and was at 20 comments over-saturation at the time of posting”.

The formula to translate supply:demand to cred, I think can take on lots of different shapes and needs further research. However I think it should be a gradual scale instead of implementing hard limits or falling off a cliff, in order to make sure a type of contribution that is incorrectly considered “saturated” is not so aggressively de-incentivized a well intentioned contributor can no longer earn cred if they feel it’s a worthwhile contribution to make. Nor make it super risky for contributors to go ahead with their contributions when they feel the earlier supply should be disqualified because it looks like it’s created in bad-faith.

3 Likes
2

This is an interesting perspective; thanks for sharing it. I think the conceptual framework of looking at contributions in terms of “supply” and “demand” is really promising. However, I think minting cred based on fixed types, even with a supply/demand model, still has the same structural weaknesses of the current model.

Minting cred based on node type is the real flaw

The way I chose the weights was aiming towards getting an overall distribution that matched my intuitions. In particular, I wanted to give meaningful cred to the participants on this Discourse; to do so, I set very high weights on Discourse activity. However, as you point out very clearly, these weights don’t make economic sense. If we keep these weights, in the medium term the community will likely become unbalanced as you describe.

However, the issue with weights goes deeper than their current paramter choices being wrong; minting a fixed amount of cred per unit of activity is just a bad model. This is because just knowing that something is a PR or is a post tells you approximately nothing about its actual value. Fortunately SourceCred isn’t all this dumb, the cred itself gets awarded based on the graph structure, but minting the cred based on the node type is really naive.

Supply/Demand by node type has the same issues

Overall, I think that thinking about supply and demand at the level of node types suffers some of the same issues as setting fixed weights based on node type. If we have a “demand curve” for pull requests, won’t that incentivize a spate of low-effort minor PRs every week? (Possibly with people “saving up” small PRs to send in on Monday morning.)

Also, since PRs are not interchangeable, a static supply/demand system will always grossly misprice some PRs. E.g. I get 4 PRs fixing minor UI issues, so my “demand curve” says I’m saturated, but then I get a 5th PR adding an important new feature. I’d be a lot more excited about that 5th one than about the 4 earlier ones.

I think there are a few cases where the “demand curve” model may make sense, like code reviews. Maybe I want every PR to get at least code reviews, and so we’re willing to incentivize more to get the first few reviews.

Alternative: Mint cred based on human review

Going back to @kronosapiens’s thoughts on PvE vs PvP dynamics. I think your proposal is basically a more sophisticated “PvE” system, but what if we just switch cred minting to “PvP”? Here’s how it could work:

Weekly Updates

For every week, we write a “Weekly Update” that summarizes what happened that week (interesting forum posts, links to pull requests that merged, new features added, etc). Through some offline process, we choose an amount of cred to mint, and flow it through the weekly update to the contributions mentioned therein. We can use the existing heuristics (and any other heuristics we come up with) as an input to our deliberation about how much cred to issue.

Obviously, this process requires a lot of trust, involves conflicts of interest, and won’t scale; over time we will find better ways to systematize it. (E.g. moving to a prediction-market based curation mechanism.) But in the spirit of Do things that don’t scale: this would give us much better short-term results than any system of heuristics based on node types.

Roadmap bounties

As discussed during office hours: once we do our roadmap prioritization we can set large cred bounties on accomplishing different goals for SourceCred. Once the goal is accomplished, we will “unlock” a large amount of cred that then flows to the nodes relevant to accomplishing that goal.

These approaches will implicitly involve supply and demand, since we (when setting the weekly weight) will have an implicit “demand” function that underlies our judgements about how to score each week.

2 Likes
3

Unaddressed problems

Great points on this. And I agree some of those same problems come back. A supply/demand model in this way would not solve the problem of valuing the contents a node represents. Distinguishing between minor fixes, crucial features or attempts at gaming, is probably not even really a problem with the granularity of the unit you’re trying to evaluate. It’s that doing this at the node level requires a PvE style algorithm to scale, and a PvE implementation to evaluate this is really difficult.

The implementation I described didn’t make any attempts at solving that problem any better than the current system. An issue’s an issue. A post’s a post. To illustrate:

Yes, though I didn’t have a interval based total demand for PRs in mind just yet, that’s conceivable and interesting! However the “low-effort minor” PR qualification is something neither the current nor proposed system tries to make (at least at the minting stage). So the same could be said of having a 5x weight for [node type] being a perpetual incentive to flood with low-effort minor [node type] under the current system.

However

The main difference is that it tries to use the supply/demand model to redirect incentives to the classes of demand that are undeserved instead of relying on observation and weight changes to do so.

I did set out to just improve over the current system. And it does have one leg up over that. If I look back to the previous scenario of the forum weights possibly incentivizing it too much for the mid-term, how would this model provide both in the need to value forum contributions more highly until now to match your intuitive ranking of people and prevent eclipsing other contribution types in the future?

I’m imagining this in a few stages. This is really a rough scenario so consider it hypothetical and inaccurate.

Very early days

decentralion and wchargin are chugging away, doing all the types of work. However being primarily the two it’s perhaps in need most of all of extra input. Ideas are the most undeserved, and thus incentivized. Supply/demand might look like this.

image

Later on

More contributors came in, and they’ve helped out in various areas. Particularly ideas are not super costly to contribute and have a networking effect. The new people together were able to storm up a whole array of fresh input. Claiming the high incentives here from the previous stage, the 8x equivalent? Some of the work getting done now in the ideas and research stages have increased the demand for coding to follow up on them. Overall more help in any area is still very welcome :smiley:

image

Near future

Because of some period of very productive brainstorming, new versions being released, experience and insights gained a new flood of ideas follows from those insights. All this progress brought in curious new users and contributors too. Given that there’s now a whole wave of ideas formulated, there’s a big surge in new follow-up work from that to be done. Normally, all areas would have increased in demand. However a bottleneck is forming at the research and coding areas to ingest ideas. So in relative terms, demand for ideas didn’t grow as strongly and got saturated. Creating greater incentives to focus on research and coding in particular.

image

Granularity

Obviously this scenario uses really coarse focus areas and they don’t translate well to node types. So in a way this is quite similar to what you would want a roadmap to do.

Improvements

Given your reply I think we can do better than just a level of indirection for weights. How about using the terms supply and demand as new concepts, in order to move away from minting at the node level?

Scrapping the X comments until saturated rules for demand. But looking at ones that can include some of the PvP we’re missing. To cast your examples into these terms:

A weekly update system to mint cred would be for example: one weekly update post is a unit of demand, it’s met by multiple units of supply (each item in this post), curation and putting a cred number to each demand and supply unit is done by people.

A roadmap bounty system would be: each priority is a demand. The relative importance of each priority is represented by the size of the demand. The details of how supply units are decided upon and how cred is released from the demand, depends on the type of priority. A concrete deliverable would be an all-at-once unlocking of cred once it’s delivered probably allowing you to defer identifying and valuing the supply units around time of delivering. If the priority is a fuzzy goal like “improve the docs”, “easier installation”, it can gradually release cred, and classification of the supply can also happen gradually for example with github labels or milestones.

I’m not entirely sold on the demand / supply terms for this as we’re starting to dilute this analogy with a lot of “custom determining of value”. So it might as well be sources and sinks. Or source nodes with weighted edges to it’s sinks. Or just call it the manual plugin :sweat_smile:

4

I’m coming around to the value of this framing. One way of thinking about my node weight choices for Discourse: I believe having an active community discussing SC is very important for community health. So I have a lot of demand for Discourse posts, and I’m willing to “pay” a lot of cred to fund or jumpstart that system. However, as it grows, the marginal amount I’m willing to pay for each post (not knowing anything else about it, other than that it is a post) drops off steeply.

So it does make sense to think of this as a demand curve. The total cred minted for Discourse posts might asymptotically peak at say, 200 cred per week. After that, it’s still possible for more cred to be minted based on Discourse activity, but that minting will happen through more judicious systems (e.g. we define “demand” for weekly update posts, and then implement that demand via the manual plugin).

5

What if, in addition to things like weekly update posts, there was a link between discussions/ideation and open Issues that actually get built?

Not all conversations and ideas are created equal. If someone (or a group) has a great idea that would add tons of value to the system, then people should be able to boost that idea. Then there should be a way to link that idea (and it’s boost) from Discourse to an open Issue on GitHub. Then if the idea gets built and used a lot (implying there was demand), some of that should flow back to the person or group that created the idea in the first place (and everyone that boosted it along the way). That way there’s still an incentive to contribute to discussions, but there’s a connection between those discussions and tangible value that is derived/realized from them.

This ties in with the Cred Weighted Roadmap. The overall goal being to incentivize people to create and support ideas that are wanted and actually add value to the system :slight_smile:

1 Like
6

I emphatically agree, @burrrata. The only diff is instead of using GitHub issues, we should use initiatives. This is why every initiative has a “references” section.

I want the Cred that people earn from :heart:s to be the “small change” of the SourceCred world. The bigtime cred should come from having ideas that are relevant to things that really get built.

1 Like