Reference Cases for Credit Attribution

This thread is intended for defining Reference Cases for Credit Attribution.

The Original reference cases discussion was for a line graph or chain of dependencies from an early discussion with @decentralion and Evan from Protocol labs. This reference case and others are briefly explored here using some preliminary research infrastucture. See GitHub issues in SourceCred Research for how we are testing and expanding our infrastructure.

There is not necessarily one right answer for how credit should be attributed in a reference case, rather reference cases are simple cases one can reason about how we might want credit to accumulate. I would like this Topic to be used for proposing reference cases, sharing results of experiments on proposed reference cases, and discussing how we think the cred should flow, including discussing how it might need to flow differently depending on context.

Here are some more ideas.

  1. A reference case containing a normal pattern of development for a small project (mix of ppl creating issues, submitting pulls, etc) and one person who spam-added 100 issues with identical text.

  2. Same as above, except one person added a spam comment to the end of every legit issue and PR

  3. A tiny codebase consisting of a few commits from two authors. One author just writes implementation, one author just writes docstrings. How does the cred differ? Do we have a way to filter for “documentation cred” vs “implementation cred”?

  4. A codebase consisting of two coders. One coder adds code, the other comes afterwards and changes the indentation in each file.

  5. A codebase with two people. One works on file A, one works on file B. Do we have a way to query “who has cred n file A” and get the right answer?

  6. A codebase with three people. One works on file A, one works on file B, the third then renames file A to file C. If we ask “who has cred in file C” do we get a coherent answer?

outsider perspective:

it appears that mzargham has a strong grasp of the very deep theory involved here. however, i wasn’t able to tie the theoretical issues discussed to real-world, higher-level issues and attack vectors brought up by @decentralion. do either of you see any obvious translations between these two levels of abstraction? the attacks @decentralion brings up seem like they could be addressed by simpler heuristics/filters. e.g. if someone started adding a spam comment to every issue, that could be easily identified and filtered out by fairly simple logic. no need to mess with delicate parameters in the page rank algo. less obvious, more sophisticated malipulation (e.g. a smart human selectively commenting on issues they see are ranked higher), is more difficult. here you’d likely need to bring a “human in the loop”. someone trusted that could “spotlight” abusive behavior. but here you run into difficult issues of trust and corruption (e.g. can you trust a maintainer of a repo to downvote a spammer they may be colluding with?).

there is some low-hanging fruit here though. for instance, it shouldn’t be difficult to tell documentation from code. simple heuristics could get you 80% there. a basic classifier machine learnign algo (i’m sure there’s research in this area) could get it to 99% (number out of my ass, but i stand by them). fortunately for sourcecred, identification of spam was one of the first internet problems. decades of research and billions invested already. i’m sure there’s open-source solutions that could do the job good enough.

You’ve got two maintainers A and B, with all of the project’s cred (500 cred apiece). A new contributor C, starting with 0 cred, submits their first pull request. B works with C extensively to merge request. A doesn’t do anything.

I’m semi-suspecting that B might actually end up w/ less cred than A ?

1 Like

If maintainer B does all this (perhaps undervalued) work, and A doesn’t do any other work, presumably B comes out ahead. But if in the meantime A does other work that is valued more (e.g. reviews some important code from a long-time contributor), then presumably B comes out ahead. In my project, there’s an informal (mostly unspoken, though sometimes spoken) understanding that maintainers won’t invest time in a new contributor unless they feel it’s a good investment (i.e. that the contributor will “ROI”). I’m actually essentially contributor C in your scenario, but have worked my way up a little bit. I found a maintainer B that was kind (opportunistic?) enough to start reviewing my small contributions, and I’ve slowly worked my way up to larger contributions. I currently have my first significant code contribution sitting in a PR waiting for maintainer B to review. It’s been about a week, and I’m mildly anxious. But also see that they’re super busy with a big launch, and know my priority.

If I were to try to quantify this, I might have a heuristic like: if maintainer B works extensively (many comments on a PR, etc.) with contributor C, and contributor C disappears (makes no more (or few)) commits. Or keeps committing but keeps requiring extensive reviews. Then maintainer C’s cred would not increase (perhaps at the expense of maintainer A). If maintainer B works with contributor C, and contributor C eventually becomes a major contributor, maintainer B should accrue some cred for that retroactively. This actually may not need to be encoded explicitely, as contributor C may exibit loyalty to maintainer B, flowing cred back to C over time.

My thinking was that nodes pass their cred to other nodes, and if you have a high-cred node interacting with a low-cred node, most of the transaction would be the high-cred node flowing cred to the low-cred node, since the low-cred doesn’t have much cred to flow back.

I think that might be 100% wrong though, from messing with networkx.

That makes total sense to me. It reflects the current, baseline reality. It also raises baseline questions such as, what is the goal of a score? Is there only one goal? Are these goals determined by sourcecred, or the project?

Another way to ask this might be, is sourcecred looking to meansure cred as-is, or trying to actively create cred/value?

I paused the attempt to manually produce code level reference cases because our manual mode plugin makes its a lot more user friendly to run such experiments.

My ideal case would be to make it possible for anyone to run test cases that interest them with minimal overhead (coding), but I think that is still a pretty far off state. In the mean time, I am focusing on experimental infrastructure with the help of @decentralion and as that makes progress I will continue to use this thread a source of ideas for reference cases.

Most likely I’ll curate it into github issues that include both the reference cases and methodology of the analysis. I’ll checkin here and cross link those issues when we get to that point.

Researcher’s reference case in narrative form:

I spent over a year of research on methods and tools for designing crypto-systems using bio-memetic and physical systems analogies to yield prove-able properties.

I then spent several months designing bonding curve variant as part of a continuous funding system based on discussions with friends.

Collaboration really heated up as we iterated and refined the ideas, more and more people got involved.

I helped one of those friends write 2 articles on the topic, and provided advanced support while a team developers implemented a first prototype at a hackathon.

The idea has expanded and now people are working on it, even presenting on it that I don’t know personally.

I am extremely thrilled to see the design work expanding out into the world, getting refined, used; however, there’s still a little bit of me that thinks…

It would be really nice if i just got tagged in this tweet.

I really don’t mind being way down the base of the attribution network as a researcher, ideator, system designer, but I really prefer a world where you can always see the trace back to that contribution. #sourcecred

1 Like

Via @mzargham, a real-life reference case on comment chains.

I’ll admit that it amuses me to create a reference directly to the link where @mzargham suggested the comment chain wasn’t adding value.

1 Like