On Code Review Cred

You raise some good points; both that using the system I described creates more cognitive load / need for learned practices by PR authors, and creates potential social tension where the reviewers feel that they deserve a :+1: and didn’t get it. Starting with a “default trust” assumption and relying on moderation / social pressure to curb bad behavior makes sense.

It makes me think of @mzargham’s model of three phases of trust in SourceCred. While we are in the “high trust / low adversarial” mode, we can use heuristics that might be brittle at scale, like giving cred to the first pull requests. Then maybe we can skip straight past the “explicit feedback from authors” and towards having impartial curators make necessary judgement calls and adjustments based on the value of the PR.

This will make the behavior of the system more consistent (curators can have a more “standardized” policy and playbook, vs idiosyncracies of the individual pull authors), and moves the cognitive load off of every contributor and onto a specialized bureaucracy.

For right now, I don’t think we even need the “adjust cred based on # of review” heuristic; it’s extremely rare to see a pull with >2 reviews. So to start, we can just give PR reviews a positive weight (the existing behavior). And we can revisit if / when we feel like it’s no longer working.

1 Like