# Make the GitHub Plugin Robust

**URL:** <https://discourse.sourcecred.io/t/make-the-github-plugin-robust/256>\
**Category:** Initiatives\
**Tags:** 🌱up-for-adoption\
**Created:** [October 4, 2019, 4:30pm UTC](https://discourse.sourcecred.io/t/make-the-github-plugin-robust/256 "2019-10-04T16:30:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![decentralion](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/decentralion/32/7_2.png) [@decentralion](https://discourse.sourcecred.io/u/decentralion)\
**Post date:** [October 4, 2019, 4:30pm UTC](https://discourse.sourcecred.io/t/make-the-github-plugin-robust/256/1 "2019-10-04T16:30:30Z")

</div>

# Make the GitHub Plugin Robust

## Status: proposal

## Champion:

## Initiative Description:

Currently, the GitHub plugin is quite fragile. When loading a large repository, it is quite likely to fail in a variety of ways, for example:

- [invariant violation on missing Reaction author](https://github.com/sourcecred/sourcecred/issues/1398)
- [GitHub hourly rate limit exceeded](https://github.com/sourcecred/sourcecred/issues/1352)
- [Maximum call stack size exceeded](https://github.com/sourcecred/sourcecred/issues/1354)
- [Failures due to entity typename changes](https://github.com/sourcecred/sourcecred/issues/998)
- [Failures when a node has been deleted](https://github.com/sourcecred/sourcecred/issues/938)
- [Other contract violations in the API](https://github.com/sourcecred/sourcecred/issues/996)

These issues can sometimes be worked around by updating the list of blacklisted object IDs, but this is itself a high-friction and poorly documented process, and requires the user to [invalidate their cache](https://github.com/sourcecred/sourcecred/issues/996) and re-download the whole project.

The net consequence of all these issues is that running SourceCred on arbitrary GitHub projects is a substandard experience which creates a lot of frustration for users.

However, thanks to the [object blacklisting system](https://github.com/sourcecred/sourcecred/pull/972), none of these issues block SourceCred’s dogfooding, so it’s been easy for us to ignore these problems.

## Benefits:

- We’ll stop frustrating early users
- It will become possible to reliably run SourceCred on large repositories

## Implementation plan:

TBD

## Estimated Work (hours):

40-80? I think it’s a lot of work.

## Dependencies:

- [GraphQL Mirror fidelity awareness](https://discourse.sourcecred.io/t/graphql-mirror-fidelity-awareness/275)

## References:

- [@wpank](https://github.com/wpank) started a [series of pull requests](https://github.com/sourcecred/sourcecred/pulls?q=is%3Apr+author%3Awpank+is%3Aclosed) addressing some of these issues, but they didn’t merge.
- @wchargin wrote a detailed [implementation plan](https://github.com/sourcecred/sourcecred/issues/998) for fixing the inconsistent typename issues

## Contributions:

TODO (add contributions that already apply, and new ones)

---

<div class="post-metadata">

**Author:** ![decentralion](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/decentralion/32/7_2.png) [@decentralion](https://discourse.sourcecred.io/u/decentralion)\
**Post date:** [October 4, 2019, 4:42pm UTC](https://discourse.sourcecred.io/t/make-the-github-plugin-robust/256/2 "2019-10-04T16:42:12Z")

</div>

This issue has been an active pain point for at least a year, and has been near the top of my mind that whole time. However, I have not seriously prioritized it, and am still not prioritizing it. Basically, I want to focus on making the CredSperiment a big success (see: [deep before wide](https://discourse.sourcecred.io/t/deep-then-wide/102)).

You can think of prioritizing the CredSperiment over external users as being like putting on our oxygen mask before others’. Basically, for any feature within SourceCred, having a high-performing and well-rewarded core community around SourceCred will make that feature easier to ship, because we’ll have more resources. Hence, I’m focused on features that improve SourceCred’s internal ability to coordinate and execute.

That said, I do think this is a very important issue, and if anyone else feels called to start working on it, I encourage them. As soon as I get the ability to put my cred where my mouth is via [initiative bounties](https://discourse.sourcecred.io/t/enable-initiative-bounties/257), I’ll give this one a hefty cred bounty.

---

<div class="post-metadata">

**Author:** ![s\_ben](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/s_ben/32/816_2.png) [@s\_ben](https://discourse.sourcecred.io/u/s_ben)\
**Post date:** [October 5, 2019, 6:32am UTC](https://discourse.sourcecred.io/t/make-the-github-plugin-robust/256/3 "2019-10-05T06:32:30Z")

</div>

Yes! While I realize why this is lower priority than other things, I do badly want to run this on Bitcoin, ethereum-go, and some other big repos that SC is currently choking on. Creating those scores, cool looking data viz, could be very interesting to the respective communities, and probably make the rounds on social media/crypto outlets, start some interesting discussions.

---

<div class="post-metadata">

**Author:** ![wchargin](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/wchargin/32/1397_2.png) [@wchargin](https://discourse.sourcecred.io/u/wchargin)\
**Post date:** [March 2, 2020, 12:50am UTC](https://discourse.sourcecred.io/t/make-the-github-plugin-robust/256/4 "2020-03-02T00:50:25Z")

</div>

Quick status update:

- The completion of the fidelity awareness initiative (already listed as a dep) resolves half of the listed failure modes: “invariant violation on missing Reaction author”, “Failures due to entity typename changes”, “Other contract violations in the API”. The set of blacklisted IDs is no longer used at all (but the blacklisting infrastructure still exists in case we need a quick stopgap later).
- The “Maximum call stack size exceeded” bug is fixed by [#1684](https://github.com/sourcecred/sourcecred/pull/1684), which I just merged.

So, the two remaining items are GitHub rate limiting and node deletions. I hope that [#1687 (“Don’t fetch GitHub `Commit` parents”)](https://github.com/sourcecred/sourcecred/issues/1687) will have a significant impact on both rate limiting and load times, as (from qualitative experience) this seems to come up most often on repositories with long chains of commits none of which was merged by a pull request. Proper node deletion handling is still an issue (as linked in the OP).