# Refactoring the load system

**URL:** <https://discourse.sourcecred.io/t/refactoring-the-load-system/629>\
**Category:** Initiatives\
**Tags:** ✔️completed\
**Created:** [February 5, 2020, 9:59am UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629 "2020-02-05T09:59:46Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [February 5, 2020, 9:59am UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/1 "2020-02-05T09:59:46Z")

</div>

# Description

* * *

The existing load system was quite complex, which made it hard to change. And it needed to change, in order to support [Unified reference detection](https://discourse.sourcecred.io/t/unified-reference-detection/301)

This Initiative is about biting the bullet to break the existing load system into smaller parts that aren’t as difficult to change.

  

# Status: Completed

The refactor itself has been implemented! 🎉[Switch to LoadContext based loading](https://github.com/sourcecred/sourcecred/pull/1622) 🎉

# Champion

![Avatar Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/45/98_2.png "Beanow") @Beanow

  

# Benefits

* * *

- Divides and conquers the complexity of the load system.
- More robust unit tests.
- Easier to implement changes in the future.  
(better tests = more confidence, less complexity = less headaches for devs)
- Many implicit features of the load system become explicit.  
For example, we now have a definition of the [“SourceCred data directory”](https://github.com/sourcecred/sourcecred/blob/master/src/backend/dataDirectory.js#L17-L20) we didn’t have before.

  

# Implementation Plan

* * *

![Apply head to keyboard](https://canada1.discourse-cdn.com/flex032/uploads/sourcecred1/original/1X/139600e70797dd987d31f88399cd5125bc8d75e9.gif)

  

# Deliverables

* * *

A [design document](https://github.com/sourcecred/sourcecred/issues/1586) which outlines the new concepts we’ll use to break the load system into smaller parts. And plenty of PRs to implement the refactor.

  

# Dependencies

* * *

None at the moment.

  

# References

* * *

**Discussion / trackers**

- [Reduce load complexity #1559](https://github.com/sourcecred/sourcecred/issues/1559)
- [Static or dynamic loading refactor #1506](https://github.com/sourcecred/sourcecred/issues/1506)
- [Remove deprecated load functionality #1629](https://github.com/sourcecred/sourcecred/issues/1629)

**Related refactors, used for reference**

- [add api/loadWeightedGraph](https://github.com/sourcecred/sourcecred/pull/1604)
- [add `loadWeightedGraph` methods for plugins](https://github.com/sourcecred/sourcecred/pull/1602)

  

# Contributions

* * *

**Design doc**

- [Load refactor design doc #1586](https://github.com/sourcecred/sourcecred/issues/1586)

**LoadPlan version (concept rejected)**

- [Backend: introduce LoadPlan](https://github.com/sourcecred/sourcecred/pull/1565)
- [Discourse: implement DiscourseLoader interface](https://github.com/sourcecred/sourcecred/pull/1566)
- [Identity: implement IdentityLoader interface](https://github.com/sourcecred/sourcecred/pull/1567)
- [Github: implement GithubLoader interface](https://github.com/sourcecred/sourcecred/pull/1570)
- [Backend: implement pluginMirrorPlan](https://github.com/sourcecred/sourcecred/pull/1571)
- [Backend: implement pluginGraphPlan](https://github.com/sourcecred/sourcecred/pull/1572)
- [Backend: implement timelineCredPlan](https://github.com/sourcecred/sourcecred/pull/1573)
- [Backend: implement createPlan](https://github.com/sourcecred/sourcecred/pull/1574)
- [API: implement loadDefaultPlan](https://github.com/sourcecred/sourcecred/pull/1575)
- [Use loadDefaultPlan instead of load](https://github.com/sourcecred/sourcecred/pull/1577)

**LoadContext version (concept accepted)**

- [Backend: implement PluginLoaders.declarations](https://github.com/sourcecred/sourcecred/pull/1616)
- [Backend: implement PluginLoaders.updateMirror](https://github.com/sourcecred/sourcecred/pull/1617)
- [Backend: implement PluginLoaders.createPluginGraphs](https://github.com/sourcecred/sourcecred/pull/1618)
- [Backend: implement PluginLoaders.contractPluginGraphs](https://github.com/sourcecred/sourcecred/pull/1619)
- [Backend: add ComputeFunction](https://github.com/sourcecred/sourcecred/pull/1620)
- [Backend: implement LoadContext](https://github.com/sourcecred/sourcecred/pull/1621)
- [Switch to LoadContext based loading](https://github.com/sourcecred/sourcecred/pull/1622) 🎉 🚀

**Prerequisite refactoring**

- [Use stable stringify for cred.json & graph.json in load](https://github.com/sourcecred/sourcecred/pull/1576)
- [Backend: create DataDirectory class](https://github.com/sourcecred/sourcecred/pull/1564)
- [Backend: implement MemoryCacheProvider](https://github.com/sourcecred/sourcecred/pull/1613)
- [Github: add CacheProvider support for fetchGithubRepo](https://github.com/sourcecred/sourcecred/pull/1568)
- [Github: switch to CacheProvider for fetchGithubRepo](https://github.com/sourcecred/sourcecred/pull/1614)
- [Github: add fetchGithubRepoFromCache](https://github.com/sourcecred/sourcecred/pull/1569)
- [Github: derive mirror database name from repoId](https://github.com/sourcecred/sourcecred/pull/1581)
- [Add WeightedGraph.overrideWeights](https://github.com/sourcecred/sourcecred/pull/1606)
- [Add `identity/contractIdentities`](https://github.com/sourcecred/sourcecred/pull/1601)
- [Generate Weights from PluginDeclarations](https://github.com/sourcecred/sourcecred/pull/1595)

**Cleanup 🧹**

- [Remove deprecated load function](https://github.com/sourcecred/sourcecred/pull/1630)
- [Remove github/loadGithubData](https://github.com/sourcecred/sourcecred/pull/1596)
- [Improve test readability of loadContext and pluginLoaders](https://github.com/sourcecred/sourcecred/pull/1650)

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [February 5, 2020, 10:26am UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/2 "2020-02-05T10:26:36Z")

</div>

@decentralion can you give this a look to see if it makes sense? I’ve included some of your related work. Though not all, as `WeightedGraph` could be it’s own Initiative / Artifact.

Some I’m directly using, so they’re contributions:

- [Add WeightedGraph.overrideWeights](https://github.com/sourcecred/sourcecred/pull/1606)
- [Add `identity/contractIdentities`](https://github.com/sourcecred/sourcecred/pull/1601)
- [Generate Weights from PluginDeclarations](https://github.com/sourcecred/sourcecred/pull/1595)

Some are replaced but provided input. I’ve opted for references here, because they were intended to support WeightedGraph, not directly to address the complexity of load:

- [add api/loadWeightedGraph](https://github.com/sourcecred/sourcecred/pull/1604)
- [add `loadWeightedGraph` methods for plugins](https://github.com/sourcecred/sourcecred/pull/1602)

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [February 5, 2020, 2:12pm UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/3 "2020-02-05T14:12:57Z")

</div>

Fun fact, if you’re wondering why so many contributions. Some of the first work started way back in mid-December. It’s been a loooong time coming 😅

![image](https://canada1.discourse-cdn.com/flex032/uploads/sourcecred1/original/1X/fc11d4d04a60bce7e1c3fa7e8c82998f0782861c.png)

---

<div class="post-metadata">

**Author:** ![decentralion](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/decentralion/32/7_2.png) [@decentralion](https://discourse.sourcecred.io/u/decentralion)\
**Post date:** [February 5, 2020, 5:39pm UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/4 "2020-02-05T17:39:03Z")

</div>

Looks good to me.

---

<div class="post-metadata">

**Author:** ![s\_ben](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/s_ben/32/816_2.png) [@s\_ben](https://discourse.sourcecred.io/u/s_ben)\
**Post date:** [February 5, 2020, 5:56pm UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/5 "2020-02-05T17:56:19Z")

</div>

Wow. So much work!

Selfish question: will this allow me to run SourceCred on /Bitcoin and other old, messy GitHub repos with lots of dirty data that SourceCred currently chokes on?

---

<div class="post-metadata">

**Author:** ![decentralion](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/decentralion/32/7_2.png) [@decentralion](https://discourse.sourcecred.io/u/decentralion)\
**Post date:** [February 5, 2020, 7:04pm UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/6 "2020-02-05T19:04:32Z")

</div>

> [@s\_ben](#):
>
> Selfish question: will this allow me to run SourceCred on /Bitcoin and other old, messy GitHub repos with lots of dirty data that SourceCred currently chokes on?

No, although @wchargin is working on a change that likely will. See: [GraphQL Mirror fidelity awareness](https://discourse.sourcecred.io/t/graphql-mirror-fidelity-awareness/275)

---

<div class="post-metadata">

**Author:** ![s\_ben](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/s_ben/32/816_2.png) [@s\_ben](https://discourse.sourcecred.io/u/s_ben)\
**Post date:** [February 5, 2020, 7:18pm UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/7 "2020-02-05T19:18:41Z")

</div>

Oooooh. ❤'d Initiative for what it’s worth.

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [February 28, 2020, 12:52pm UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/8 "2020-02-28T12:52:12Z")

</div>

Added: [Improve test readability of loadContext and pluginLoaders](https://github.com/sourcecred/sourcecred/pull/1650)

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [February 28, 2020, 3:24pm UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/9 "2020-02-28T15:24:25Z")

</div>

Believe this is now completed! 🚀

---

<div class="post-metadata">

**Author:** ![decentralion](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/decentralion/32/7_2.png) [@decentralion](https://discourse.sourcecred.io/u/decentralion)\
**Post date:** [February 29, 2020, 2:56am UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/10 "2020-02-29T02:56:50Z")

</div>

Yeah, I think this is completed. Although based on my experience hacking on CredRank during ETHDenver, I’m worried that the new load system goes overboard on DI in a way that is frustrating/has lots of boilerplate for making changes. So I think we may still have more iterations to arrive at a system that feels like a good balance of testability but also legibility / ease of working with. (cc @wchargin)

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [February 29, 2020, 1:48pm UTC](https://discourse.sourcecred.io/t/refactoring-the-load-system/629/11 "2020-02-29T13:48:43Z")

</div>

> [@decentralion](#):
>
> I’m worried that the new load system goes overboard on DI in a way that is frustrating/has lots of boilerplate for making changes

Perhaps a good moment to share some retrospect thoughts :]

Agree there’s frustration/boilerplate left. However, this initiative/refactor started because the previous version was impossible spaghetti, where all the tests would break if you change how the pipeline itself worked. (Such as breaking up the mirror+graph step into two separate steps).

Introducing such pipeline changes have gone from nightmare to doable now, for example [Expose a ReferenceDetector to createPluginGraphs](https://github.com/sourcecred/sourcecred/pull/1673) made a change in the pipeline.

Something that I feel became more effortful is having to declare a Loader interface and object for every plugin. While more work, I think it’s an improvement over the before-situation of directly depending on plugin-internal functionality in the load pipeline.

Another trade-off that creates more effort, is how `LoadContext` is required to be a “dumb” composition class, [not allowed](https://github.com/sourcecred/sourcecred/pull/1658#discussion_r376811872) to implement any logic. It makes for way more robust and easier testing, but is also an extra step and not the most intuitive abstraction.

* * *

A main source of frustration I experience still (before the refactor as well) is how configuration values propagate in multiple formats throughout the entire system. From `src/cli` to `src/core` to `src/backend` to `src/plugins`.

Having the instance system, reworked CLI and a better configuration abstraction, I think will greatly reduce code churn here.

Possibly some of these points will be included in [Productivity / Quality of Life for instance maintainers](https://discourse.sourcecred.io/t/productivity-quality-of-life-release/651). However:

> This initiative is to **save time and make it easier** to maintain a SourceCred instance.

I’ve deliberately not included **easier / time saving for developers** here. So if these further refactors are time consuming and don’t help instance maintainers much, I’m inclined to punt on it for a while.