# Unified reference detection

**URL:** <https://discourse.sourcecred.io/t/unified-reference-detection/301>\
**Category:** Initiatives\
**Tags:** 🏗️in-progress\
**Created:** [October 19, 2019, 4:57pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301 "2019-10-19T16:57:36Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [October 19, 2019, 4:57pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/1 "2019-10-19T16:57:36Z")

</div>

## Status: in progress

The core types and plumbing have been done. However existing plugins (discourse, github) have not migrated to use it yet. When they’re migrated this can be considered completed.

## Champion[?](https://discourse.sourcecred.io/t/about-champions-and-heroes/291):

![Avatar Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/45/98_2.png "Beanow") @Beanow

## Description:

In a recent update Discourse reference detection was added. Where GitHub detection was already present. However currently the two don’t leave their own ecosystem. GitHub can’t link to Discourse and vice-versa.

This proposes a system where each plugin can instead define it’s “referencable” types and reference detection functions to SourceCred’s core, allowing the core to cross-reference between different plugins

## Benefits:

- Cross-plugin reference detection.
- Makes it easier for new plugins to add more reference detection.

## Implementation plan:

- Discuss the system requirements.
- Implement abstractions in SourceCred core.
- Move existing plugins (GitHub, Discourse) over to use it.

## Deliverables:

- Abstractions in SourceCred core.
- Compatible GitHub and Discourse plugins.

## Dependencies:

- [Refactoring the load system](https://discourse.sourcecred.io/t/refactoring-the-load-system/629)

## References:

- [Discourse reference detection](https://discourse.sourcecred.io/t/discourse-reference-mention-detection/270)
- GitHub reference detection

## Contributions:

- [Chat about API with decentralion](https://discourse.sourcecred.io/t/unified-reference-detection/301/4)
- [UniRef: declare ReferenceDetector interface](https://github.com/sourcecred/sourcecred/pull/1508)
- [UniRef: add MappedReferenceDetector implementation](https://github.com/sourcecred/sourcecred/pull/1509)
- [UniRef: add CascadingReferenceDetector implementation](https://github.com/sourcecred/sourcecred/pull/1510)
- [UniRef: adding TranslatingReferenceDetector implementation](https://github.com/sourcecred/sourcecred/pull/1511)
- [UniRef: implement DiscourseReferenceDetector](https://github.com/sourcecred/sourcecred/pull/1530)
- ~~[UniRef: implement Initiatives reference detector](https://github.com/sourcecred/sourcecred/pull/1531)~~
- [UniRef: add urlReferenceMap to RelationalView](https://github.com/sourcecred/sourcecred/pull/1542)
- [UniRef: add GithubReferenceDetector](https://github.com/sourcecred/sourcecred/pull/1543)
- [Expose a ReferenceDetector to createPluginGraphs](https://github.com/sourcecred/sourcecred/pull/1673)

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [October 19, 2019, 5:08pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/2 "2019-10-19T17:08:35Z")

</div>

In terms of requirements, currently Discourse has one EdgeType per combination of source and destination node. With names and weights for all of them.

As is, having to define each of those edges and needing code to select those edge types, this scales rather poorly when going cross-plugin.

A possible alternative to keep the best of both worlds would be a system where you can create a `{sourceNodePrefix, destinationNodePrefix}` selector for edges and override some of it’s parameters, like weights, or the forward and backward names.

This would allow for example a source of “anything discourse” and a destination of “discourse user” to be called a “mention” instead of “reference”. Similar for GitHub users.

Another requirement. Some data is only available to the plugin itself.

For example, a Discourse posts’ ID does not appear in the URLs. We need to look this up in the mirrored database.

Meaning, only the plugin of the link destination can realistically gather all the data about that destination. I think plugins therefore should have some kind of way to register a function that allows reference detection into their destinations.

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [October 22, 2019, 2:35pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/3 "2019-10-22T14:35:10Z")

</div>

A possible requirement is to check for existence and/or timestamp of a node. See [https://github.com/sourcecred/sourcecred/pull/1417#discussion\_r336755808](https://github.com/sourcecred/sourcecred/pull/1417#discussion_r336755808)

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [December 14, 2019, 6:27pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/4 "2019-12-14T18:27:06Z")

</div>

Had a chat with @decentralion about the API for the unified reference detection and how this should interact with the loading code.

We’ve concluded that the reference detectors should have a dependency on the mirrored data. And the loading code should separate the mirror updating stage from the graph creation stage.

The reference detector would be created as a result from updating the mirror. And a continuation would be provided, which can accept a combination of all reference detectors.

```auto
oldLoad :: () => Promise<Graph>
newLoad :: () => Promise<[
  ReferenceDetector,
  (ReferenceDetector) => Promise<Graph>
]>

```

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [December 14, 2019, 6:43pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/5 "2019-12-14T18:43:43Z")

</div>

Updated status to planned. And offering to / adding myself as Champion for it. Conservative guess is it will carry over the holidays, but first work on it should start around next week.

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [December 21, 2019, 2:46pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/6 "2019-12-21T14:46:11Z")

</div>

Updated to in progress. Currently I’ve implemented the reference detectors locally in a prototype.

Supplying the necessary data to these detectors creates an issue with how the loading currently works. Adding [Static or dynamic loading refactor](https://github.com/sourcecred/sourcecred/issues/1506) as a dependency for this.

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [December 22, 2019, 8:51pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/7 "2019-12-22T20:51:39Z")

</div>

Added a first wave of PRs with building blocks in the core.

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [January 15, 2020, 7:44pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/8 "2020-01-15T19:44:21Z")

</div>

Tracking another set of PRs

Wondering, whether the PR that was closed (because file-based initiatives make it unnecessary) should be listed 🤔

> <https://github.com/sourcecred/sourcecred/pull/1531>

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [January 20, 2020, 2:25pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/9 "2020-01-20T14:25:31Z")

</div>

As a status update, responding to:

> [@s\_ben](#):
>
> Or are we ready to beta test the Initiatives plugin @Beanow?

There’s still a fair amount of technical work to do I’m afraid.

[Initiatives](https://discourse.sourcecred.io/t/write-the-initiatives-plugin/269) depends on reference detection: [Unified reference detection](https://discourse.sourcecred.io/t/unified-reference-detection/301)

And reference detection depends on a refactor of loading:

- [Static or dynamic loading refactor · Issue #1506 · sourcecred/sourcecred · GitHub](https://github.com/sourcecred/sourcecred/issues/1506)
- [Reduce load complexity · Issue #1559 · sourcecred/sourcecred · GitHub](https://github.com/sourcecred/sourcecred/issues/1559)

I’ve submitted a wave of changes (14 PRs, took me ~3 weeks so far) for said loading refactor this weekend. I’m expecting there will be some iterations of that required.

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [February 28, 2020, 12:53pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/10 "2020-02-28T12:53:44Z")

</div>

Added: [Expose a ReferenceDetector to createPluginGraphs](https://github.com/sourcecred/sourcecred/pull/1673)

---

<div class="post-metadata">

**Author:** ![Beanow](https://yyz2.discourse-cdn.com/flex032/user_avatar/discourse.sourcecred.io/beanow/32/98_2.png) [@Beanow](https://discourse.sourcecred.io/u/Beanow)\
**Post date:** [February 28, 2020, 3:28pm UTC](https://discourse.sourcecred.io/t/unified-reference-detection/301/11 "2020-02-28T15:28:55Z")

</div>

Added status update:

> The core types and plumbing have been done. However existing plugins (discourse, github) have not migrated to use it yet. When they’re migrated this can be considered completed.