Unified reference detection

The CredSperiment Initiatives
1

Status: in progress

The core types and plumbing have been done. However existing plugins (discourse, github) have not migrated to use it yet. When they’re migrated this can be considered completed.

Champion?:

Avatar Beanow @Beanow

Description:

In a recent update Discourse reference detection was added. Where GitHub detection was already present. However currently the two don’t leave their own ecosystem. GitHub can’t link to Discourse and vice-versa.

This proposes a system where each plugin can instead define it’s “referencable” types and reference detection functions to SourceCred’s core, allowing the core to cross-reference between different plugins

Benefits:

  • Cross-plugin reference detection.
  • Makes it easier for new plugins to add more reference detection.

Implementation plan:

  • Discuss the system requirements.
  • Implement abstractions in SourceCred core.
  • Move existing plugins (GitHub, Discourse) over to use it.

Deliverables:

  • Abstractions in SourceCred core.
  • Compatible GitHub and Discourse plugins.

Dependencies:

References:

Contributions:

1 Like
2

In terms of requirements, currently Discourse has one EdgeType per combination of source and destination node. With names and weights for all of them.

As is, having to define each of those edges and needing code to select those edge types, this scales rather poorly when going cross-plugin.

A possible alternative to keep the best of both worlds would be a system where you can create a {sourceNodePrefix, destinationNodePrefix} selector for edges and override some of it’s parameters, like weights, or the forward and backward names.

This would allow for example a source of “anything discourse” and a destination of “discourse user” to be called a “mention” instead of “reference”. Similar for GitHub users.

Another requirement. Some data is only available to the plugin itself.

For example, a Discourse posts’ ID does not appear in the URLs. We need to look this up in the mirrored database.

Meaning, only the plugin of the link destination can realistically gather all the data about that destination. I think plugins therefore should have some kind of way to register a function that allows reference detection into their destinations.

3

A possible requirement is to check for existence and/or timestamp of a node. See https://github.com/sourcecred/sourcecred/pull/1417#discussion_r336755808

4

Had a chat with @decentralion about the API for the unified reference detection and how this should interact with the loading code.

We’ve concluded that the reference detectors should have a dependency on the mirrored data. And the loading code should separate the mirror updating stage from the graph creation stage.

The reference detector would be created as a result from updating the mirror. And a continuation would be provided, which can accept a combination of all reference detectors.

oldLoad :: () => Promise<Graph>
newLoad :: () => Promise<[
  ReferenceDetector,
  (ReferenceDetector) => Promise<Graph>
]>
5

Updated status to planned. And offering to / adding myself as Champion for it. Conservative guess is it will carry over the holidays, but first work on it should start around next week.

6

Updated to in progress. Currently I’ve implemented the reference detectors locally in a prototype.

Supplying the necessary data to these detectors creates an issue with how the loading currently works. Adding Static or dynamic loading refactor as a dependency for this.

7

Added a first wave of PRs with building blocks in the core.

8

Tracking another set of PRs

Wondering, whether the PR that was closed (because file-based initiatives make it unnecessary) should be listed :thinking:

9

As a status update, responding to:

There’s still a fair amount of technical work to do I’m afraid.

Initiatives depends on reference detection: Unified reference detection

And reference detection depends on a refactor of loading:

I’ve submitted a wave of changes (14 PRs, took me ~3 weeks so far) for said loading refactor this weekend. I’m expecting there will be some iterations of that required.

1 Like
10

Added: Expose a ReferenceDetector to createPluginGraphs

11

Added status update:

The core types and plumbing have been done. However existing plugins (discourse, github) have not migrated to use it yet. When they’re migrated this can be considered completed.