SourceCred Instance System

Status: Proposal

Currently documenting the initiative and implementation plan. Next step is working on a design doc.


Avatar Beanow @Beanow


The SourceCred instances system is a way to simplify the workflows around creating and updating “instances”.

Currently, SourceCred uses a data directory which can hold any number of “projects”. The caller is responsible for providing the project config, weights, initiatives and possibly the project ID depending on the command.

Instead, an “instance” would take the form of a directory that SourceCred manages. It holds only one project along with all of it’s config, weights, initiatives, data, output, etc. Which we’d know is an instance by a configuration file in the root. Very similar to how a git repo has a .git folder in it’s root. Or how package.json makes something a package. Or how a config.yml is used in the root of a jekyll / hugo website.

So with that, you would no longer need to point SourceCred to information about your project per command, you simply need to call SourceCred commands from an instance directory.

$ cd path/to/your/instance
$ sourcecred update

Implementation plan

  1. Create a design doc describing the interface:
  • Configuration outline
  • Folder structure
  • New CLI
  1. Develop new constructs side-by-side with existing ones.
  2. Deprecate previous approach.
  3. Remove previous approach.







About export-frontend. I’ve been wondering for a bit if the explorer shouldn’t be a pre-compiled module you can embed. Or host separately. Only pointing to data.

As currently every “instance” includes a lot of extra stuff too. Like The into page here, which is largely superseded by to the updated homepage. I think most instances will link only to the project list or directly to an explorer instance.

Currently we’re also using server-side rendering to have URLs for these. However I think many more parameters are interesting to create URLs for. Like narrowing down to a timeframe, or linking to the github repositories view. Something like using GET parameters or hash parameters make sense to use instead. What that would also allow is to be completely driven by static files in the data folder. e.g. no need to use webpack to generate folders and minimal HTML files just so you can have your URLs the way you’d like. Which would eliminate the need for end-users to even compile the frontend.

1 Like

Yep, that sounds great to me. I think maybe we should scope out a sub-initiative which does the precursor work:

  • Remove the ability for a SourceCred instance to map to multiple projects (which requires the build step to compile the routes in)
  • Remove the old intro page (which is superseded by
  • Pre-compile the frontend and commit it (?)

I’m not exactly sure what the right intermediate state there is between where we are now and having the instnace system, but it feels like doing these 3 steps would be a good start.

Cross referencing earlier discussion:

@Beanow, you should be able to add it directly to the references list on the initiative (it’s setup as a wiki post)

Good point. Didn’t realize that was what you were going for with the References section till filling out the template once.

A quick question - is a full fledged running instance really necessary? In that we can produce a static prototype with data, and have some job run every day (or more) why not just have a workflow that takes a set of parameter files, runs at some frequency, and updates a static interface? Usage would come down to a community forking a repository, editing the configuration files, and letting it run. This seems much more likely to be adopted than requiring someone running an actual server, at least for academic / open source projects.

This general need (and keeping the intended communities in mind) fits in really well with an action on the GitHub marketplace - essentially we would have an action repository with the appropriate action.yml file to define inputs and outputs, and then it would also use the sourcecred/sourcecred:dev container base to run the commands. The user would then enable it for a repository by adding a few lines to a workflow, and defining the parameters.

Think there’s some confusion there. It’s not about running a dedicated server :]

Currently we use a bunch of glue scripts and you need to provide lots of parameters to every call, and update the data and site with bin/sourcecred load and yarn build. What we’d want is to get rid of a lot of this glue and make a first-class “instance”.

Think of it like with git. There’s git the tool you’ve installed. But there’s also local git repositories on your filesystem. This equivalent of a local repository is what an “instance” is. And to go with that, we’d need commands that work with those.

(Indeed you can store those instances and github, and have CI update those instances :smile:)

There’s a discussion about possible CLIs here:

Ah ok, my misunderstanding! I associate instance with cloud instance, which usually refers to a server.

And I think I’m understanding the goals and how this could be done, give me a few days to play around and I’ll report back.

Want to adopt this initiative and Champion it :smiley:

1 Like

Aren’t we both working on this @Beanow? I would suggest we both are assigned. On one of the calls I remember a mention that we would also try to work together. I hope that hasn’t changed, because you are awesome to work with! :slight_smile:

Yes. Working on this together.

Because the new interfaces are relevant for both the changes to the JS codebase and action, we’ll spec it out. From there I will lead on the JS side, I believe actions are more your lead.

This initiative is not up to date, and it only mentions the JS side. First thing I would like to do is fix that. I am not on SourceCred for a few days, then I will clear this up :]

Sounds like a plan! Have a good tiny break, I’m going to be working on other fun projects this weekend too :slight_smile:

I’ve updated the descriptions here. It’s a bit more clear how it covers the instance system specifically. And created The SourceCred GitHub Action to track the GitHub action initiative.

They are both part of a common goal, which is this initiative: Productivity / Quality of Life for instance maintainers

Taking from my notes of the previous Team Call.

The next step for this and The SourceCred GitHub Action

Write a design doc for instance system (interface only).

  • Make config spec
  • Make dir structure spec
  • Make CLI spec
  • Request: avoid clutter in git history

@vsoch and I plan to work on this, and ask for review before moving on to implementations :]