Adding precise code navigation to CI/CD workflows

Language-specific guides

We are working on creating language specific guides for use with different indexers, so make sure to check for the documentation for your language! If there isn't a guide for your language, this general guide will help you through the precise code navigation setup process.

Benefits of CI integration

Setting up a source code indexing job in your CI build provides you with fast code navigation that gives you more control on when source code gets indexed, and ensures accuracy of your code navigation by keeping in sync with changes in your repository. Due to the large number of CI frameworks and languages we may not have specific documentation for your use case. Feel free to file an issue if you're having difficulties and we can help troubleshoot your setup.

Using indexer containers

We're working on creating containers that bundle indexers and the Sourcegraph CLI (src). All the languages in our language-specific guides are supported, and this section provides a general guide to using those containers. We've pinned the containers to the latest tag in these samples so they stay up to date, but make sure you pin them before going live to ensure reproducibility.

Basic usage

Some indexers will work out of the box by just running them in your project root. Here's an example using GitHub actions and Go:

YAML
jobs:
  scip-go:
    #this line will prevent forks of this repo from uploading lsif indexes
    if: github.repository == '<insert your repo name>'
    runs-on: ubuntu-latest
    container: sourcegraph/scip-go:latest
    steps:
      - uses: actions/checkout@v1
      - name: Generate index
        run: scip-go
      - name: Install src CLI
        run: |
          curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src
          chmod +x /usr/local/bin/src
      - name: Upload index
        # this will upload to Sourcegraph.com, you may need to substitute a different command
        # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
        run: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure

Sub-projects

If your repository contains multiple code projects in different folders, your CI framework likely provides a "working directory" option so that you can do something like:

YAML
jobs:
  scip-go:
    # Prevent forks of this repo from uploading SCIP indexes
    if: github.repository == '<insert your repo name>'
    runs-on: ubuntu-latest
    container: sourcegraph/scip-go:latest
    steps:
      - uses: actions/checkout@v1
      - name: Generate index
        working-directory: backend/
        run: scip-go
      - name: Install src CLI
        working-directory: backend/
        run: |
          curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src
          chmod +x /usr/local/bin/src
      - name: Upload index
        # note that the upload command also needs to happen in the same directory!
        working-directory: backend/
        # this will upload to Sourcegraph.com, you may need to substitute a different command
        # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
        run: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure

Custom build environments

Depending on which language you're using, you may need to perform some pre-indexing steps in a custom build environment. You have a couple options for this:

• Add the indexer and Sourcegraph CLI (src) to your environment. To make this easier, you could either directly build a Docker image from our indexer container, or use its Dockerfile as inspiration for getting the tools installed in your environment.
• Do the pre-indexing steps in your environment, and then make those artifacts available to our container.

This second step is easy in GitHub actions because our container can be used as an action. Here's an example for a TypeScript project:

YAML
jobs:
  scip-typescript:
    # this line will prevent forks of this repo from uploading scip indexes
    if: github.repository == '<insert your repo name>'
    runs-on: ubuntu-latest
    container: my-awesome-container
    steps:
      - uses: actions/checkout@v1
      - name: Install dependencies
        run: <install dependencies>
      - name: Generate index
        uses: docker://sourcegraph/scip-typescript:latest
        with:
          args: scip-typescript index
      - name: Upload index
        uses: docker://sourcegraph/src-cli:latest
        with:
          # this will upload to Sourcegraph.com, you may need to substitute a different command
          # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
          args: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure

GitHub Action Examples

The following projects have example GitHub Action workflows to generate and upload indexes.

• elastic/kibana
• golang/go
• kubernetes/kubernetes
• lodash/lodash
• moby/moby
• ReactiveX/IxJS

Circle CI Examples

Certain frameworks like Circle CI may require you to explicitly cache artifacts between jobs. In CircleCI this might look like the following:

YAML
version: 2.1
 
jobs:
  install-deps:
    docker:
      - image: my-awesome-container
    steps:
      - checkout
      - <install dependencies>
      - save_cache:
          paths:
            - node_modules
          key: dependencies
 
jobs:
  scip-typescript:
    docker:
      - image: sourcegraph/scip-typescript:latest
    steps:
      - checkout
      - restore_cache:
          keys:
            - dependencies
      - run: scip-typescript index
        # this will upload to Sourcegraph.com, you may need to substitute a different command
        # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
      - run: src code-intel upload -github-token=<<parameters.github-token>> -ignore-upload-failure
 
workflows:
  scip-typescript:
    jobs:
      - install-deps
      - scip-typescript:
          requires:
            - install-deps

The following projects have example CircleCI configurations to generate and upload indexes.

• angular/angular
• facebook/jest
• facebook/react
• grafana
• helm
• prometheus/prometheus
• ReactiveX/rxjs

Travis CI Examples

The following projects have example Travis CI configurations to generate and upload indexes.

• aws/aws-sdk-go
• etcd-io/etcd
• expressjs/express
• hugo/hugo
• Microsoft/TypeScript
• moment/moment
• sindresorhus/got

CI from scratch

If you're indexing a language we haven't documented yet in our language-specific guides, follow the instructions in this section. We want to have containers available for every language with a robust indexer, so please also consider filing an issue to let us know we're missing one.

Set up your CI machines

Your CI machines will need two command-line tools installed. Depending on your build system setup, you can do this as part of the CI step, or you can add it directly to your CI machines for use by the build.

1. The Sourcegraph CLI (src).
2. The indexer for your language.

Add steps to your CI

1. Generate an index for a project within your repository by running the indexer in the project directory (consult your indexer's docs).
2. Upload that generated index to your Sourcegraph instance.

Recommended upload frequency

We recommend that you start with a CI job that runs on every commit (including branches).

If you see too much load on your CI, your Sourcegraph instance, or a rapid decrease in free disk space on your Sourcegraph instance, you can instead index only the default branch, or set up a periodic job (e.g. daily) in CI that indexes your default branch.

With periodic jobs, you should still receive precise code navigation on non-indexed commits on lines that are unchanged since the nearest indexed commit. This requires that the indexed commit be a direct ancestor or descendant no more than 100 commits away. If your commit frequency is too high and your index frequency is too low, you may find commits with no precise code navigation at all. In this case, we recommend you try to increase your index frequency if possible.

Uploading indexes to Sourcegraph.com

Indexes can be uploaded to a self-hosted Sourcegraph instance or to Sourcegraph.com. Using the Sourcegraph.com endpoint will surface code navigation for your public repositories directly on GitHub via the Sourcegraph browser extension and at https://sourcegraph.com/github.com/<your-username>/<your-repo>.

Using the Sourcegraph.com endpoint is free and your index is treated as User-Generated Content (you own it, as covered in our Sourcegraph.com terms of service). If you run into trouble, or a situation arises where you need all of your index expunged, please reach out to us at support@sourcegraph.com.