# Adding precise code navigation to CI/CD workflows

## Language-specific guides

We are working on creating language specific guides for use with different indexers, so make sure to check for the documentation for your language! If there isn't a guide for your language, this general guide will help you through the precise code navigation setup process.

## Benefits of CI integration

Setting up a source code indexing job in your CI build provides you with fast code navigation that gives you more control on when source code gets indexed, and ensures accuracy of your code navigation by keeping in sync with changes in your repository. Due to the large number of CI frameworks and languages we may not have specific documentation for your use case. Feel free to [contact us](https://help.sourcegraph.com/hc/en-us/requests/new) if you're having difficulties and we can help troubleshoot your setup.

## Using indexer containers

We're working on creating containers that bundle indexers and the [Sourcegraph CLI (`src`)](https://github.com/sourcegraph/src-cli). All the languages in our [language-specific guides](/code-navigation/precise-code-navigation#setting-up-code-navigation-for-your-codebase) are supported, and this section provides a general guide to using those containers. We've pinned the containers to the `latest` tag in these samples so they stay up to date, but make sure you pin them before going live to ensure reproducibility.

### Basic usage

Some indexers will work out of the box by just running them in your project root. Here's an example using GitHub actions and Go:

```yaml
jobs:
    scip-go:
        #this line will prevent forks of this repo from uploading lsif indexes
        if: github.repository == '<insert your repo name>'
        runs-on: ubuntu-latest
        container: sourcegraph/scip-go:latest
        steps:
            - uses: actions/checkout@v1
            - name: Generate index
              run: scip-go
            - name: Install src CLI
              run: |
                  curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src
                  chmod +x /usr/local/bin/src
            - name: Upload index
              # this will upload to Sourcegraph.com, you may need to substitute a different command
              # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
              run: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure
```

### Sub-projects

If your repository contains multiple code projects in different folders, your CI framework likely provides a "working directory" option so that you can do something like:

```yaml
jobs:
    scip-go:
        # Prevent forks of this repo from uploading SCIP indexes
        if: github.repository == '<insert your repo name>'
        runs-on: ubuntu-latest
        container: sourcegraph/scip-go:latest
        steps:
            - uses: actions/checkout@v1
            - name: Generate index
              working-directory: backend/
              run: scip-go
            - name: Install src CLI
              working-directory: backend/
              run: |
                  curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src
                  chmod +x /usr/local/bin/src
            - name: Upload index
              # note that the upload command also needs to happen in the same directory!
              working-directory: backend/
              # this will upload to Sourcegraph.com, you may need to substitute a different command
              # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
              run: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure
```

### Custom build environments

Depending on which language you're using, you may need to perform some pre-indexing steps in a custom build environment. You have a couple options for this:

-   Add the indexer and [Sourcegraph CLI (`src`)](https://github.com/sourcegraph/src-cli) to your environment. To make this easier, you could either directly build a Docker image from our indexer container, or use its Dockerfile as inspiration for getting the tools installed in your environment.
-   Do the pre-indexing steps in your environment, and then make those artifacts available to our container.

This second step is easy in GitHub actions because our container can be used as an action. Here's an example for a TypeScript project:

```yaml
jobs:
    scip-typescript:
        # this line will prevent forks of this repo from uploading scip indexes
        if: github.repository == '<insert your repo name>'
        runs-on: ubuntu-latest
        container: my-awesome-container
        steps:
            - uses: actions/checkout@v1
            - name: Install dependencies
              run: <install dependencies>
            - name: Generate index
              uses: docker://sourcegraph/scip-typescript:latest
              with:
                  args: scip-typescript index
            - name: Upload index
              uses: docker://sourcegraph/src-cli:latest
              with:
                  # this will upload to Sourcegraph.com, you may need to substitute a different command
                  # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
                  args: src code-intel upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure
```

### Circle CI Examples

Certain frameworks like Circle CI may require you to explicitly cache artifacts between jobs. In CircleCI this might look like the following:

```yaml
version: 2.1

jobs:
  install-deps:
    docker:
      - image: my-awesome-container
    steps:
      - checkout
      - <install dependencies>
      - save_cache:
          paths:
            - node_modules
          key: dependencies

jobs:
  scip-typescript:
    docker:
      - image: sourcegraph/scip-typescript:latest
    steps:
      - checkout
      - restore_cache:
          keys:
            - dependencies
      - run: scip-typescript index
        # this will upload to Sourcegraph.com, you may need to substitute a different command
        # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
      - run: src code-intel upload -github-token=<<parameters.github-token>> -ignore-upload-failure

workflows:
  scip-typescript:
    jobs:
      - install-deps
      - scip-typescript:
          requires:
            - install-deps
```

## CI from scratch

If you're indexing a language we haven't documented yet in our [language-specific guides](/code-navigation/precise-code-navigation#setting-up-code-navigation-for-your-codebase), follow the instructions in this section. We want to have containers available for every language with a robust indexer, so please [contact](https://help.sourcegraph.com/hc/en-us/requests/new) to let us know we're missing one.

### Set up your CI machines

Your CI machines will need two command-line tools installed. Depending on your build system setup, you can do this as part of the CI step, or you can add it directly to your CI machines for use by the build.

1. The [Sourcegraph CLI (`src`)](https://github.com/sourcegraph/src-cli).
2. The [indexer](/code-navigation/writing-an-indexer) for your language.

### Add steps to your CI

1. **Generate an index** for a project within your repository by running the indexer in the project directory (consult your indexer's docs).
1. **[Upload that generated index](/code-navigation/precise-code-navigation#setting-up-code-navigation-for-your-codebase)** to your Sourcegraph instance.

## Recommended upload frequency

We recommend that you start with a CI job that runs on every commit (including branches).

If you see too much load on your CI, your Sourcegraph instance, or a rapid decrease in free disk space on your Sourcegraph instance, you can instead index only the default branch, or set up a periodic job (e.g. daily) in CI that indexes your default branch.

With periodic jobs, you should still receive precise code navigation on non-indexed commits on lines that are unchanged since the nearest indexed commit. This requires that the indexed commit be a direct ancestor or descendant no more than [100 commits](https://github.com/sourcegraph/sourcegraph-public-snapshot/blob/e7803474dbac8021e93ae2af930269045aece079/lsif/src/shared/constants.ts#L25) away. If your commit frequency is too high and your index frequency is too low, you may find commits with no precise code navigation at all. In this case, we recommend you try to increase your index frequency if possible.

## Uploading indexes to Sourcegraph.com

Indexes can be uploaded to a self-hosted Sourcegraph instance or to [Sourcegraph.com](https://sourcegraph.com). Using the [Sourcegraph.com](https://sourcegraph.com) endpoint will surface code navigation for your public repositories directly on GitHub via the [Sourcegraph browser extension](/integration/browser-extension) and at `https://sourcegraph.com/github.com/<your-username>/<your-repo>`.

Using the [Sourcegraph.com](https://sourcegraph.com) endpoint is free and your index is treated as User-Generated Content (you own it, as covered in our [Sourcegraph.com terms of service](https://about.sourcegraph.com/terms-dotcom#3-proprietary-rights-and-licenses)). If you run into trouble, or a situation arises where you need all of your index expunged, please reach out to us at [support@sourcegraph.com](mailto:support@sourcegraph.com).