Troubleshooting

This docs helps you to troubleshoot and eliminate problems when trying to execute a batch spec with src batch [apply|preview] or managing an already created batch change and its changesets.

Executing batch change steps

Since src batch [apply|preview] executes a batch spec on the host machine on which it is executed (and not on the Sourcegraph instance), there are a lot of different possibilities that can cause it to fail, from missing dependencies to missing credentials when trying to connect to the Sourcegraph instance.

The following questions can be used to determine what's causing the problem.

Is src connected to the right Sourcegraph instance?

Run the following command, replacing sourcegraph.example.com with the URL of your Sourcegraph instance, to make sure src is configured correctly:

BASH
src login https://sourcegraph.example.com

If src is correctly configured, then the output should look similar to the following:

BASH
✔️  Authenticated as my-username on https://sourcegraph.example.com

Are dependencies installed?

When executing batch specs, src uses Docker and Git. Make sure that those are installed and accessible to you on your machine.

To test whether git is installed and accessible, run the following:

BASH
git version

To test whether Docker is installed and configured correctly, run the following:

BASH
docker run hello-world

The above command will pull Docker's hello-world image and try to execute a container using that image. If it works, you should see a Hello from Docker! message telling you your installation works.

Does "Hello World" work?

If executing your batch spec fails and you haven't tested Batch Changes with another batch spec before, it can help to run the Hello World batch change to identify the problem.

Go through the Quickstart to run a batch change that adds Hello World to README files with the following batch spec:

YAML
name: hello-world
description: Add Hello World to READMEs
 
# Find one repository with a README
on:
  - repositoriesMatchingQuery: repohasfile:README count:1
 
# In each repository, run this command. Each repository's resulting diff is captured.
steps:
  - run: IFS=$'\n'; echo Hello World | tee -a $(find -name README)
    container: alpine:3
 
# Describe the changeset (e.g., GitHub pull request) you want for each repository.
changesetTemplate:
  title: Hello World
  body: My first batch change!
  branch: hello-world # Push the commit to this branch.
  commit:
    message: Append Hello World to all README files
  published: false

If that doesn't work, you can exclude the possibility that only something with your batch spec is wrong.

Does it work with a single repository? Five? Ten?

Debugging large batch changes that make changes in hundreds of repositories is hard. To find out whether a problem is related to the size or scope of a batch change or what it's trying to achieve, try reducing the scope of your batch change.

You can do so by changing the on.repositoriesMatchingQuery to yield fewer results or by using a concrete list of repositories with on.repository.

For the former, you can use Sourcegraph's search filters. For example, this query will only yield repositories that have github.com/my-org in their name:

YAML
# [...]
on:
  - repositoriesMatchingQuery: repo:^github.com/my-org

This one will only return a single repository matching that exact name:

YAML
# [...]
on:
  - repositoriesMatchingQuery: repo:^github.com/my-org/my-repo$

That can also be achieved with the mentioned on.repository attribute:

YAML
# [...]
on:
  - repository: github.com/my-org/my-repo1
  - repository: github.com/my-org/my-repo2

Once you know that executing the batch spec works with one or more repositories, you can expand the scope back to its original form and identify which repository execution breaks.

Can you download a repository archive?

If src is stuck in the Initializing workspace phase for a repository or fails to initialize the workspace, try to see whether you can download an archive of the repository manually on your command line with curl:

BASH
curl -L -v -X GET -H 'Accept: application/zip' \
  -H "Authorization: token $SRC_ACCESS_TOKEN" \
  "$SRC_ENDPOINT/github.com/my-org/my-repo@refs/heads/master/-/raw" \
  --output ~/tmp/my-repo.zip

That command is equivalent to what src does under the hood when downloading an archive of github.com/my-org/my-repo@master to execute batch spec steps.

If that fails, it points to the Sourcegraph setup or infrastructure as a likely source of the problem, not src.

Can you manually execute the steps.run command in a container?

If executing the steps.run command fails, you can try to recreate whether executing the step manually in a container works.

An approximation of what src does under the hood is the following command:

BASH
docker run --rm --init --workdir /work \
  --mount type=bind,source=/unzipped-archive-locally,target=/work \
  --mount type=bind,source=/tmp-script,target=/tmp-file-in-container \
  --entrypoint /bin/sh -- <IMAGE> /tmp-file-in-container

Make sure that you put your steps.run command in /tmp-script (or any other location), replace <IMAGE> with the name of the Docker image, and /unzipped-archive-locally (or any other location) with a local copy of the repository in which you want to execute the steps.

Does it work if you switch to using the workspace mode using Docker volumes?

If executing the steps in the batch spec fails with a message that looks similar to this one (i.e., permission error).

BASH
/bin/sh: can't open '/tmp/tmp.IbdkiA': Permission denied

Or if you are in a locked-down environment, it's possible that Docker bind mounts won't work.

Try using the -workspace volume flag (see src batch apply for a list of all flags) to make src use Docker volumes instead:

BASH
src batch apply -workspace volume -f my-spec.yaml
# or:
src batch preview -workspace volume -f my-spec.yaml

If you're using SELinux, then neither workspace is fully supported. See this issue for more details.

Are the Docker images running as different users?

Running steps with images that run with different user IDs is unsupported.

While doing so may work in bind workspace mode on macOS due to specific implementation details of how Docker for Mac mounts from the host filesystem, this is a common source of confusing permission errors similar to the previous step.

Are you on the latest version of Docker?

If not, please update to the latest version of Docker Desktop.

Have you pruned your Docker Build Cache and restarted the Docker Daemon?

If you're experiencing src-cli hanging at the Determining Workspace Type step of the Batch Change, then clearing the Docker build cache using docker builder prune and restarting the Docker Daemon will resolve the issue. Please contact support if this does not resolve your issue.

Using Rancher Desktop or Colima?

If you encounter the error docker: Error response from daemon: invalid mount config for type 'bind': bind source path does not exist, try configuring the env var SRC_BATCH_TMP_DIR to reference an accessible directory.

For Colima:

BASH
export SRC_BATCH_TMP_DIR=/tmp/colima/batchchange

For Rancher Desktop:

BASH
export SRC_BATCH_TMP_DIR=/tmp/rancher-desktop/batchchange

Publishing changesets

Do you have the right credentials?

When publishing changesets fails, make sure that you have configured credentials with all of the required scopes and from an account with write access to the changeset's repository on the code host.

Do you have email privacy enabled on GitHub?

In case you encounter an error informing you that your push was rejected due to the presence of an email address that isn't permitted to be public, there are a couple of possible solutions.

You can choose either to:

• Permit your email address to be public.
• Disable the feature Block command line pushes that expose my email.

Instructions for both of these potential solutions can be found in the GitHub Email Settings.

Debug running batch changes server side

When you run batch changes server-side, you can monitor various execution artifacts, like per-step logs, timing information, and output variables in a persisted, sharable UI.

From the UI, you can start from a high-level perspective for the entire batch spec and then drill down to find the problem.

Start by going to the workspace that failed. Then, you get an overview of all the steps that ran and the ones that failed. From there on, you can dive into logs, previous steps, outputs, and more.