Deploying Sourcegraph executors on linux machines

Installation

Note: See offline installation guide for instructions on how to install executors in an air-gapped environment.

The following steps will guide you through the process of installing executors on a linux machine.

Dependencies

In order to run executors on your machine, a few things need to be set up correctly before proceeding.

  • Executors only support linux-based machine with amd64 processors
  • Docker has to be installed on the machine (curl -fsSL https://get.docker.com | sh)
  • Git has to be installed at a version >= v2.26
  • The ability to run commands as root on the host machine and configure networking routes

If Firecracker isolation will be used: (recommended)

  • The host has to support KVM (for AWS that means a metal instance, on GCP that means enabling nested virtualization)
  • The following additional dependencies need to be installed:
    • dmsetup
    • losetup
    • mkfs.ext4
    • iptables
    • strings (part of binutils)
    • systemd (optional)

Step 0: Confirm that virtualization is enabled (if using Firecracker)

KVM (virtualization) support is required for our sandboxing model with Firecracker. The following command checks whether virtualization is enabled on the machine (it should print something):

BASH
$ lscpu | grep Virtualization
 
Virtualization:                  VT-x
Virtualization type:             full

On Ubuntu-based distributions, you can also use the tool kvm-ok available in the cpu-checker package to reliably validate KVM support on your host:

BASH
# Install cpu-checker
$ apt-get update && apt-get install -y cpu-checker
 
# Check for KVM support
$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Step 1: Download the latest executor binary

Below are the download links for the latest release of executors:

Note: Executors need to match the version of Sourcegraph they're running against. Latest will usually only work for you when you run the latest version of Sourcegraph.

Download and setup the executor binary:

BASH
# Plug in the version of your Sourcegraph instance, like v4.1.0.
# Before Sourcegraph 3.43.0, tagged releases of executors are not available and you should default to using "latest" instead.
# Using latest is NOT recommended, because it might be incompatible with your Sourcegraph version.
curl -sfLo executor https://storage.googleapis.com/sourcegraph-artifacts/executor/${SOURCEGRAPH_VERSION}/linux-amd64/executor
chmod +x executor
# Assuming /usr/local/bin is in $PATH.
mv executor /usr/local/bin

Step 2: Setup environment variables

The executor Linux binary is configured through environment variables which need to be passed to it when you run it (including for the install, validate and test-vm comamnds). You can add these to your shell profile, or an environment file. The EXECUTOR_FRONTEND_URL, EXECUTOR_FRONTEND_PASSWORD and EXECUTOR_QUEUE_NAME or EXECUTOR_QUEUE_NAMES are required and will need to be set prior to running the executor service for the first time.

See Executor configuration for a full list of configuration options for the executor service.

BASH
# Example:
export EXECUTOR_FRONTEND_URL=http://sourcegraph.yourcompany.com
export EXECUTOR_FRONTEND_PASSWORD=SUPER_SECRET_SHARED_TOKEN
export EXECUTOR_QUEUE_NAMES=batches,codeintel

Step 3: Configure your machine

To be able to run workloads in isolation, a few dependencies need to be installed and configured. The executor CLI can do all of that automatically.

To run all of the required setup steps, just run the following commands as root:

BASH
executor install all
# OR run the following, to see how to install/configure components separately.
executor install --help

Step 4: Validate your machine is ready to receive workloads

All set up! Before letting the executor start receiving workloads from your Sourcegraph instance, you might want to verify your setup. Run the following command:

BASH
executor validate

If any issues are found, correct them before proceeding.

If you use our sandboxing model with Firecracker (recommended), you can also verify that the executor is able to spin up the isolation VMs properly. For that, use the following command:

BASH
# Optionally provide a repo to clone into the VMs workspace, to verify that cloning works properly as well.
executor test-vm [--repo=github.com/sourcegraph/sourcegraph --revision=main]

This should succeed and print a command to use to attach to the guest VM. If it is able to spin up properly, that is usually good indication that everything is set up properly. If you need to debug some issue further, this can be helpful as you can exec into the guest VM from here.

Step 5: Make executor a systemd service (optional, systemd-based distributions only)

To make sure that the executor runs post-boot of the machine, you might want to wrap it in a systemd service. This is an example of how that could look like:

BASH
cat <<EOF >/etc/systemd/system/executor.service
[Unit]
Description=Sourcegraph executor
After=docker.service
BindsTo=docker.service
 
[Service]
ExecStart=/usr/local/bin/executor
Restart=on-failure
EnvironmentFile=/etc/systemd/system/executor.env
Environment=HOME="%h"
Environment=SRC_LOG_LEVEL=dbug
 
[Install]
WantedBy=multi-user.target
EOF
 
# Create environment file (this can also be sourced in your shell)
cat <<EOF >/etc/systemd/system/executor.env
EXECUTOR_FRONTEND_URL="https://sourcegraph.yourcompany.com"
EXECUTOR_FRONTEND_PASSWORD="SUPER_SECRET_SHARED_TOKEN"
EXECUTOR_QUEUE_NAMES="batches,codeintel"
EOF
 
systemctl enable executor

Step 6: Start receiving workloads

If you use the systemd service, simply run systemctl start executor, otherwise run executor run. Your executor should start listening for jobs now! All done!

Upgrading executors

Upgrading executors is relatively uninvolved. Simply follow the instructions below. Also, check the changelog for any Executors related breaking changes or new features that you might want to configure.

Step 1: First, grab the executor binary for the new target Sourcegraph version.

NOTE: Keep in mind that only one minor version bumps are guaranteed to be disruption-free.

BASH
curl -sfLo executor https://storage.googleapis.com/sourcegraph-artifacts/executor/${SOURCEGRAPH_VERSION}/linux-amd64/executor
chmod +x executor
# Assuming /usr/local/bin is in $PATH.
mv executor /usr/local/bin

Step 2: Make sure all ambient dependencies and configurations are up-to-date:

Ensure env vars has been configured.

BASH
executor install all
# OR run the following, to see how to install/configure components separately.
executor install --help

Step 3: Validate your machine is ready to receive workloads

All set up! Before letting the executor start receiving workloads from your Sourcegraph instance, you might want to verify your setup. Run the following command:

BASH
executor validate

Step 4: Restart your running executor / spin up a new machine

Depending on how you set up executors, you might want to restart the systemd service, or restart/replace the machine running them, so the new binary is running. If you use the systemd service, simply run systemctl start executor, otherwise run executor run. Your executor should start listening for jobs now and be visible under the Executors > Instances section of the Site Configuration.