Problem

When working with CI systems, sometimes you want to connect to a server through SSH and execute some commands. Often, you do it like this

ssh user@host <<'EOL'
	# your commands here
EOL

which will send the commands until the EOL is reached.

Those commands sometimes require to pass some arguments which values you have stored in environmental variables, for example

export AWS_REGION='eu-central-1'
export AWS_ACCOUNT_ID='1234567890'
ssh user@host <<'EOL'
	aws ecr get-login-password | docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
EOL

Now, this command uses AWS_ACCOUNT_ID and AWS_REGION variables. When it's executed on CI server, one could wonder

The answer is remote server [[1]]. Even though those env variables are available on the host machine, they won't be interpolated into the command. The remote server will receive the command and interpolate with variables it knows about.

Solutions

Upload env variables to the server

One of the solutions (that I usually use when working with CircleCI ) is to create a file with all environmental variables the remote server should know about, upload it, source it and then execute the commands which require the presence of the env variables.

Since the CI server needs to have some credentials to connect to the remote server via SSH, I usually use scp to upload the file with environmental variables

Example

# prepare env file
echo "export GITHUB_TOKEN=${GITHUB_TOKEN}" >> .docker.env
echo "export GITHUB_USERNAME=${GITHUB_USERNAME}" >> .docker.env

# upload env file
scp -p .docker.env root@$SSH_REMOTE_HOST:/usr/src/app/.docker.env

ssh user@host <<'EOL'
	cd /usr/src/app
	source ./.docker.env # add required env variables
	echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_USERNAME --password-stdin
	docker compose up -d --pull always
	rm .docker.env # remove the file
EOL

Use ~/.ssh/environment file with PermitUserEnvironment option

You can create a ~/.ssh/environment file and put the env that you want to pass into your connections. The file accepts VAR=value format, no need to export the variables.

However, this configuration file is ignored by default by the SSH server, unless the option PermitUserEnvironment is set to yes on the remote server.

If you have full control over the remote server, you could edit the /etc/ssh/sshd_config file to add/edit this parameter.

You can read more about PermitUserEnvironment in the manual of sshd.

Example

echo "GITHUB_TOKEN=${GITHUB_TOKEN}" > ~/.ssh/environment # replaces existing content
echo "GITHUB_USERNAME=${GITHUB_USERNAME}" >> ~/.ssh/environment # appends to the file

ssh user@host <<'EOL'
	cd /usr/src/app
	echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_USERNAME --password-stdin
	docker compose up -d --pull always
EOL

Use SendEnv and AcceptEnv

~/.ssh/config: (locally)

SendEnv MYVAR

/etc/ssh/sshd_config: (on the remote end)

AcceptEnv MYVAR

Now, whatever the value of $MYVAR locally is, it becomes available in the remote session too.
If you login multiple times, each session will have its own copy of $MYVAR, with possibly different values.

Example

Using local ssh config

echo "SendEnv GITHUB_USERNAME GITHUB_TOKEN" >> ~/.ssh/config
ssh user@host <<'EOL'
	cd /usr/src/app
	echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_USERNAME --password-stdin
	docker compose up -d --pull always
EOL

Sending vars explicitly (without editing local config)

ssh user@host -o "SendEnv GITHUB_TOKEN" -o "SendEnv GITHUB_USERNAME" <<'EOL'
	cd /usr/src/app
	echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_USERNAME --password-stdin
	docker compose up -d --pull always
EOL

Using variables with LC_ prefix

This one uses the SendEnv as well. The default configuration of sshd usually have a setting of AcceptEnv like this

AcceptEnv LANG LC_*

So the remote host will accept any environmental variable starting with LC_ prefix (if no one changed the sshd default config!).
This is, obviously, an exploit of the remote system.

Example

export LC_GITHUB_TOKEN="$GITHUB_TOKEN"
export LC_GITHUB_USERNAME="$GITHUB_USERNAME"

ssh user@host -o "SendEnv LC_*" <<'EOL'
	cd /usr/src/app
	echo $LC_GITHUB_TOKEN | docker login ghcr.io -u $LC_GITHUB_USERNAME --password-stdin
	docker compose up -d --pull always
EOL

[[1]]: Unless the CI uses some additional steps of processing parameters passed into command. For example, when using parameters with CircleCI, they are evaluated before the script would run, so the values would be basically replaced with the values coming from parameters.

To see more CI tricks (especially for CircleCI), check out this post

Configuration tricks and snippets for CircleCI

I spent a lot of time writing CircleCI configuration files for various projects. Here’s a compilation of tricks/snippets that someone else might find useful. What is CircleCI? If you found yourself on this post and have no idea what CircleCI is - don’t worry! Here’s a short description. CircleCI

arkus.spaceArkadiusz Żmudzin