SSH Tailscale magic

It really did feel like magic when I was able to configure my EC2 workstations with no inbound rules and still connect to them privately and securely from my laptop. This is possible with the help of Tailscale and a bit of automation.

I do recommended reading the posts on how Tailscale works and more specifically, how NAT traversal works, for more context on the WireGuard® VPN protocol and how Tailscale establishes secure connections.

Prerequisite Setup

Before we go ahead, let’s make sure we have the neccessary environments.

• Set up the Tailscale client on your local workstation. There are clients for Linux, macOS, and Windows.
• You’ll need access to a cloud provider. The examples to follow will be for AWS, but the principles can be applied to Azure and GCP as well. The default VPC will suffice for this demonstration. The region used in these examples is eu-west-2.

Client Authentication

The default authentication method in Tailscale is through an interactive login, but we want to use an alternative method to provision infrastructure automatically. Generate a reusable key and store the secret in AWS SystemsManager. SystemsManager is good enough for this purpose and the standard options don’t cost anything compared to SecretsManager.

# generate auth key from Tailscale admin web console

aws ssm put-parameter \
    --name "tailscale-workstation" \
    --type "SecureString" \
    # where authkey-000 is the auth key generated from Tailscale
    --value "authkey-000" \
    --region "eu-west-2"

# parameter ARN
aws ssm get-parameter --name tailscale-workstation --query 'Parameter.ARN' --output text

The EC2 instances will fetch that secret to authenticate with Tailscale. Create an IAM role for EC2 with the following inline policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ssm",
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:eu-west-2:123456789012:parameter/tailscale-workstation"
        }
    ]
}

Launching Instances

In the user data script the instance will be configured to start Tailscale using the pre-auth key that was generated and stored earlier. I was using Ubuntu at the time, so do replace the installation instructions to match your distro of choice. The auth key secret is fetched using the AWS CLI command for SystemsManager and its output is passed to the Tailscale command to connect to the network.

#!/bin/bash
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/focal.gpg | apt-key add -
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/focal.list | tee /etc/apt/sources.list.d/tailscale.list
apt-get update
apt-get install -y tailscale awscli

AUTHKEY=$(aws ssm get-parameter --name tailscale-workstation --with-decryption --query 'Parameter.Value' --output text --region eu-west-2)
tailscale up -authkey $AUTHKEY

The last piece is the aforementioned security group. Create a security group for the default VPC with no inbound rules and an outbound rule allowing all traffic to anywhere. This is my personal workstation setup, so I do need Internet access.

We can now launch our instances! The instances will be provisioned into the default VPC, with the security group with no inbound rules, with the IAM role to allow the retrieval of the secret from SystemsManager, and a key pair for ssh access.

When the instances are successfully launched, you should be able to see them in Tailscale. Find the Tailscale 100.x.y.z IP addresses for the newly registered instances and use it to ssh in with the EC2 key pair.

Final Thoughts

Setting up Tailscale was a breeze. All of the effort went into automating the process, but now I can spin up workstations in the cloud with no public access. The security group does not allow ssh in to the instance’s public IP address and I can connect privately over VPN. I tear down my instances when I’m done and can quickly provision new ones when I’m doing something new. Neat.