devmachine

How it works

Setting up a server for the first time

Setting up a server you have never logged into is the hardest moment this CLI handles, and the first thing a new user hits. This page explains what devmachine setup does, and why the steps run in this order.

It does not ask which situation you are in

Most providers let you paste an SSH key in when you create the server, so a lot of people arrive with a key that already works — asking them for a root password would ask for something they never had. So setup finds out for itself: where the machine is, which account and port, which key to use, then tries the key first. Only if that fails does it ask for the password.

It checks what the machine runs first

The first thing setup does on a connection is ask the machine what it is: uname -s for the kernel, then ID in /etc/os-release on Linux, or sw_vers -productVersion on a Mac. It sets up debian, ubuntu and arch (and archarm, Arch on ARM), and macOS.

When ID is none of those, it reads ID_LIKE, the systems the distribution says it is based on, closest first. If one of them is in the list, the machine is set up as that one, and setup says nobody tested it:

203.0.113.10 runs manjaro, which is based on arch: it is set up the arch way, but devmachine is not tested on it.

Why accept a derivative: Manjaro, EndeavourOS and CachyOS keep Arch’s pacman and package names, and Linux Mint and Pop!_OS keep Ubuntu’s apt. Refusing them stopped people whose machine would have worked. The sentence is there because it is a guess the CLI has not checked: a derivative can still rename or leave out a package its base has.

Anything else stops the run before the first change — before the key is installed, on a server you reached with a password:

this CLI does not set up "fedora" yet: it supports debian, ubuntu and arch, and systems based on them
"FreeBSD" is not a system this CLI sets up: it supports Linux (debian, ubuntu, arch) and macOS

Why refuse instead of trying: every step after this one is written for a system it has been run on, or for one based on it. A guess gets you halfway through a first run — a key installed, password login off — and then leaves you on a machine that cannot install Ansible. Stopping early leaves the server as you bought it. doctor makes the same check and accepts the same list; on a derivative its operating system check is a warning, not a pass.

The proof is a new connection

Installing a key does not mean it works — a wrong file permission, a world-writable home folder, SELinux, any of these can let a key install cleanly and still refuse every login. The connection that installed the key cannot prove it works either, since that connection used the password. So setup closes it, opens a brand new connection using only the key, and only counts that as proof.

Locking down comes after the proof, never with it

Turning password login off before proving the key works is locking the door with the key still inside. If the proof fails, setup stops and says plainly:

password login is still on: you can still get in with the password

--no-harden skips locking the server down; the key is still installed and proved.

machines add writes the machine last

machines add records the new machine only after the whole bootstrap worked: key proved, password login off, Ansible installed. Both files wait for that moment — config.yml and known_hosts. Until then the host key you approved is trusted in a file of the run’s own, which is thrown away when the run ends. If any step fails, both files are as they were, so you fix the cause and run the same command again: it is not refused as a name already configured, and a corrected address or a rebuilt server that presents another host key is not refused as a changed key either. The new key still has to pass --fingerprint or your yes, like any first contact.

A host key already in known_hosts under a name that is not configured — left by an older CLI, or by a machine you removed — is replaced, not compared: a name that is not a machine has nothing to protect.

What a failed run leaves is harmless to the next one: the key it made is reused, and a key it already installed now logs in, so no password is asked for.

setup writes config.yml earlier, once the key logs in, and running it again resumes from that file instead of starting over.

Nothing is written before the key logs in

Neither setup nor machines add writes a machine whose key has not logged in. A refused key with no password, a failed proof, an address that does not answer, a declined fingerprint: each leaves config.yml as it was, or absent if there was none, and records no agent key for the machine.

Why: a machine in config.yml is one every command reaches for — doctor, sessions, the macOS app every minute or two. With a key the server refuses, each of those is a failed login in the server’s SSH log. With fail2ban on, a few of them ban your address, and from then on the server refuses the connection outright, even after the key is fixed. See the connection is refused on a machine that answered before.

Once the key logs in, there is no such risk, so setup writes the machine before it changes anything else. A later failure — no way to root, Ansible not installing — keeps it, and running setup again finishes from there.

A fingerprint of any key type

A server holds several host keys, ED25519, ECDSA and RSA, and a handshake shows one. --fingerprint may name any of them: when it is not the one shown, the CLI opens one handshake per other type, in which the server proves it holds that key, and trusts the key whose fingerprint matches. From then on the CLI asks for that type only. A fingerprint no key of the server has is refused, so this accepts no key the operator did not name.

A local machine trusts its first host key

Everywhere else, an unattended run needs --fingerprint: trusting whatever answers, with nobody looking, is how a wrong server gets your key. machines create-local <name> --add is the one exception. It trusts the key the VM presents, because the same command created that VM a moment ago and the VM answers only on 127.0.0.1, through a port Lima forwards on your own computer — there is no network in between for anybody to stand on.

Locking down means writing one file that turns password login off, checking it is valid, and only reloading SSH if it passes — an invalid file is taken back: the one that was there before is put back, or the new one deleted when there was none, since a bad file left behind would break the next SSH reload, even a routine reboot.

setup asks first whether every account loses its password, or some keep it. See password login, account by account.

Why the file name starts with 00

SSH uses the first value it finds for each setting, so a drop-in file only wins if it sorts before others. A cloud image often ships its own settings as 60-cloudimg-settings.conf; a file named 99- would read after it and silently do nothing — no warning, password login still on despite a successful run.

Then it asks SSH, not the file

A valid file is not proof that SSH reads it. A server whose main sshd_config has no Include line for sshd_config.d, or sets PasswordAuthentication yes above that line, accepts the file and ignores it. So after the reload setup runs sshd -T, which prints the settings SSH is really running with, and requires passwordauthentication no. If SSH still allows passwords, the file is taken back, SSH is reloaded without it, and setup stops — it never says “password login is off” about a server that still takes passwords.

On a Mac

A Mac gets the same file in the same folder, with two differences. The check is /usr/sbin/sshd -t alone: macOS has no /run, and its / is read-only. And nothing is reloaded: macOS starts SSH fresh for each connection (through launchd), so the next connection reads the new file. The sshd -T proof is the same.

An admin login that is not root

Many servers arrive with root closed and an ordinary account that has sudo: a home server, a VM made by Lima or Multipass, some providers’ images. The CLI works the same on them, as long as that account’s sudo asks for no password.

Everything that changes the system — turning password login off, installing Ansible, unpacking a sync’s bundle under /opt/devmachine, running the play — first asks id -u. Root runs it directly; any other account runs it through sudo -n. A server where the admin is root never sees sudo at all, and neither does your own computer as a self machine.

Either way the step runs in bash when the server has it, and in sh only when it does not. Root’s login shell was bash before any of this existed, and the two differ where it matters: sh on Debian and Ubuntu is dash, which stops a whole command when it cannot read a file it was told to load. A package’s help loads its credential first, and before credentials push that file is not there yet; help must still answer.

It asks, rather than trying without sudo and trying again with it. A retry would run a half-finished step twice, and would read any failure at all — a full disk, a refused SSH setting — as a missing permission.

-n because nobody is there to type a password. So before anything is changed, setup, machines add and sync check that sudo -n true works, and stop with one sentence when it does not:

the admin login cannot become root: "alice" is not root, and `sudo -n true` fails as it.
Log in as root, or give it passwordless sudo on the machine:
echo 'alice ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/devmachine-alice

The whole play runs as root, not only the tasks that ask for it. A task that becomes a workspace account would otherwise go from one unprivileged account to another, which Ansible refuses without ACLs on its temporary files.

Tailscale SSH proves nothing about a key

A server with Tailscale SSH turned on answers port 22 on its tailnet address with Tailscale, not with sshd. Tailscale lets anyone on your tailnet in by tailnet identity and accepts any key, or none. “The key already logs in” would be true and mean nothing: the key was never checked, and may not be on the server at all.

So when the server identifies itself as Tailscale, setup and machines add install the key anyway and say why, instead of taking the login as proof — and leave password login as it was, since locking down comes after the proof and there was none. Without the key, the machine is unreachable the day Tailscale SSH is off, or from anywhere outside the tailnet. Running devmachine setup --machine <name> again repairs a machine an older CLI set up this way.

The password is never stored

It lives in memory for one connection, then is gone — never written to config.yml, the lock file, or the log. Typed at a terminal, it is never shown on screen.

Why this does not just run the ssh command

If your SSH agent holds several keys, a password login can fail before you see the prompt: the agent offers key after key, and the server cuts the connection after too many tries — on a server you are meeting for the first time, that happens every time.

setup talks to SSH directly in code, so it offers exactly one thing at a time — the key, or the password, never both. Sessions you use yourself, ssh and mosh, still run the real command, since there you want your own terminal and agent.

After this step

setup installs ansible — the last thing done by hand. From then on, every change goes through devmachine sync. It installs ansible rather than ansible-core, since the smaller package leaves out a piece the firewall package needs.

On Debian and Ubuntu that is apt-get install ansible. On Arch it is pacman -S --noconfirm --needed ansible, with the package lists the machine already has. It never runs pacman -Sy: refreshing the lists without upgrading is a partial upgrade, which Arch does not support and which can install an Ansible built for a Python the machine does not have. A full pacman -Syu upgrades everything, and that is your call, not a side effect of setup. On an Arch server whose lists are too old for the mirrors, setup stops and says so — see troubleshooting. A distribution based on one of them takes its base’s path: pacman on Manjaro, apt-get on Linux Mint.

On a Mac the CLI installs nothing itself: the mac-brew or mac-ports package’s bootstrap does, after you agree. See what a machine needs.