Taking a Linux desktop, and the eleven things in the way

Remote control has worked on Windows since we shipped it. From agent 0.3.94 it works on Linux too: open a Ubuntu device, press Control, and you have its screen and its mouse and keyboard, through the same browser panel and the same scoped, per-session credentials.

That sentence took a day and eleven distinct obstacles, most of which were only visible from inside a real session on a real machine. Since the interesting part of any remote-access feature is what it cannot do and why, here is the whole thing.

Why Linux is not Windows with different paths

On Windows the agent captures the desktop and injects input from one helper process, and the operating system cooperates throughout. Linux offers neither half for free.

There is no XTEST on Wayland. The X11 mechanism every remote-access tool used for twenty years does not reach a Wayland client. The modern replacement asks the compositor for permission, through a desktop portal that raises a dialog. A dialog is fine for screen sharing in a meeting. It is useless for supporting a machine whose user is not sitting at it, which is the entire point.

And screen capture is behind the same door. The portal's screencast interface prompts as well.

So the route had to avoid both. For capture we use org.gnome.Mutter.ScreenCast, the compositor's own interface, which is what GNOME's own remote desktop uses and which raises no dialog: measured, it returns a session in about a millisecond. For input we go underneath the compositor entirely, to the kernel's /dev/uinput, creating a virtual pointer and keyboard. Events entering at that layer are indistinguishable from a real mouse, so no compositor can refuse them, and they work identically on X11, Wayland, the login greeter and the lock screen.

Consuming the video without adding a C library to the agent was the third constraint. GStreamer already ships an element that reads the compositor's stream, and it is present on any desktop that uses PipeWire for audio, which since Ubuntu 22.10 is all of them. The agent runs it as a subprocess and parses frames off a pipe, and stays a single static binary.

The split, and an accidental security improvement

Two facts turn out to be incompatible. Capture must run as the logged-in user, because the compositor's interface lives on that user's session bus and a root process asking for it is refused before it manages a method call. Input must run as root, because /dev/uinput is root-only.

So the work splits across two processes. And once it is split, the interesting question is which half holds the credentials, because on Windows the single helper holds them all: it publishes frames itself, so it is handed the device's messaging credentials in a file.

The Linux capture helper needs none of that. It captures and writes frames down a pipe to the agent, which does the publishing. There is no credential file on disk for a desktop user to read, and the process running inside someone's login session holds nothing at all. That is a better arrangement than the one we already had, and we only arrived at it because the platform forced the split.

The eleven hoops

1. It recorded the wrong screen. The development machine has three monitors. The code took the first one the compositor listed, which was a side screen nobody was using. A static picture out of a working pipeline looks exactly like a broken pipeline, and it cost an hour. Displays are now read from the compositor's logical monitor list, which is the one carrying the primary flag and the desktop coordinates.

2. Every early performance number was of a screensaver. The screen had blanked during testing. While it is blanked, the compositor refuses to start a recording at all, with an error that says "session creation inhibited" and means "the screen is off". Frame rates measured in that state read 0.1 to 0.6 per second and looked like a catastrophic performance problem. They were a measurement of a blank screen.

3. The fix for that silently did nothing. The agent nudges the pointer to wake the screen before it starts. It did this by sending relative movement on the pointer device, which is configured for absolute positioning only, and the kernel silently discards events a device has not declared it can send. It looked like it worked and changed nothing.

4. The helper could not read its own instructions. It is handed its parameters on standard input, and it opened /dev/stdin by name. By that point it has dropped from root to the desktop user, and the pipe still belongs to root, so opening it by path is refused. The file descriptor it already holds is perfectly readable.

5. One frame, then silence. The video arrives as multipart data with a length on each part, and each part is followed by a blank line. Reading that blank line as "end of headers" produced exactly one good frame and then a parse failure. It only showed up against a real capture, because the fixture written from imagination did not have the trailing blank line in it.

6. The permission check landed on the wrong class. The rule for whether a device can be controlled was inserted next to the first matching anchor in the models file, which belonged to the battery model rather than the device model. Every search for the code found it. The server insisted the property did not exist.

7. And it read the wrong shape. The device's connectivity record is a list rather than a single object, for reasons buried in how the relationship is declared. Reading it as an object returned nothing for every device on the fleet, which failed the version check everywhere.

8. And the page never saw it. The device page is handed a plain dictionary rather than the model, so asking it for a model property is a key lookup that misses and renders as false. The rule was right, the server agreed, and the button still did not exist.

9. Then the button existed and did nothing. Remote control is gated in three separate places in the device page: the button, the panel, and the script that makes the button work. Fixing the first and leaving the other two produced a Control button that threw an error when clicked. Half a gate is worse than none, because it looks available.

10. The deadlock. This is the one worth reading twice.

The compositor only produces a frame when something on screen changes. On a desktop nobody is touching, that is nothing at all: measured on the same machine in the same session, a hundred frames while the pointer moved and zero over an equivalent window with the desktop still.

The browser reads silence as failure. Six seconds without a message and it marks the stream stalled. And a stalled stream is one it deliberately refuses to send input on, because clicking blind on a frozen picture is how you close the wrong window on someone's production server.

Put those together and you get a trap. Connect to an idle machine. Receive one frame. Go quiet. Get marked stalled. Now you cannot click, so nothing changes, so no frame is produced, so it stays stalled. You can watch the machine and never touch it.

The fix is a heartbeat: a message with no pixels every two seconds whenever nothing real has been sent. Windows never hit this because it always had one.

11. The quality selector was not connected. The browser publishes the operator's quality, frame rate and display choices on a control channel that the Linux agent was not listening to. So the Eco setting did nothing, and there was no way down from the default. On a fast network that is an annoyance. On a satellite link on a ferry, which is where this was found, it is the difference between a working session and a useless one.

What it costs

Measured against a real desktop, at the defaults:

	frame rate	per frame	bandwidth
uncapped	40 fps	136 KiB	5.5 MB/s
default 1280px, quality 50, 8 fps	8 fps	116 KiB	7.2 Mbit/s
Eco 960px, quality 35, 4 fps	4 fps	~60 KiB	~1.5 Mbit/s

The important row is the one not in the table: an idle screen costs almost nothing. Because the compositor only sends frames when the picture changes, you pay for motion and nothing else. A session left open on a quiet machine is a heartbeat every two seconds.

Forty frames a second is far more headroom than remote support needs, which is why the default is eight. If you are on a poor connection, Eco at four frames per second is about a fifth of the bandwidth and entirely usable for the things remote control is actually for.

What it will not do

It is GNOME only, for now. The interface we use to avoid the consent dialog is GNOME's. KDE has its own, and other desktops need the portal, with its prompt. Ubuntu is the target and the rest can follow the same seam.

Capture stops at the lock screen. While the session is locked or blanked, the compositor refuses to record it. The agent wakes a blanked screen automatically and retries, so that case is handled, but a genuinely locked session cannot be viewed until it is unlocked. Input still works, so an operator who knows the password can type it, and the picture returns.

There is no Ctrl+Alt+Del equivalent, because Linux has no secure attention sequence for one to be.

Headless servers are not the use case. A Linux server with no desktop has nothing to capture, and it already has the better tool: the browser shell, which has worked on every platform since it shipped.

Getting it

Agent 0.3.94, through the usual update channel. The Control verb appears on a Linux device once its agent reports that version or newer; older agents do not show it, because a button that opens a session which then fails is worse than no button.

Windows and macOS are unchanged by this release.