pmx-cork-front

The port 443 front of the suite. The runtime's proxy has no TCP listener, so this is the only way in: it makes the stock PVE web interface and the pmx-cork runtime appear as one origin. Open the UI on https://<node>/; port 8006 still serves plain PVE and does not load the pmx-cork UI.

One package, one file: the Caddyfile. No unit, no drop-in and no service account, because the web server, its service and its account are all the caddy package's, unmodified.

The web server is Debian's caddy, installed from the archive as a dependency and run from a configuration and a unit of this package's own. Nothing is built here and no version is pinned, so the process that terminates TLS for the whole node is patched by apt like everything else on it.

It also runs as that package's account rather than one of ours. The caddy postinst does usermod -aG www-data caddy, and www-data is the one membership the front needs: /etc/pve/local/pve-ssl.key is 0640 root:www-data and /run/pmx-cork-proxy is 0750 www-data:www-data, so both are reached through the group and neither through a copy. The install removes any stale pmx-cork service account it finds, and says so when a process still holds one.

The configuration is /etc/caddy/Caddyfile, which is the path that package's unit already reads, so there is nothing to point anywhere. The admin socket sits in /var/lib/caddy, which exists and belongs to that account, and caddy reload finds it there out of the configuration it is loading, so the unit's own ExecReload works unchanged. The access log's directory is made by the install, because Caddy creates the file and not the folder.

Borrowing the unit is also what makes a caddy security update reach the running process. That package's postinst ends with deb-systemd-invoke try-restart caddy.service, so the front comes back on the patched binary rather than holding the replaced one in memory. Debian backported eight CVEs into this caddy in August alone.

The front is therefore caddy.service: systemctl status caddy, journalctl -u caddy, systemctl reload caddy, and an admin restarting caddy restarts the PVE interface. Nothing in the unit is changed, so systemd-analyze security says 8.8 EXPOSED, which is Debian's number for its own unit.

Two things follow from owning that path. This package cannot ship a file there, because it is the caddy package's conffile and dpkg refuses to unpack one package's file over another's; the default is shipped under /usr/share/pmx-cork-front/ and the postinst puts it in place, keeps caddy's own as Caddyfile.caddy-orig, and never overwrites an edited one, putting the new default beside it as Caddyfile.dist instead. And /var/log/pmx-cork is made with mode 0750 by that same postinst rather than by a LogsDirectory, which matters because PVE's console URLs carry a vncticket in the query string and Caddy logs the query string; pveproxy keeps its own access log closed the same way.

What it does

• Terminates TLS on 443 with a PVE certificate the Caddyfile names, shipped pointing at the cluster pair. Nothing is issued here, so nothing expires unnoticed. Point it at your own pair if you serve one, and at the same one pveproxy serves, for the reason under The certificate below.
• Sends /pmx-cork/* and /api2/extjs/pmx-cork/* to the runtime's Unix socket, and everything else to pveproxy on 127.0.0.1:8006, console WebSockets included.
• Serves / as PVE's own page with one <script> after it: the loader for /pmx-cork/web/pmx-cork-ui.js. No PVE file is modified, so a PVE upgrade cannot undo it and cannot be undone by it.
• Answers every client that reaches it. PVE ships 8006 open and expects a firewall to narrow it, and a front that dropped packets on 443 would make a blocked cross-node peer look like a dead node rather than a refused one.

The hop to pveproxy does not verify the certificate, deliberately and only there, and not because it could not be. On a stock node it would: pve-ssl.pem carries 127.0.0.1 in its SANs and chains to /etc/pve/pve-root-ca.pem, and verifying against that pool returns Verification: OK.

It is the node with its own certificate that has no setting that works, which is the node whose operator has done the most careful work. Measured on pve01, which serves Let's Encrypt: against the cluster CA the hop fails with unable to get local issuer certificate, and against the system store with IP address mismatch, because that certificate's only SAN is the FQDN. Neither pool covers both nodes, tls_trusted_ca_certs replaces the system pool rather than adding to it, and dialling the FQDN instead of 127.0.0.1 would take the hop off the loopback for a name check.

So what identifies the peer on that hop is that it is bound to 127.0.0.1:8006 on this machine and the connection never reaches a network.

The index page

/ is the one path pveproxy renders a document for, and it renders it per request: the username comes from the ticket cookie, the CSRF token from the cluster authkey (which rotates daily), and ?console=, ?mobile= and the language and theme cookies select a different template from a different directory. A cached copy is therefore not an option.

Rendering /usr/share/pve-manager/index.html.tpl here instead was considered and rejected. Not because the template language is hard: across the four templates it is four IF/ELSE pairs, a couple of or defaults and one date.now. The reason is what surrounds it. Reproducing the page means reproducing PVE::Service::pveproxy::get_index: fifteen variables, the four-way choice between the templates, is_phone() on the User-Agent, wtversion parsed out of the first line of proxmoxlib.js. None of that is an API and none of it is versioned, so a variable added in a point release renders as empty and nothing says so. It would also mean minting PVE's own CSRF token for PVE's own interface, which pveproxy does today and this front never touches.

So the page is asked for, per request, and the loader follows it.

Debian's caddy cannot edit a response body, so the page is composed rather than rewritten. The index route answers with one line of its own: it replaces the single </head> in the page httpInclude fetches through /__pmx-cork-index, an internal second path that exists for the include and answers nothing else, with the loader and that tag. respond answers text/plain, so the route sets the type. The query string is carried across, because ?console=, ?mobile= and ?debug= select a different template on the far side.

The anchor is </head> because each of PVE's four templates has exactly one, checked on the shipped files and on all three rendered variants. Appending the element to the included page instead is a line shorter and produces markup that is not valid: it lands after </html>, where every parser moves it back into the body and no validator accepts it.

Measured against pveproxy on a node: the composed page differs from the one PVE serves on 8006 by exactly one line, the loader. Sending a theme cookie and a language cookie through the front returns the same stylesheet and the same locale file pveproxy picks directly, which is how the ticket cookie behind the username and the CSRF token gets there too.

Composing rather than rewriting costs two things, and they are the price of staying on the archive's caddy:

• The template engine executes the included body. Demonstrated on a node: an upstream answering <<%env "HOME"%>> came back as the value of HOME. The included body is PVE's page, into which datacenter.cfg's consent-text arrives unescaped. The delimiters are therefore <<% %>> and not the Go default, which is mitigation and not a fix. Setting a consent-text needs Sys.Modify on /, so this is an administrator reaching the front's process, not an unprivileged user.
• The page is held in memory for the length of one replacement. replace works on a string, so the include cannot stream. PVE's index is 2.7 kB, and it is the only route that does this; everything else is proxied straight through.
• Status and headers are not passed through. The route builds a response from a file rather than forwarding pveproxy's, which is why it sets Cache-Control and Pragma itself.

There is no guard in the configuration and no JavaScript in it at all. PVE serves its xterm.js console from this same path (/?console=shell&xtermjs=1) and that page carries no ExtJS, so the core checks for Ext itself and stands down. The thing that needs ExtJS is the thing that checks for it.

The package is one file. The Caddyfile carries a line or two per decision and nothing more, and the rest of the reasoning is here. The weight is on make lint, which fails if the templates handler, the delimiters, the respond body, the include, the query string, the </head> anchor, defer, the second path and its rewrite, the UI-core path or the absence of an address filter stop being right. Each of the fifteen rules was sabotaged and reports exactly one thing.

The one comment that carries weight is on the delimiters. Demonstrated on a node with a real consent-text of A {{env "HOME"}} B: on <<% %>> the login page shows it as written, on the Go default it shows A /var/lib/caddy B, the front process's own environment, before anyone has logged in.

The certificate

The Caddyfile names it, and nothing else touches it. Two paths:

tls /etc/pve/local/pve-ssl.pem /etc/pve/local/pve-ssl.key

That is the cluster certificate, which every PVE node has, so a fresh install serves something valid without being asked anything.

If you serve your own certificate, point those two paths at it. PVE keeps the operator's pair at /etc/pve/local/pveproxy-ssl.pem with the key beside it, and pveproxy prefers that pair over the cluster's whenever it exists, so on such a node the line becomes:

tls /etc/pve/local/pveproxy-ssl.pem /etc/pve/local/pveproxy-ssl.key

Then systemctl reload caddy. The same applies after a renewal: Caddy reads these files when it loads the configuration and not again, so an ACME renewal that rewrites the file in place is not picked up until you reload.

Nothing here chooses the certificate for you, watches it or reloads on a change. That choice depends on what certificate you run and why, and a front that quietly follows PVE is a front that quietly changes what it serves.

The two still have to agree. A call from another node pins the SHA-256 of the leaf that this node's /etc/pve/nodes/<node>/ says it serves, reading pveproxy-ssl.pem before pve-ssl.pem, exactly as PVE::CertCache does. A front on the other file is a node that is up, answering, and refused as a mismatch, and the browser meets a different certificate on 443 than on 8006. So the install looks once: if the node has a pveproxy-ssl.pem and the Caddyfile does not mention it, it says so and names the line to change. It does not change it.

A reload ends open connections. Every Caddy reload that takes effect closes live WebSockets, measured four ways: reload --force, a reload after a real config change, and a PATCH on apps/tls alone; a reload that would keep them is skipped by Caddy and changes nothing, and grace_period makes no difference. So changing or renewing the certificate costs every console open on the node a reconnect.

PVE does the same change without it. pveproxy's ExecReload is pveproxy restart, which PVE::Daemon implements as a SIGHUP: the listening socket survives an exec of itself, leave_children_open_on_reload leaves the old workers running, and wait_end_loop stops them accepting and then waits without a timeout until their last connection closes. Every path PVE takes when a certificate changes reaches for that, with systemctl reload-or-restart pveproxy.

nginx and Apache both behave like PVE here and were measured doing so. pocs/front-nginx.conf is the nginx version of this front with the numbers: what it buys, and the 21 extra configuration lines plus a logrotate entry it costs. It is upstream request caddyserver/caddy#6933, so the arithmetic may change.

The caddy package's own service

Installing this package pulls in caddy, and that package's postinst enables and starts caddy.service on a first install. Its shipped configuration listens on :80 and serves a placeholder directory, so it does not collide with this front on :443, but a Proxmox node has no use for it and this package is the reason it appeared.

So the postinst stops and disables it, and only when dpkg --verify caddy reports the configuration unchanged. An operator who has edited /etc/caddy/Caddyfile is running caddy for something, and guessing whose service matters is worse than leaving it alone.

Install

make front-install     # builds the package and installs it
make front-uninstall

Nothing is compiled and nothing is downloaded beyond the caddy package apt fetches.

What is where

In the repository:

Path	What it is
Caddyfile	the shipped default configuration
debian/	the packaging, and the postinst that puts the Caddyfile in place

On the node:

Path	Purpose
/usr/bin/caddy	the web server, from the caddy package
/etc/caddy/Caddyfile	listener, the certificate, both upstreams, the composed index. The caddy package's path, written by this package's postinst
/etc/caddy/Caddyfile.caddy-orig	what caddy shipped there, put back when this package is removed
/etc/pve/local/pve-ssl.pem, .key	the certificate served on 443, as shipped; the Caddyfile names it and nothing follows it
/var/log/pmx-cork/front-access.log	the request log, rolled by Caddy

The Caddyfile is a dpkg conffile, so an edited one survives an upgrade: dpkg keeps yours and puts the new default beside it as Caddyfile.dpkg-dist. make front-install passes --force-confold and then says so, because a non-interactive run has nobody to ask. A routing or TLS change is an edit followed by systemctl reload caddy. Change the delimiters and change the respond body with them; make lint checks that the two agree.

caddy fmt is safe on this file and leaves it unchanged. It was not on the previous one: the formatter cannot round-trip a backtick token containing braces, and the replacement string was markup. That token went away with the module that needed it.

The access log is json rather than console, because console writes ANSI colour into a file. Caddy empties Cookie and Authorization on its own; PVE's CSRFPreventionToken has no such rule, so the log block drops it by name. The query string is logged, and PVE's console URLs carry a vncticket in it; pveproxy's own access log records the same, so this is parity with PVE rather than something the front adds.