Unexpected Writable Behavior in Systemd's mstack
Systemd's new mstack tool, introduced in version 260 and enhanced in 261, allows for describing overlayfs mounts as simple directories. This feature aims to simplify the management of layered file systems, typically used in containerized environments or for managing system configurations. Instead of complex mount -t overlay -o lowerdir=... commands, mstack utilizes a directory structure with symlinks like layer@0, layer@1, and a rw/ subdirectory for the writable layer. The intention is to provide a more user-friendly interface for managing these layered mounts.
However, initial testing reveals a critical and potentially dangerous default behavior: mstack mounts the single writable layer with default permissions that allow overwriting even read-only files. In a demonstration, a two-file directory was mounted using mstack. A single line of text was written into this mounted directory, and it successfully overwrote a file that had been explicitly marked as read-only. This operation completed without any error messages or warnings, exiting with a status code of 0, indicating success to the user.
How mstack Works
The mstack functionality is built upon the Linux kernel's overlayfs module. Overlayfs allows multiple directories (layers) to be stacked, with one designated as writable. Changes made through the writable layer are typically isolated from the read-only lower layers. The mstack tool abstracts this complexity. Users create a directory, for example, my_fs.mstack, containing subdirectories like layer@0, layer@1, etc., to represent read-only layers. A special subdirectory named rw/ is designated as the writable layer.
When mstack is invoked, it configures the overlayfs mount point. The read-only layers are specified via the lowerdir option, and the rw/ directory becomes the upperdir. The workdir is also managed internally. The tool simplifies the command-line interface, making it easier to set up such layered file systems, particularly within systemd's ecosystem, including tools like systemd-nspawn.
The Default Writable Behavior and Its Implications
The core issue lies in the default mount options applied to the writable layer when mstack creates the overlayfs. The test case demonstrated that a file within a read-only layer could be modified or deleted directly through the mstack mount point. This bypasses the expected behavior of overlayfs, where writes to files that exist in read-only layers should create a new file in the upper (writable) layer or modify a copy of the original file. The observed behavior suggests that the writable layer itself might be incorrectly configured, or the overlayfs options are being set in a way that permits direct modification of lower-layer content.
This behavior is counterintuitive to the principle of layered file systems, which often rely on immutability of lower layers for security and predictability. For instance, in immutable infrastructure scenarios or when using read-only root file systems, this default behavior poses a significant risk. A user or process with sufficient permissions to trigger an mstack mount could inadvertently or maliciously alter critical system files that were intended to be protected.
The absence of errors or warnings during the overwrite operation is particularly concerning. Standard file system operations that attempt to modify read-only files typically result in a permission denied error (errno 13). The fact that mstack allows this to proceed silently with an exit code of 0 could lead users to believe their operations were successful, while in reality, they may have corrupted their system or data in ways that are difficult to diagnose.
Use Cases and Potential Risks
mstack is designed to be useful in various contexts. For example, it can simplify the setup of chroots or containers managed by systemd-nspawn. It could also be used for managing configuration overlays, where a base system image is overlaid with custom configurations. The ability to easily define multiple read-only layers and a single writable layer is powerful for scenarios requiring dynamic updates or temporary modifications without altering the base system.
However, the default behavior introduces significant risks. If mstack is used in an environment where read-only base images are critical, such as for security or reproducibility, this default write-through behavior could undermine those guarantees. Developers and system administrators must be acutely aware of this behavior. It implies that any file present in a read-only layer could potentially be modified if it is also present in the rw/ directory or if the overlayfs is configured to allow such operations by default.
What remains unaddressed is whether this behavior is an intended feature, a misconfiguration in the default overlayfs options used by mstack, or a bug that will be patched in future releases. Without explicit documentation clarifying this behavior, users are left to assume the worst-case scenario: that read-only layers are not truly protected by default when mounted with mstack.
Mitigation and Future Considerations
Until this behavior is clarified or patched, users of mstack should proceed with extreme caution. It is advisable to avoid mounting sensitive or critical read-only file systems with mstack until its default behavior is understood and confirmed to be secure. If mstack is necessary, users should rigorously test their specific use cases and ensure that no unintended modifications occur. Explicitly setting overlayfs options that enforce read-only integrity for lower layers, if possible through mstack's configuration, would be a prudent step.
Systemd developers should prioritize clarifying this behavior. If it is a bug, a fix is urgently needed to prevent potential data corruption or security vulnerabilities. If it is an intended feature, it requires extensive documentation and clear warnings to users about the implications for read-only layers. The current state, where a critical function operates with silent, destructive side effects, is untenable for a system component as foundational as systemd.