Many cybersecurity products rely on highly privileged "endpoint agents" deployed on every system in an organization. The damage possible if such an agent is compromised (such as through a supply-chain attack) is extraordinary. That such agents frequently perform unattended upgrades increases risk. On Linux systems, vendors commonly deploy these agents as systemd services that run with root privileges. Yet even if running as root, systemd offers extensive and granular configurations to sandbox services and limit what they can do. systemd systemd The Principle of Least Privilege requires vendors to use these controls to restrict their systemd services to only what is necessary. An analysis of the services installed by four commercial vendors, however, shows no use of sandboxing or security controls. And thus CrowdStrike (and others) can adjust my system clock, reboot my system, directly read physical RAM, and execute other highly privileged operations despite no operational need. systemd The four services examined are the CrowdStrike Falcon Sensor (endpoint detection and response), the Airlock Enforcement Agent (application allow listing), the ManageEngine UEMS Agent (vulnerability management), and the DriveStrike Client Service (remote device locking and wiping). Using the command line, this article will show you how to find, extract, and understand service definitions, as well as measure their exposure. We will then survey the powerful and practical systemd controls available to reduce service risk. We are left with the glaring contrast between the abundance of flexible controls and the lack of any effort by commercial vendors to use them. systemd Services on Linux systemd is a system and service manager for Linux: it is the first process initiated after the kernel and is responsible for managing core system functions like logging and networking as well as security-critical applications like SSH daemons, web services, and third-party endpoint security agents. While there were many predecessors, `systemd` has been adopted by all major Linux distributions. systemd While systemd supports numerous types of "units," only "service" units will be considered here. Users can create and (with root privileges) install their own services by defining a ".service" file, an INI-style configuration file with required sections and other systemd-specific features (such as supporting multiple values for the same key). It is common to install the relevant binaries and systemd ".service" files through a package (e.g., a ".deb" file) or a shell-script installer. systemd systemd systemd Service units require three INI sections: [Unit]: Describes the service and specifies ordering relationships (e.g. After=network.target) as well as dependencies (e.g. Requires=postgresql.service)[Service]: Defines what command to run (ExecStart=/path) as well as optional sandboxing, security controls, and resource limits.[Install]: Defines hierarchical activation triggers (e.g. WantedBy=multi-user.target) that are implemented with symbolic links created when systemctl enable is called. [Unit]: Describes the service and specifies ordering relationships (e.g. After=network.target) as well as dependencies (e.g. Requires=postgresql.service) [Unit] After=network.target Requires=postgresql.service [Service]: Defines what command to run (ExecStart=/path) as well as optional sandboxing, security controls, and resource limits. [Service] ExecStart=/path [Install]: Defines hierarchical activation triggers (e.g. WantedBy=multi-user.target) that are implemented with symbolic links created when systemctl enable is called. [Install] WantedBy=multi-user.target systemctl enable For this analysis, the [Service] section is most relevant. Reference documentation for service configuration can be found at the command line with man systemd.service. [Service] man systemd.service Examining Service Definitions The four commercial services examined are listed below (with service name and version number): CrowdStrike Falcon Sensor (falcon-sensor.service, 7.31.18410.0)Airlock Enforcement Agent (airlock-client.service, 6.1.0.6070)ManageEngine UEMS Agent (dcservice.service, 11.4.2540.03)DriveStrike Client Service (drivestrike.service, 2.1.22.31) CrowdStrike Falcon Sensor (falcon-sensor.service, 7.31.18410.0) Airlock Enforcement Agent (airlock-client.service, 6.1.0.6070) ManageEngine UEMS Agent (dcservice.service, 11.4.2540.03) DriveStrike Client Service (drivestrike.service, 2.1.22.31) To see all services running on a system enter systemctl list-units --type=service. After finding the name of a service (e.g. "falcon-sensor.service"), the complete definition can be viewed with the systemctl cat command (e.g. systemctl cat falcon-sensor.service). systemctl list-units --type=service systemctl cat systemctl cat falcon-sensor.service Below are the [Service] sections from each configuration: [Service] falcon-sensor.service [Service]
Type=forking
ExecStartPre=/opt/CrowdStrike/falconctl -g --cid
ExecStart=/opt/CrowdStrike/falcond
PIDFile=/var/run/falcond.pid
Restart=no
TimeoutStopSec=60s
KillMode=control-group
KillSignal=SIGTERM [Service]
Type=forking
ExecStartPre=/opt/CrowdStrike/falconctl -g --cid
ExecStart=/opt/CrowdStrike/falcond
PIDFile=/var/run/falcond.pid
Restart=no
TimeoutStopSec=60s
KillMode=control-group
KillSignal=SIGTERM airlock-client.service [Service]
ExecStart=/usr/bin/airlock
LimitNOFILE=65535 [Service]
ExecStart=/usr/bin/airlock
LimitNOFILE=65535 dcservice.service [Service]
ExecStart=/usr/local/manageengine/uems_agent/bin/dcservice -t &
ExecStop=/usr/local/manageengine/uems_agent/bin/dcservice -p
KillMode=process [Service]
ExecStart=/usr/local/manageengine/uems_agent/bin/dcservice -t &
ExecStop=/usr/local/manageengine/uems_agent/bin/dcservice -p
KillMode=process drivestrike.service [Service]
Type=simple
ExecStart=/usr/bin/drivestrike run
Restart=always
RestartSec=10
SyslogIdentifier=drivestrike [Service]
Type=simple
ExecStart=/usr/bin/drivestrike run
Restart=always
RestartSec=10
SyslogIdentifier=drivestrike Measuring Exposure The systemd-analyze security command can be used to analyze the "exposure" of services, delivering scores between 0 and 10, where higher scores have more exposure. Scores above 9 are labelled "UNSAFE". systemd-analyze security Likely to maintain backwards compatibility with legacy services, the default systemd service configuration is extremely permissive. Given the nearly complete reliance on defaults in the above examples (made explicit by their brevity), all receive "UNSAFE" scores, showing no attempt to apply the principle of least privilege. The four analyzed services receive the following scores: systemd CrowdStrike Falcon Sensor (falcon-sensor.service): 9.5Airlock Enforcement Agent (airlock-client.service): 9.6ManageEngine UEMS Agent (dcservice.service): 9.6DriveStrike Client Service (drivestrike.service): 9.6 CrowdStrike Falcon Sensor (falcon-sensor.service): 9.5 Airlock Enforcement Agent (airlock-client.service): 9.6 ManageEngine UEMS Agent (dcservice.service): 9.6 DriveStrike Client Service (drivestrike.service): 9.6 The reports produced by systemd-analyze security itemize the exposure of over 80 characteristics. The reports for airlock-client.service, dcservice.service, and drivestrike.service are all identical, resulting in the same score of 9.6. This is essentially a baseline score for a service that applies no restrictions. systemd-analyze security While the falcon-sensor.service score is better by a mere 0.1, it is not due to applying controls. Unique among these services, the [Unit] section of this service opts into DefaultDependencies=no, permitting the service to run in a "special boot phase" (explicitly After=local-fs.target). While this boot phase initialization slightly improves the systemd-analyze security exposure level (by invalidating a few checks), the deeper system integration makes the service more dangerous. [Unit] DefaultDependencies=no After=local-fs.target systemd-analyze security While endpoint security agents legitimately require elevated privileges, providing, as shown here, literally no restrictions is indefensible, particularly when systemd offers extensive, well-documented mechanisms to limit access while preserving functionality. systemd Enabling Service Sandboxing and Security With over 60 controls, the configuration of systemd sandboxing and security can seem complex. The small collection of powerful configurations shown below offer substantial protection with minimal complexity, a fact that makes the lack of controls applied by commercial cybersecurity vendors so egregious. systemd Full documentation of these controls is available via man systemd.exec and man systemd.resource-control. man systemd.exec man systemd.resource-control Privilege & Capabilities Management By default, systemd runs services as root. However, systemd permits defining a non-root user (with User=) created on the fly (with DynamicUser=yes). Root-like capabilities can then be explicitly and narrowly opted-in for this user. Combined with the NoNewPrivileges=yes configuration, a process can be prevented from performing privilege escalation. systemd systemd User= DynamicUser=yes NoNewPrivileges=yes When running as non-root, specific Linux capabilities can be granted via AmbientCapabilities=, such as CAP_NET_BIND_SERVICE to bind ports. Conversely, CapabilityBoundingSet= restricts which capabilities a service can ever acquire, whether running as root or non-root (e.g., using the ~ to invert selections such as CapabilityBoundingSet=~CAP_SYS_TIME). AmbientCapabilities= CapabilityBoundingSet= ~ CapabilityBoundingSet=~CAP_SYS_TIME While some services may legitimately require root privileges, running as a dedicated non-root user eliminates entire classes of vulnerabilities. File System Isolation With file system controls, services can make portions of the file system inaccessible or read-only. For example, with ProtectSystem=strict, nearly the entire file system is read-only. Services requiring write access must explicitly define writable paths with ReadWritePaths=. Similarly, the ProtectHome=yes configuration makes /home and related directories inaccessible. Read-only access is possible with ProtectHome=read-only. ProtectSystem=strict ReadWritePaths= ProtectHome=yes /home ProtectHome=read-only The PrivateTmp=yes gives a service its own private /tmp directory, a desirable configuration for any process that might write to /tmp. Additionally, UMask=0077 ensures any files written are readable only by the service owner. PrivateTmp=yes /tmp /tmp UMask=0077 All services benefit from limiting write access to only where needed. Kernel & System Protection While some cybersecurity services might require kernel extensions, services that do not should forbid loading or unloading kernel modules with ProtectKernelModules=yes. The ProtectKernelTunables=yes configuration can be used to prohibit editing kernel configuration files. ProtectKernelModules=yes ProtectKernelTunables=yes By default, root services can change the system and hardware clock. Using ProtectClock=yes, such access can be restricted. While seemingly trivial, system clock manipulation enables serious attacks: a compromised service could avoid expiration of one-time passwords or session tokens, accept revoked TLS certificates, tamper with log timestamps, or bypass scheduled maintenance operations. It is inconceivable that any third-party service (even a cybersecurity endpoint agent) would have a legitimate reason to change the clock. ProtectClock=yes Device & Hardware Access With PrivateDevices=yes, the default access to all storage, media, and input devices can be removed. This prohibits direct access to, among other things, raw disk partitions and physical RAM, access that can completely bypass file permissions and process memory segmentation. Fine-grained access, if needed, can be enabled with DeviceAllow=. It is extremely unlikely that a cybersecurity endpoint (or any service) would need such low-level hardware access. PrivateDevices=yes DeviceAllow= System Call Filtering System call filtering can restrict what kernel-level system calls a service can execute. For example, SystemCallFilter=read write open close defines a strict allow list of only four system calls. System call groups (such as "@clock" or "@reboot") provide more general ways to enable or disable (with ~) system calls. The configuration options are extensive and can be viewed with systemd-analyze syscall-filter. SystemCallFilter=read write open close ~ systemd-analyze syscall-filter While some of the functionality potentially constrained by CapabilityBoundingSet overlaps with SystemCallFilter, this configuration offers lower-level control. Thus, while setting ProtectClock=yes and SystemCallFilter=~@clock may seem redundant, they operate at different system levels, providing defense-in-depth. CapabilityBoundingSet SystemCallFilter ProtectClock=yes SystemCallFilter=~@clock While specific groups can be allowed or denied, the SystemCallFilter=@system-service configuration provides sensible defaults for most services. An explicit deny list suitable for many services would look like this: SystemCallFilter=@system-service SystemCallFilter=~@clock
SystemCallFilter=~@chown
SystemCallFilter=~@cpu-emulation
SystemCallFilter=~@debug
SystemCallFilter=~@module
SystemCallFilter=~@mount
SystemCallFilter=~@obsolete
SystemCallFilter=~@privileged
SystemCallFilter=~@raw-io
SystemCallFilter=~@reboot
SystemCallFilter=~@resources
SystemCallFilter=~@setuid
SystemCallFilter=~@swap SystemCallFilter=~@clock
SystemCallFilter=~@chown
SystemCallFilter=~@cpu-emulation
SystemCallFilter=~@debug
SystemCallFilter=~@module
SystemCallFilter=~@mount
SystemCallFilter=~@obsolete
SystemCallFilter=~@privileged
SystemCallFilter=~@raw-io
SystemCallFilter=~@reboot
SystemCallFilter=~@resources
SystemCallFilter=~@setuid
SystemCallFilter=~@swap Resource Limits & Denial of Service Prevention Independent of capabilities or system call access, a process can consume memory, CPU, and tasks (threads and processes). systemd offers powerful control of these resources. systemd With MemoryMax, the maximum memory a service can use can be set. Flexible values, such as a percentage of all memory (MemoryMax=50%) are practical. Similarly, the CPUQuota configuration can be used to restrict CPU-intensive tasks. For example, with CPUQuota=50%, a single process can only consume half of one core. MemoryMax MemoryMax=50% CPUQuota CPUQuota=50% Finally, by setting TasksMax, the total number of threads and processes can also be capped. On my system the default of over 76 thousand (i.e. systemctl show -p DefaultTasksMax) is far more generous than any service could possibly need. TasksMax systemctl show -p DefaultTasksMax These controls can prevent a compromised service from consuming excessive resources in a denial-of-service attack. Again, it is inconceivable that even a cybersecurity endpoint agent would need unbound RAM, CPU, or tasks. Conclusion Given the default power of systemd services and the ease with which they can be constrained, it remains indefensible that, among the four commercial services examined here, none have applied even the most elementary controls. systemd This analysis was possible due to systemd's transparency. While Windows and MacOS lack equivalent introspection tools, whether similar over-provisioning occurs should be investigated. systemd Even if we trusted an endpoint agent on installation, the use of unattended updates means that vendor infrastructure compromise (e.g., SolarWinds in 2019) could silently convert millions of systems into malicious rootkits. An adversary is not even required: in 2024, CrowdStrike's own flawed unattended update disabled 8.5 million Windows systems. Properly scoped services can limit damage in both scenarios. Cybersecurity vendors do not deserve the trust required to run fully unconstrained root services performing unattended updates. Fortunately, on Linux, we do not need trust: with modest effort, systemd's controls can narrowly scope services to the appropriate least privilege. What will it take for vendors to implement these basic protections? systemd

This story contains new, firsthand information uncovered by the writer.

CrowdStrike Can Manipulate Your Clock, And Other Ways Cybersecurity Vendors Ignore Least Privilege

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Microsoft's Forced Updates: A Critique of the Global IT Outage

How Could an Update Go So Wrong?

Billions of Computers Not Working | Unraveling the Issue Behind the Global CrowdStrike Chaos

The Microsoft and CrowdStrike Catastrophe: A Run-Down

CrowdStruck: How the CrowdStrike Outage Taught Me the Value of Listening

Microsoft Rips Into Delta Airlines For Sluggish Recovery After Global CrowdStrike Outage

Microsoft's Forced Updates: A Critique of the Global IT Outage

How Could an Update Go So Wrong?

Billions of Computers Not Working | Unraveling the Issue Behind the Global CrowdStrike Chaos

The Microsoft and CrowdStrike Catastrophe: A Run-Down

CrowdStruck: How the CrowdStrike Outage Taught Me the Value of Listening

Microsoft Rips Into Delta Airlines For Sluggish Recovery After Global CrowdStrike Outage

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps