Systemd 服务启动故障排查实战指南

一、背景介绍

Systemd 是现代 Linux 发行版的标准初始化系统和服务管理器。在生产环境中，服务无法正常启动是运维人员经常遇到的问题。本文将系统性地介绍 systemd 服务启动故障的排查方法和实战案例。

二、Systemd 基础概念

2.1 核心组件

组件	说明
systemd	系统和服务管理器
systemctl	控制 systemd 的命令行工具
journalctl	查看 systemd 日志的工具
.service 文件	服务单元配置文件

2.2 服务状态

# 查看服务状态
systemctl status <service-name>

# 查看服务是否启用
systemctl is-enabled <service-name>

# 查看服务是否活跃
systemctl is-active <service-name>

三、常见故障类型与排查方法

3.1 服务启动失败

症状

1 2	$ systemctl start myapp Job for myapp.service failed because the control process exited with error code.

排查步骤

步骤 1：查看详细状态

1	systemctl status myapp.service -l --no-pager

步骤 2：查看服务日志

# 查看该服务的最新日志
journalctl -u myapp.service -n 50 --no-pager

# 查看本次启动尝试的日志
journalctl -u myapp.service -b --no-pager

步骤 3：检查依赖服务

# 查看服务依赖
systemctl list-dependencies myapp.service

# 检查依赖服务状态
systemctl status <dependency-service>

步骤 4：手动执行启动命令

# 从 service 文件中获取 ExecStart
systemctl cat myapp.service

# 手动执行启动命令排查问题
/usr/bin/myapp --config /etc/myapp/config.yaml

3.2 服务启动后立刻退出

常见原因

配置文件错误

# 检查配置文件语法
myapp --check-config

# 查看配置文件权限
ls -la /etc/myapp/config.yaml

端口被占用

# 检查端口占用
ss -tlnp | grep <port>
netstat -tlnp | grep <port>
lsof -i :<port>

权限不足

# 检查服务运行用户
systemctl cat myapp.service | grep User

# 检查文件权限
ls -la /var/log/myapp/
ls -la /var/lib/myapp/

资源限制

# 查看服务资源限制
systemctl show myapp.service | grep -E "Limit|Memory|CPU"

# 检查系统资源
free -h
df -h
ulimit -a

3.3 服务超时启动

症状

1 2	$ systemctl start myapp Job for myapp.service timed out.

排查方法

步骤 1：增加超时时间临时测试

1	systemctl start myapp.service --timeout=300

步骤 2：修改服务超时配置

# 编辑服务配置
systemctl edit myapp.service

# 添加以下内容
[Service]
TimeoutStartSec=300

步骤 3：检查启动阻塞原因

# 分析启动时间
systemd-analyze blame | head -20

# 查看服务启动详细时间线
systemd-analyze critical-chain myapp.service

3.4 循环重启故障

症状

$ systemctl status myapp.service
● myapp.service - My Application
   Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
   Active: activating (auto-restart)

排查方法

步骤 1：查看重启次数

1	systemctl show myapp.service \| grep -E "NRestarts\|RestartSec"

步骤 2：临时禁用自动重启

systemctl edit myapp.service
# 添加：
[Service]
Restart=no

步骤 3：分析重启原因

1 2	# 查看完整重启历史 journalctl -u myapp.service --since "1 hour ago" \| grep -E "Started\|Stopped\|Failed"

四、实战案例

案例 1：Nginx 服务启动失败

问题现象

1 2	$ systemctl start nginx Job for nginx.service failed.

排查过程

# 1. 查看状态
$ systemctl status nginx.service
● nginx.service - A high performance web server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
   Active: failed (Result: exit-code)
  Process: 12345 ExecStart=/usr/sbin/nginx (code=exited, status=1/FAILURE)

# 2. 查看日志
$ journalctl -u nginx.service -n 20
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

# 3. 检查端口占用
$ ss -tlnp | grep :80
LISTEN  0  128  0.0.0.0:80  0.0.0.0:*  users:(("apache2",pid=5678,fd=4))

# 4. 解决方案
$ systemctl stop apache2
$ systemctl start nginx

案例 2：MySQL 服务启动超时

问题现象

1 2	$ systemctl start mysql Job for mysql.service timed out.

排查过程

# 1. 分析启动时间
$ systemd-analyze critical-chain mysql.service
mysql.service +1min 30s

# 2. 查看日志发现大量 InnoDB 恢复
$ journalctl -u mysql.service | grep "InnoDB"
InnoDB: Starting crash recovery...
InnoDB: This may take several minutes...

# 3. 增加超时时间
$ systemctl edit mysql.service
[Service]
TimeoutStartSec=600

# 4. 优化 InnoDB 配置（长期方案）
# 编辑 /etc/mysql/mysql.conf.d/mysqld.cnf
innodb_fast_shutdown=0

$ systemctl daemon-reload
$ systemctl start mysql

案例 3：自定义服务权限问题

问题现象

1 2	$ systemctl status myapp Active: failed (Result: exit-code)

排查过程

# 1. 查看服务配置
$ systemctl cat myapp.service
[Service]
User=myapp
Group=myapp
ExecStart=/opt/myapp/bin/myapp
WorkingDirectory=/opt/myapp

# 2. 检查目录权限
$ ls -la /opt/myapp/
drwxr-xr-x  3 root root 4096 Mar 18 10:00 .
drwxr-xr-x 20 root root 4096 Mar 18 10:00 ..
drwxr-xr-x  2 root root 4096 Mar 18 10:00 logs

# 3. 修正权限
$ chown -R myapp:myapp /opt/myapp
$ chmod 755 /opt/myapp/bin/myapp

# 4. 重启服务
$ systemctl start myapp
$ systemctl status myapp
● myapp.service - My Application
   Active: active (running)

五、高级排查技巧

5.1 使用 strace 追踪系统调用

# 获取服务启动命令
EXEC_START=$(systemctl cat myapp.service | grep ExecStart | cut -d= -f2)

# 使用 strace 追踪
strace -f -o /tmp/myapp.strace $EXEC_START

# 分析 strace 输出
grep -E "EACCES|ENOENT|EPERM" /tmp/myapp.strace

5.2 使用 systemd 调试模式

# 启用调试日志
systemctl set-log-level debug

# 重启服务并查看详细日志
systemctl restart myapp.service
journalctl -u myapp.service --priority=debug

5.3 服务依赖树分析

# 可视化服务依赖
systemd-analyze dot | dot -Tsvg > /tmp/systemd.svg

# 查看特定服务的依赖树
systemctl list-dependencies myapp.service --reverse

六、预防措施

6.1 服务配置最佳实践

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Service
Documentation=https://docs.myapp.com
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=notify
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStartSec=120
TimeoutStopSec=30

# 资源限制
MemoryLimit=2G
CPUQuota=80%

# 安全加固
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/myapp /var/lib/myapp

[Install]
WantedBy=multi-user.target

6.2 监控与告警

# 创建监控脚本
cat > /usr/local/bin/check-systemd-services.sh << 'EOF'
#!/bin/bash
FAILED_SERVICES=$(systemctl --failed --no-legend | wc -l)
if [ $FAILED_SERVICES -gt 0 ]; then
    echo "CRITICAL: $FAILED_SERVICES systemd services failed"
    systemctl --failed --no-legend
    exit 2
fi
echo "OK: All systemd services healthy"
exit 0
EOF

chmod +x /usr/local/bin/check-systemd-services.sh

七、常用命令速查

命令	说明
`systemctl status <svc>`	查看服务状态
`systemctl start/stop/restart <svc>`	启动/停止/重启服务
`systemctl enable/disable <svc>`	启用/禁用开机自启
`journalctl -u <svc> -f`	实时查看服务日志
`systemctl cat <svc>`	查看服务配置文件
`systemctl edit <svc>`	编辑服务覆盖配置
`systemctl daemon-reload`	重新加载 systemd 配置
`systemd-analyze blame`	分析启动时间
`systemctl list-dependencies <svc>`	查看服务依赖