104 commands you actually type, every flag explained — grouped by what you are trying to do
Find, batch-process, archive — a wrong flag here usually means deleting the wrong file or silently skipping dotfiles
-l long format: permissions, owner, size, time
-a include . .. and all dotfiles
-h human-readable sizes (1.2K not 1234); only works with -l
-t sort by modification time, newest first
-r reverse; with -t you get oldest first
-S sort by size, largest first — more direct than -t when hunting disk hogs
--time-style=long-iso print 2026-10-29 09:18 instead of the default
# newest files first, with size and mtime ls -laht --time-style=long-iso # biggest space hogs first (-S) ls -lhS /var/log | head
-name match filename, case-sensitive; -iname ignores case
-type f files only; -type d directories; -type l symlinks
-mtime +7 modified more than 7 days ago; -mmin -30 within 30 minutes
-size +50M larger than 50M; -size -1k smaller than 1k
-maxdepth 1 do not recurse — add this first when debugging
-ls runs ls -dils on each hit: size, inode and time in one go
-delete removes directly (empty dirs only); -exec ... \; runs a command per hit
-prune skips a directory; must be combined with -path
-o means OR when chaining conditions; -print forces output (required with -prune)
# large files changed in the last 30 min (disk spike triage) find /var -type f -mmin -30 -size +50M -ls # bulk delete logs older than 7 days find /var/log -name '*.log.*' -mtime +7 -delete # skip .git and node_modules while searching find . -path ./node_modules -prune -o -name '*.js' -print
-r recursive; -R recursive and follow symlinks
-i ignore case; -n show line numbers; -v invert (only non-matching lines)
-E extended regex (alternation requires -E or it is treated literally)
-l filenames only — good for narrowing down before reading
-A3 / -B3 / -C3 show 3 lines after / before / around a match
--include='*.conf' restrict by suffix; --exclude-dir skip directories
-w whole-word match, so cat does not match category
-rn is the everyday pair: recursive plus line numbers, otherwise you cannot find the line
# find an upstream in the nginx conf, 3 lines of context grep -rn -C3 'upstream' /etc/nginx --include='*.conf' # match any of several keywords (requires -E) grep -rnE 'timeout|refused|reset' /var/log/app/error.log # list only filenames with matches, then open each grep -rli 'proxy_pass' /etc/nginx
-c create; -x extract; -t list only; -z gzip; -j bzip2
-f takes the archive name, so -f always goes last
-v verbose; slows large archives a lot, redirect it when scripting
-C extract into a directory (it must already exist)
--exclude skip paths while packing; the pattern matches relative paths
-czf / -xzf / -tzf are the three combinations worth memorising
Always -t before extracting — it prevents a tarbomb from overwriting cwd
# archive, excluding cache dirs tar -czf app.tar.gz --exclude='node_modules' --exclude='.git' -C /srv app # list contents first, then extract to a chosen dir tar -tzf app.tar.gz | head -20 tar -xzf app.tar.gz -C /tmp/restore
rm -rf recursive force, no confirmation — with an empty variable, rm -rf "$DIR/" becomes rm -rf /
-f force, no prompts; -r recurse into directories — together they are the dangerous pair
cp -a archive mode: keeps permissions, times, symlinks — the right flag for backups
cp -n never overwrite (no-clobber); same for mv -n
mv across filesystems is copy-then-delete, not atomic; an interrupt leaves both sides incomplete
Files removed while a process holds them keep consuming space until that process exits
test -n "$DIR" checks the variable is non-empty — do this before any rm
-- ends option parsing, so a filename starting with - is not read as a flag
# safe delete: verify the var is non-empty first [ -n "$DIR" ] && rm -rf -- "$DIR" # full backup preserving attributes cp -a /srv/app /backup/app-$(date +%F)
df -h usage per mount point; -i inode usage (a full inode table also blocks writes)
du -sh total size of one directory; -s prints one total, -h human readable
du -sh * | sort -h largest first — sort -h understands 1K/2M/3G
--max-depth=1 one level only, far faster than full recursion
A big gap between du and df almost always means deleted-but-held-open files
2>/dev/null drops permission-denied noise, which otherwise floods the output
# drill down from root; enter whichever level balloons du -h --max-depth=1 / 2>/dev/null | sort -h | tail -10 # deleted files held by a process (df full but du small) lsof +L1 2>/dev/null | head
ln -s TARGET LINKNAME — target comes first; reversing it creates a broken link
-s symbolic; without -s it is a hard link (same filesystem only, not for directories)
-f overwrite an existing link
-n with -f, replace a link that points to a directory instead of nesting inside it
-sfn is the standard release switch (current points at the new release)
Symlinks store a path and break when the target moves; hard links share an inode
ln -sfn /srv/app/releases/20261029 /srv/app/current
stat gives size, inode, mode and three timestamps — the direct answer to "was this file really modified"
-c custom format: %y mtime, %s bytes, %U:%G owner, %a octal mode, %n filename
-L follow the symlink and describe the target (without it you describe the link itself)
Three times: Access read, Modify content changed, Change metadata (mode/owner) changed
-f filesystem-level facts (block size, inode totals), complementary to df
touch -d '2026-10-01 08:00' FILE sets mtime; pair with find -newer to bound changes
file reports the real type, which is more trustworthy than the extension
stat -c '%a' is the fastest way to confirm the actual mode bits
stat -c '%y %s %U:%G %a %n' /etc/nginx/nginx.conf stat -L -c '%y %s' /srv/app/current # show the real file a symlink points to file /usr/local/bin/app
Editing configs, column stats, building command lines — anything touching 200 lines belongs to sed/awk
-i edits the file in place — on production, run it once WITHOUT -i first and read the output
-i.bak keeps a .bak copy while editing, so you can roll back; worth making a habit
-n suppresses auto-printing; with p it prints only the lines you ask for
-E extended regex (+ ? | ( ) need no escaping); on older systems it is -r, same meaning
-e chains expressions; they can also be separated by semicolons inside one expression
The g in s/old/new/g replaces every occurrence per line; without g only the first
Address ranges: '/^server/,/}/ s/a/b/' edits only inside one config block
\1 \2 reference capture groups; & is the whole match
Order matters: sed -E -i.bak 's/x/y/' FILE
sed -n '1,20p' /etc/nginx/nginx.conf # first 20 lines only sed -i.bak 's/listen 80;/listen 8080;/' /etc/nginx/sites-enabled/app sed -E -i.bak 's/^server_name[[:space:]]+([^;]+);/server_name \1.bak;/' app.conf sed '/^[[:space:]]*#/d;/^[[:space:]]*$/d' app.conf | head # strip comments and blank lines
-F sets the separator: -F: colon, -F'\t' tab, default is runs of whitespace
'{print $1,$3}' takes columns 1 and 3; $0 is the whole line; NF is field count; NR is line number
Filters precede the program: '$9==500' means column 9 equals 500
Force numeric comparison with +0: $9+0 >= 500, otherwise string compare surprises you
'{s+=$7} END {print s}' is the sum template; count with '{c++} END {print c}'
Array stats: {c[$1]++} END {for (k in c) print k, c[k]}
-v passes variables in: awk -v lim=500 '$3>lim' — -v comes before the program
-F and -v are parsed by awk itself, so they must precede the quoted program
In the examples: sort -rn sorts numerically in reverse, uniq -c counts, ss -tn lists TCP connections
awk -F: '$3>=1000 {print $1, $3}' /etc/passwd
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
awk -v lim=500 '{ if ($9+0 >= lim) print NR": "$7 }' access.log
ss -tn | awk 'NR>1 {c[$1]++} END {for (k in c) print k, c[k]}'sort -n numeric order (default lexicographic puts 10 before 9)
sort -h understands 1K/2M/3G — required for du and df output
sort -r reverse; -u sort and dedupe in one pass
-k2 sorts by field 2, -k3 likewise; multiple keys as -k1,1 -k2,2
-t: sets the separator to colon (default splits on whitespace)
uniq -c counts consecutive repeats — hence the sort first, or equal values split into blocks
uniq -d shows only duplicated lines, uniq -u only unique ones
The Top N pattern: sort | uniq -c | sort -rn | head
In the example du -sh: -s prints one total instead of per-entry lines (see du / df)
du -sh /var/log/* | sort -h | tail -5
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
sort -t: -k3 -n /etc/passwd | headcut -d: -f1,7 takes fields by separator: -d sets it, -f picks fields (multiple allowed)
cut -c1-19 slices by character position — good for the timestamp part of fixed-width logs
cut understands only one separator; space-aligned output breaks it, so use awk instead
tr 'a-z' 'A-Z' maps characters one-to-one; for string replacement use sed
tr -d deletes characters — most often tr -d '\r' to strip Windows line endings
tr -s squeezes repeated characters into one, handy for collapsing runs of spaces
tr reads stdin only, not filenames, so redirect with <
In the examples: df -h human-readable, cut -f5 takes field 5, sort -u dedupes
cut -d: -f1,7 /etc/passwd | head cut -c1-19 /var/log/nginx/access.log | head -3 tr -d '\r' < crlf.txt > lf.txt df -h | tr -s ' ' | cut -d' ' -f5 | sort -u
xargs turns stdin lines into arguments — that is how find | xargs rm works
-0 pairs with find -print0 so filenames with spaces or newlines are not split (mandatory for user uploads)
find -print0 terminates each hit with \0, exactly what xargs -0 expects
-n1 passes one argument per invocation, safer and it shows which item failed
-I {} is a placeholder, so the argument can sit mid-command: xargs -I {} cp {} /backup
-p asks before running; -t prints the command first — use -t when unsure what gets built
-r skips execution on empty input (GNU), otherwise an empty list still fires one rm
-P4 runs four processes in parallel — several times faster for batch compress or download
cp --parents recreates the directory structure, which helps when restoring backups
In the examples: find -name / -mtime / -type filter, and rm -f forces without prompting
find /var/log -name '*.log.*' -mtime +7 -print0 | xargs -0 -r rm -f
find /srv/uploads -type f -print0 | xargs -0 -I {} cp --parents {} /backup
printf '%s\n' a b c | xargs -n1 -t echotee writes its input to a file and stdout at the same time — the go-to for keeping evidence
-a appends instead of truncating; always use it for logs
Why sudo tee matters: in sudo echo x > /etc/f the redirection is done by your shell, so it fails
-i ignores interrupts, so a long capture is not cut short by Ctrl-C
Merge stderr with 2>&1 before the pipe, otherwise errors never reach tee
Timestamp a stream with ts, or: awk '{print strftime("%F %T"), $0}'
./deploy.sh 2>&1 | tee -a /var/log/deploy-$(date +%F).log echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf
jq parses JSON by structure instead of matching lines, which grep cannot do safely
. reads fields: .data .items[0].name; .[] iterates arrays
-r prints raw strings without quotes — required when feeding variables or further pipes
-c puts each result on one line, which makes sort / uniq counting easy
'keys' lists object fields; 'length' array size; '..' dumps all paths when the shape is unknown
select() filters: '.[] | select(.status != "ok")'
-C colourises for humans, -M forces plain output for files and scripts
jq pretty-prints by default; -e turns the result into an exit code for scripting
In the examples curl -s silences progress output that would otherwise corrupt the JSON
curl -s https://api.example.com/v1/tools | jq -r '.data[].name'
curl -s 'https://api.example.com/v1/list?page=1' | jq -c '.items[] | {id, name}'
jq -r '.[] | select(.status != "ok") | .id' resp.jsondiff -u unified output is patch format and can be applied by patch directly
diff -r compares two trees; -q reports only which files differ
diff -y shows them side by side, -W 200 sets the width — useful for eyeballing configs
wc -l counts lines (the usual log metric); -c bytes; -w words
wc prints the filename too, so awk '{print $1}' when scripting
comm is better for set differences of two sorted lists
comm -13 a b prints lines only in b; check the column logic before trusting it
Sort and dedupe both sides first, otherwise "all different" is really just different order
diff -u /etc/nginx/nginx.conf.bak /etc/nginx/nginx.conf | head -40 diff -rq /srv/app/releases/20261028 /srv/app/releases/20261029 wc -l /var/log/nginx/access.log sort a.txt > a.s; sort b.txt > b.s; comm -13 a.s b.s # lines only in b
Login, keys, transfer, resume — the slowest part of the day is usually getting the file there
ssh user@host is the basic login; port 22 by default
-p 2222 sets the port (note: scp and sftp use uppercase -P — do not mix them up)
-i points at a private key; otherwise ~/.ssh/id_ed25519 and id_rsa are tried
-t forces a pseudo-terminal — required for remote sudo, top and other interactive tools
-v traces handshake and auth; tells you whether it stalls on the network or on auth; -vvv for more
-o passes options: -o ConnectTimeout=5 avoids hangs, -o BatchMode=yes forbids all prompts (mandatory in scripts)
Run one remote command without a shell: ssh HOST 'cmd' — the basis of batch checks
-J jumps through a bastion: ssh -J bastion@1.2.3.4 deploy@10.0.0.5 (older form: -o ProxyCommand)
In the examples df -h means human-readable (see du / df)
ssh -p 2222 -i ~/.ssh/id_ed25519 deploy@10.0.0.5 ssh -o ConnectTimeout=5 -o BatchMode=yes 10.0.0.5 'uptime; df -h /' ssh -v -t 10.0.0.5 'sudo systemctl restart app' ssh -J bastion@1.2.3.4 deploy@10.0.0.5
ssh-keygen -t ed25519 creates a key pair (shorter and faster than rsa; supported everywhere modern)
-t picks the type: ed25519 preferred, rsa only for old systems, with -b 4096 for length
-C writes a comment (use or email) so you can tell several keys apart
-f names the output file, avoiding an accidental overwrite that silently kills existing logins
-N '' sets an empty passphrase for automation; interactive users can just press Enter
ssh-copy-id appends the public key to the remote ~/.ssh/authorized_keys and fixes permissions
-i (of ssh-copy-id) selects which public key to send; the default picks id_*.pub
-R deletes a stale host fingerprint from known_hosts — run it after any reinstall, or you get HOST IDENTIFICATION HAS CHANGED
-lf prints fingerprint and type, to confirm both machines hold the same key
Private keys must be 600 and the directory 700, or ssh refuses to use them
ssh-keygen -t ed25519 -C 'deploy@icetools' -f ~/.ssh/id_deploy -N '' ssh-copy-id -i ~/.ssh/id_deploy.pub deploy@10.0.0.5 ssh-keygen -R 10.0.0.5 ssh-keygen -lf ~/.ssh/id_deploy.pub
After this, ssh app equals ssh -p 2222 deploy@10.0.0.5 — no more retyping
Host is the alias and the scope for the settings that follow
HostName real address; User login name; Port port; IdentityFile key path
IdentitiesOnly yes offers only that key, instead of letting the agent try a dozen (too many attempts can lock you out)
ProxyJump is the bastion, equivalent to -J, and scp / rsync inherit it automatically
ServerAliveInterval 30 sends a heartbeat every 30s; NAT and cloud load balancers kill idle connections
ServerAliveCountMax 3 drops after three failed heartbeats, paired with the interval above
Host * at the end of the file carries global defaults (heartbeat, timeouts)
Permissions: config 600 and ~/.ssh 700, or you get Bad owner or permissions
# ~/.ssh/config (chmod 600)
Host app
HostName 10.0.0.5
User deploy
Port 2222
IdentityFile ~/.ssh/id_deploy
IdentitiesOnly yes
ServerAliveInterval 30
ServerAliveCountMax 3
Host db
HostName 10.0.0.6
ProxyJump app
Host *
ServerAliveInterval 30-L local forward: ssh -L 8080:127.0.0.1:80 HOST maps remote port 80 to local 8080
-R reverse forward: exposes a local port on the remote side, so outside users reach an internal service
-D dynamic forward creates a SOCKS proxy (use with curl --socks5 or a browser)
-N forwards only, no shell; -f backgrounds it — the two almost always travel together
-g lets other hosts use the local forwarded port (by default it listens on 127.0.0.1 only)
The middle address in -L is resolved from the remote side; 127.0.0.1 there means the remote host itself
For "the database cannot be reached directly", pull the port locally with -L, then connect to 127.0.0.1
If the local port is taken, pick another; lsof -i :8080 shows the current holder
ssh -N -f -L 15432:127.0.0.1:5432 deploy@10.0.0.5 ssh -N -L 8080:10.0.0.6:80 app ssh -D 1080 -N -f app ssh -g -L 8080:127.0.0.1:80 app
scp SRC DST — direction decides upload or download: local first uploads, remote first downloads
-P (uppercase) sets the port (ssh uses lowercase -p; this is the classic mix-up)
-r recurses into directories; without it, copying a directory fails outright
-i selects a private key; -o passes ssh options such as -o ConnectTimeout=5
-C compresses in transit — real gains on text and logs, useless on already-compressed archives
-l 8000 rate-limits in Kbit/s (8000 ≈ 1MB/s) so production bandwidth is not saturated
-p (lowercase) preserves times and modes — unrelated to -P, do not confuse them
scp cannot resume; a dropped multi-GB transfer restarts, so use rsync -P for those
scp -P 2222 app.tar.gz deploy@10.0.0.5:/srv/ scp -r -i ~/.ssh/id_deploy ./dist deploy@10.0.0.5:/srv/app/ scp -C -l 8000 backup.tar.gz deploy@10.0.0.5:/backup/ scp -p deploy@10.0.0.5:/var/log/app/error.log ./
sftp user@host opens an interactive session — "ls while transferring" beats blind scp copies
-P (uppercase) port and -i key selection, same as scp
get REMOTE downloads, put LOCAL uploads; get -r pulls a whole directory
reget resumes a download and reput resumes an upload — the real edge over scp
-p (of get/put) preserves modes and timestamps
mget / mput take wildcards, but confirm each file by default — run prompt off first
ls / lls list remote and local; cd / lcd change remote and local directory (the l-prefixed ones are local)
!df -h runs a local shell command — check free space before pulling something big
bye / exit quits; a heredoc script makes batch transfers non-interactive
sftp -P 2222 -i ~/.ssh/id_deploy deploy@10.0.0.5 <<'EOF' get -r /srv/app/logs ./ reget backup.tar.gz put ./dist/app.tar.gz /srv/ EOF
-a archive mode (= -rlptgoD): keeps modes, times, symlinks, group — the everyday default
-v shows progress; -z compresses in transit, essential over the internet
-P equals --partial --progress: keep the half file and show progress — this is the resume switch
--partial keeps what arrived so the next run continues; without it every run starts over
--append only appends, good for append-only logs, wrong for files that change
--delete makes the destination mirror the source (it removes extras) — dry-run with -n first
-n and --dry-run are the same: print what would happen, change nothing
-e 'ssh -p 2222' selects the remote shell and port; rsync has no port option of its own
--exclude drops one pattern, --exclude-from reads a list from a file
--bwlimit=2000 throttles in KB/s; --checksum compares content instead of size and time
--info=progress2 shows one overall progress line, better than per-file -P on big trees
The biggest trap: a trailing slash on the source copies the CONTENTS, no slash copies the directory
rsync -avzP ./dist/ deploy@10.0.0.5:/srv/app/dist/ rsync -avzP -e 'ssh -p 2222' deploy@10.0.0.5:/var/log/app/ ./logs/ rsync -avn --delete /srv/data/ /backup/data/ # dry run, review it, then go for real rsync -avz --bwlimit=2000 --exclude='*.tmp' ./ /mnt/nas/app/
wget URL downloads into the current directory; -O names the output, -O - writes to stdout
-c resumes (curl needs -C -); rerun the same command after a drop and it continues
-q quiet mode; --progress=dot:giga is a compact progress display
--limit-rate=1m throttles, with k/m suffixes; --tries=3 sets retry count
-T 30 sets the connect timeout in seconds; --timeout is the long form
-r mirrors a site, -np stops it climbing to parent directories (which can drag in the whole domain)
--no-check-certificate skips certificate checks — internal diagnostics only, never in a script
When you need custom headers, cookies or a session, curl is the better tool
wget -c https://example.com/big.iso wget -O - https://example.com/install.sh | head -40 # read it before deciding to run it wget --limit-rate=1m --tries=3 -c -O app.tar.gz https://example.com/app.tar.gz
Numeric modes, special bits, sudo — permission bugs usually show up as "password is right but access denied"
Numeric: 4 read / 2 write / 1 execute, digits are owner / group / other
755 for directories (rwxr-xr-x); 644 for files (rw-r--r--)
600 owner-only read-write — required for private keys
Symbolic: u/g/o/a for who, +/-/= for what, X adds execute only to directories
-R recursive; use a+X rather than a+x so plain files do not become executable
go-w strips group and other write, in the same pass as a+X
After a recursive change, spot-check one file and one directory with ls -l
chmod 600 ~/.ssh/id_rsa chmod -R a+X,go-w /srv/app/public ls -l /srv/app/public | head # spot-check a few files after the recursive chmod
chown user:group path; a dot also works as the separator
Change only the group with chown :group, or chgrp
-R recursive; add -h to touch the symlink rather than its target
--reference=FILE copies owner and mode from an existing file
Only root may change ownership; users may only change group of their own files
For Permission denied on a container volume, compare host directory uid with the container user
chown -R www-data:www-data /var/www/app chown --reference=/srv/app/a /srv/app/b
4000 SUID: runs as the file owner (that is how /usr/bin/passwd works)
2000 SGID: runs as the group; on a directory, new files inherit that group
1000 sticky: only the file owner may delete inside the directory (/tmp works this way)
An s or t in ls means the bit is set; uppercase S/T means set without execute
-perm -4000 means "mode contains SUID"; -perm is find's permission test
-xdev stays on one filesystem — without it find walks NFS and container mounts and crawls
-ls prints hits as ls -dils, showing size and time together
-type f restricts to regular files, skipping directories and symlinks
Audit for privilege escalation with find / -xdev -perm -4000 -type f
# audit: list every SUID binary find / -xdev -perm -4000 -type f -ls 2>/dev/null # shared dir: group collab, only the owner can delete chmod 2770 /srv/share
umask is the set of bits to REMOVE, not the mode itself
022 gives directories 755 and files 644 (files never get execute by default)
077 gives directories 700 and files 600 — right for key material
-S prints the resulting default mode symbolically (rwxr-xr-x), easier to read than digits
Temporary: umask 077 affects only the current shell
Persistent: add it to /etc/profile or ~/.bashrc
If newly created keys are world-readable, check umask first
umask # show the current value umask 077 # applies to this session umask -S # show the real default perms in symbolic form
sudo -i opens a fresh root login shell (loads root's environment)
sudo -s root shell with your current environment and PATH
sudo -u USER run as another user — common when testing services
sudo -l list what you are allowed to run — first stop when something is denied
su - switches to root with its environment; without the dash you keep yours
Edit sudoers with visudo; -c validates the syntax without saving — run it after every edit
Hand-editing /etc/sudoers can lock you out — the classic self-inflicted outage
sudo -l sudo -u www-data /usr/bin/env # inspect the service account's environment visudo -c # syntax check only
useradd -m creates a home directory; -s sets the shell; -G adds supplementary groups
Without -m there is no home directory and many tools fail in confusing ways
usermod -aG GROUP USER — dropping the -a removes the user from all other groups (-a is append)
-G takes the group list and can be combined as -aG
id USER shows uid/gid and every group; groups USER shows just groups
userdel -r removes the home directory too; without -r you get orphaned files
Group changes do not apply to existing sessions — log out and back in
useradd -m -s /bin/bash deploy usermod -aG docker deploy id deploy
Who burns CPU, who holds a file, how to stop it politely — the usual mistake here is reaching for kill -9 too early
ps aux is BSD style: %CPU, %MEM, STAT, COMMAND — the everyday choice
ps -ef is System V style: UID, PID, PPID, C, STIME, CMD — shows parent PIDs
-e selects every process (System V style); -o picks the output columns
ps -aux runs but means something else; do not make it a habit
--sort=-%cpu sorts by CPU descending (the minus means descending); commas chain keys
--sort=-rss sorts by real memory; in STAT, Z is zombie, D is uninterruptible, R is running
RSS is real usage; VSZ is address space and mostly meaningless
ps aux --sort=-%cpu | head ps -eo pid,ppid,rss,pcpu,cmd --sort=-rss | head
In top, press P for CPU, M for memory, T for cumulative time
1 expands per-core detail — tells single-core saturation from overall load
c shows the full command line instead of truncating it
H toggles threads; u filters by user; k kills
-b is batch mode and -n1 prints a single snapshot (-n is the iteration count) — good for scripts
-p limits to given PIDs, comma separated; pgrep's -d, joins multiple PIDs with commas
Load average is 1/5/15 minutes; above the core count means requests are queuing
top -b -n1 | head -15 # take a single snapshot top -p $(pgrep -d, nginx) # watch specific processes only
kill sends TERM(15) by default — the process can flush and exit cleanly
-15 is the same as -TERM; the named form reads better, and is what you should type first
-9 is KILL: the kernel reclaims it immediately, with no chance to clean up — databases and queues can be left corrupt
-1 or -HUP is commonly used to reload config (nginx, sshd)
-0 sends nothing and just checks existence; the exit code is the answer
kill takes PIDs, not names; use pkill or killall for names
pkill -HUP nginx signals every nginx process by name
Always TERM, wait a few seconds, then KILL
kill -TERM 1234 && sleep 3 kill -0 1234 && kill -9 1234 pkill -HUP nginx # reload config
& only backgrounds it; closing the terminal sends SIGHUP and the process dies
nohup ignores SIGHUP and writes to nohup.out by default (redirect explicitly)
2>&1 merges stderr into stdout, otherwise errors vanish somewhere unknown
< /dev/null points stdin at the null device so the job cannot block on terminal input
setsid detaches it from the session entirely — stronger than nohup
disown removes an existing background job from the shell's job table
For anything long-lived use systemd; nohup is for quick verification only
nohup ./run.sh > /var/log/run.log 2>&1 & setsid ./worker.sh < /dev/null > /var/log/worker.log 2>&1 &
lsof -i :8080 shows what holds port 8080 (-i filters by network address)
-p PID lists every file one process has open
-u USER filters by user; -n skips hostname lookups, so it is faster and never stalls on DNS
+D /path recurse into a directory (slow on big trees)
+L1 lists files with link count below 1 — deleted but still open, the usual cause of unreleased disk space
Piping into sort -k7 -n sorts by column 7 (SIZE/OFF) numerically
Without lsof, use fuser -v or inspect /proc/PID/fd
lsof -i :8080 lsof -n +L1 | sort -k7 -n | tail # held deleted files, sorted by size
nice ranges from -20 (highest priority) to 19 (lowest); non-root users may only raise the number
nice -n 19 CMD starts a command at low priority — right for backups and compression
-n carries the nice value itself, not a count
renice -n 10 -p PID changes a running process; -p names the PID, -u works per user
taskset -c 0-3 pins CPUs: -c takes the CPU list, which reduces contention between instances
In taskset -cp 0-3 1234, -p means "operate on an existing process" and the PID follows
Lowering priority does not create capacity — it only protects the important work
In the examples tar -czf: -c creates, -z gzips, -f names the archive (see tar)
nice -n 19 tar -czf backup.tar.gz /srv/data renice -n 10 -p 1234 taskset -cp 0-3 1234
watch reruns the command every 2 seconds by default and clears the screen — you see change, not a snapshot
-n 1 refreshes every second; -n takes the seconds (0.5 is allowed)
-d highlights what changed between two runs, so you spot movement instantly
-d=cumulative keeps the highlight on anything that ever changed — good for intermittent spikes
-t hides the header line (command and interval), leaving a cleaner screen
Quote the command: otherwise the shell splits the pipe and watch only gets the first half
Do not use it for high-frequency metrics (connections, QPS); vmstat 1 or sar shows what watch misses
In the examples ss -s summarises connections and wc -l counts lines (that l means lines)
watch -n 1 -d 'ss -tn state established | wc -l' watch -d=cumulative -t 'ss -s'
screen -S deploy opens a named session; closing the terminal or losing the network leaves it running
-list lists sessions (also -ls) — check for an old Attached session first
-r deploy reattaches; if it says Attached, someone else holds it — use -d -r to take over
Ctrl-a d detaches in screen; Ctrl-a " lists windows; Ctrl-a k kills the current one
tmux new -s deploy creates one, tmux attach -t deploy reattaches (-t names it)
In tmux, Ctrl-b d detaches and Ctrl-b [ enters copy mode for scrolling back
Start a session before migrations, backups or big transfers — a dropped link does not stop the job, it only hides it
screen -S deploy # then Ctrl-a d to detach screen -list screen -r deploy tmux new -s deploy
When a service will not answer, separate "not listening", "not allowed" and "cannot reach" — checking them out of order wastes a lot of time
-t TCP; -u UDP; -l listening only; -n numeric (no service-name lookups, faster); -p show process
-tulnp is the everyday combination — five letters, five jobs
-tnp without the l includes the process for established connections, the usual count check
-a include established connections; -s summary statistics — first look when connection counts spike
state established filters by state, more accurate than grepping ESTAB (that is only the column abbreviation)
netstat is deprecated and often absent on modern systems; use ss
Listening on 127.0.0.1 vs 0.0.0.0 matters: the former is local-only, the number one cause of "port is open but I cannot connect"
ss -tulnp ss -tnp state established | head ss -s
-s silent: no progress bar, no error text — nearly always wanted in scripts
-S works with -s to show errors again — so -sS means "quiet, but speak up on failure"
-i include response headers; -I headers only (sends HEAD, no body)
-v full handshake and request trace — shows exactly which step stalls
-L follow redirects; essential when debugging a 301/302 chain
-X set method; -d send a body (defaults to POST); -H add a header
-o /dev/null drop the body and pair with -w for status and timing only
-w '%{http_code} %{time_total}' custom output, faster than eyeballing
-m 10 caps the whole request at 10s; --connect-timeout 3 limits only the connect phase
--resolve host:443:IP bypass DNS by pinning an IP — tells you if DNS is the problem
-x http://proxy:8080 goes through a proxy; --noproxy '*' forces a direct connection (usually needed when testing locally)
-k skips certificate checks; diagnostics only, never in a script
Inside -w, \n is a newline — without it everything collapses onto one line
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.com/
curl -m 10 --connect-timeout 3 -sS -o /dev/null \
-w 'dns:%{time_namelookup} conn:%{time_connect} ttfb:%{time_starttransfer} total:%{time_total}\n' \
https://example.com/api
curl -iL --resolve example.com:443:1.2.3.4 https://example.com/
curl -v -X POST -H 'Content-Type: application/json' -d '{"a":1}' http://127.0.0.1:8080/apinc -zv HOST PORT — -z probes without sending data, -v prints the verdict, fastest option
-w3 sets a 3 second timeout so the command cannot hang (without it you may wait a long time)
telnet HOST PORT exists on old systems; "Connected" means reachable
Bash built-in /dev/tcp needs nothing installed: echo > /dev/tcp/HOST/PORT
"refused" means the host is reachable but nothing listens; a timeout means a firewall or security group dropped it — very different meanings
Always narrow down in layers: local ss, then curl 127.0.0.1, then nc from the same subnet, then from outside
nc -zv -w3 10.0.0.5 5432 echo > /dev/tcp/10.0.0.5/5432 && echo open || echo closed
ip a shows all interface addresses (the old ifconfig); -4 shows IPv4 only, -6 IPv6 only
ip r prints the routing table; the default line is your gateway
ip neigh shows the ARP table — check it when same-subnet traffic fails
ip -s link gives packet and error counters (-s is statistics); non-zero errors/dropped means a link-layer problem
show eth0 restricts output to one interface, so virtual interfaces do not drown it
add / del change addresses temporarily and vanish on reboot
Persist per distro: netplan or /etc/network/interfaces on Debian, nmcli on RHEL
ip -4 a ip r ip -s link show eth0
+short prints the answer only — script friendly
+A / +AAAA / +MX select record type (A is the default)
@8.8.8.8 queries a specific resolver, to compare local versus public answers
+trace walks down from the root to find which delegation is wrong
When it fails, check status: NXDOMAIN means no such record, SERVFAIL means the server is broken
TTL controls how long a change takes to propagate; the remaining TTL in ANSWER SECTION tells you how stale a cache is
dig only covers DNS. If resolution works but traffic fails, look past DNS at routing and ports
dig +short example.com dig @8.8.8.8 example.com A dig +trace example.com
-i any listens on all interfaces (or name one); -n skips DNS and port-name lookups (faster, and DNS cannot stall it)
-nn means numeric IPs and ports, clearer than -n alone while debugging
port 443 filter by port; host 1.2.3.4 by host; combine with and / or
-c 100 stops after 100 packets instead of flooding the terminal
-w file.pcap saves for Wireshark, far better than reading in the terminal
-A dumps payload as ASCII, -X as hex plus ASCII (only meaningful for plaintext protocols)
tcp[tcpflags] & tcp-rst != 0 captures RST only, which tells you who closed first
Needs root; always add -c and a filter in production
tcpdump -i any -nn -c 50 'port 443 and host 1.2.3.4' tcpdump -i eth0 -nn -w /tmp/cap.pcap 'tcp[tcpflags] & tcp-rst != 0'
-connect HOST:443 inspects the peer certificate at the TLS layer, ignoring HTTP entirely
-servername HOST is mandatory (SNI), or multi-domain IPs hand you the wrong certificate
-showcerts prints the full chain, revealing a missing intermediate
-noout suppresses the raw certificate and prints only what you ask for
-dates shows notBefore / notAfter — the first step of any expiry investigation
-subject shows who the certificate was issued to; -ext subjectAltName lists the covered names
-checkend 2592000 checks whether it expires within 30 days (seconds); the exit code is monitor-friendly
</dev/null makes s_client exit after reading, otherwise it sits waiting for input
openssl s_client -connect www.icetools.cn:443 -servername www.icetools.cn </dev/null 2>/dev/null \ | openssl x509 -noout -dates -subject openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null | head -30 openssl s_client -connect example.com:443 -servername example.com -checkend 2592000 </dev/null >/dev/null \ && echo '30 天内不过期'
firewall-cmd --list-all shows what the active zone allows (RHEL family)
--add-port=8080/tcp adds a rule; only --permanent survives a reload
--reload applies permanent rules — required after adding, or they never reach the running state
ufw status verbose prints rules (Ubuntu)
iptables -L lists the filter table; -n keeps names numeric, -v shows packet counters proving whether a rule matched
--line-numbers prints rule indices, which is what you quote to iptables -D INPUT 3
-I INPUT 1 inserts at the top while -A appends — order decides the outcome
If it still fails after a rule change, suspect the cloud security group; it sits outside this machine
firewall-cmd --list-all firewall-cmd --add-port=8080/tcp --permanent && firewall-cmd --reload iptables -L INPUT -n -v --line-numbers
"Disk is full" has four distinct causes: many files, one huge file, exhausted inodes, or deleted-but-open files
Step 1: df -h to find which mount point is full — do not restart the service yet
Step 2: df -i for inodes. Exhausted inodes report "no space left on device" while df -h still shows room
Step 3: du -h --max-depth=1 to descend level by level toward the big directory
Step 4: if the two numbers disagree, lsof +L1 finds deleted-but-open files; only restarting or killing that process frees them
The awk line sums column 7 (bytes) and converts to MB — faster than eyeballing
Log growth usually shows up in step 1 — read what is spamming before deleting anything
df -h; df -i
du -h --max-depth=1 / | sort -h | tail
lsof +L1 2>/dev/null | awk '{s+=$7} END {print s/1024/1024" MB held"}'Tree layout by default, showing how LVM, partitions and mount points nest
-f shows filesystem type, UUID and mount point — check this before editing fstab
-p prints full device paths, so /dev/sda and /dev/sda1 cannot be confused
-d hides children and shows whole disks only
-o picks the columns; NAME,SIZE,FSTYPE,MOUNTPOINT,UUID is the useful set
After growing a disk, if lsblk shows the new size but df does not, the filesystem has not been resized yet
lsblk -f lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT,UUID
Growing a cloud disk takes two steps: the partition, then the filesystem — skip one and df stays the same
growpart /dev/vda 1 expands the partition (cloud-utils-growpart); the second argument is the partition number
xfs_growfs / grows XFS (grow only, cannot shrink)
resize2fs /dev/vda1 grows ext4 (can shrink, but only after umount and a filesystem check)
LVM needs lvextend as an extra step, then the filesystem resize
Confirm with df -h that the capacity really changed
Snapshot first — a power loss mid-resize can lose data
growpart /dev/vda 1 xfs_growfs / # XFS resize2fs /dev/vda1 # ext4 df -h /
Common -o options: ro read-only, remount, noatime to reduce write amplification
remount is the workhorse: change mount options without unmounting, so nothing goes down
If umount says device is busy, find the holder with lsof +D or fuser -vm
Prefer UUID over /dev/sdb in fstab — device names can change
Field 4 is usually defaults,noatime; fields 5 and 6 are dump and fsck order, use 0 0 for data disks
Always run mount -a after editing fstab; a bad line can make the next boot fail
mount -a mounts everything fstab lists that is not mounted, and names the offending line on error
mount -o remount,rw / fuser -vm /mnt/data mount -a # validate the fstab entry
Every file consumes one inode, and the total is fixed at format time
Symptom: df -h shows free space but writes fail with "No space left on device"
df -i shows IUse%; near 100% confirms it
-xdev stays on one filesystem so the count does not wander into other mounts
-type f counts files only — directories use inodes too, but the file distribution is the useful view
The sed strips everything after the last slash to leave the directory; sort | uniq -c counts files per directory
Usual culprits: session files, upload temp files, mail spools, cache directories with hundreds of thousands of tiny files
Fixing properly needs a reformat with a larger -i ratio; short term, delete small files
The trailing sort -rn means numeric reverse order: -r reverse, -n numeric
df -i find /var -xdev -type f | sed 's#/[^/]*$##' | sort | uniq -c | sort -rn | head
Most broken cron jobs are a PATH or redirect problem; a wrong clock breaks certificates, JWTs and logs together
crontab -l lists your jobs; -e edits; -r deletes them all with no confirmation
-u USER operates on another user's crontab (run as root)
Five fields are minute hour day month weekday: 0 3 * * * means 03:00 daily
*/5 every 5 minutes; 1-5 Monday to Friday; 0,12 only at 00:00 and 12:00
cron's environment is not your login shell and PATH is minimal — always use absolute paths
cron does not read ~/.bashrc; set variables at the top of the crontab or source them in the script
Redirect output: >> /var/log/x.log 2>&1, otherwise errors go to mail and you never see them
When nothing ran, check journalctl -u cron (Debian) or /var/log/cron (RHEL)
In the examples curl -sS is quiet but reports errors (-s silent, -S shows errors), -o /dev/null drops the body
crontab -e crontab -l # daily backup at 3:30, output to a log 30 3 * * * /srv/app/backup.sh >> /var/log/backup.log 2>&1 # health check every 5 minutes */5 * * * * /usr/bin/curl -sS -o /dev/null http://127.0.0.1:8080/health
Real advantages: logs go to journalctl, the next run time is visible, and missed runs can catch up
OnCalendar takes calendar syntax: *-*-* 03:30:00 daily at 03:30; OnUnitActiveSec=5min every 5 minutes
Persistent=true runs once after boot if the machine was off during the window — cron cannot do this
Unit= names the service to run; omit it and the same-named .service is used
systemctl list-timers shows every timer and its next trigger
enable --now is "start on boot plus run now"; enable alone does not trigger anything yet
To test, start the .service, not the .timer — starting the timer does not fire the job
The last line: journalctl -u selects the unit, -n 30 takes the last 30 lines, --no-pager avoids paging
# /etc/systemd/system/backup.timer [Unit] Description=Daily backup [Timer] OnCalendar=*-*-* 03:30:00 Persistent=true [Install] WantedBy=timers.target # enable and inspect systemctl list-timers systemctl enable --now backup.timer journalctl -u backup -n 30 --no-pager
date shows the clock and timezone; +%F is 2026-10-29, +%T the time, '+%F %T' is the usual pair
-d '-1 hour' computes an hour ago, -d 'next monday' also works — every time window in a script uses -d
-s sets the clock directly; never on production, it scrambles log order and certificate checks
timedatectl shows timezone and NTP sync state, more complete than date
set-timezone Asia/Shanghai fixes the most common local problem: still on UTC
set-ntp true enables automatic sync; containers inherit the host clock
A wrong clock shows up concretely: "not yet valid" certificates, instantly expired JWTs, mismatched log times, jobs firing at odd hours
date '+%F %T' date -d '-1 hour' '+%F %T' timedatectl timedatectl set-timezone Asia/Shanghai
chronyc tracking shows offset and stratum; the System time line is literally how far off you are
An offset above a few hundred milliseconds means trouble: unreachable source, UDP 123 blocked, VM clock drift
chronyc sources -v lists each source, -v printing detail (delay, jitter, whether selected)
A leading ^* marks the source in use; ^? means unreachable
chronyc makestep steps the clock immediately — manual use only when the offset is large
ntpq -p is the equivalent for old ntpd systems, -p printing the peer list
On cloud hosts prefer the vendor's internal NTP; public NTP is often rate-limited or refused
chronyc tracking chronyc sources -v chronyc makestep ntpq -p # legacy systems (ntpd)
Sequence beats syntax: bound the time window first, narrow by keyword, then read the context
-u UNIT filters to one service
-f follows; --since '10 min ago' or --since '2026-10-29 09:00' bounds the window
-p err shows error and above; priorities run from emerg down to debug
-n 100 last 100 lines; -r reverse (newest first)
--no-pager avoids the pager, which helps when redirecting or piping
-o short-iso includes the timezone offset — essential across regions
--disk-usage reports how much space the journal takes; --vacuum-time=7d keeps only 7 days
--vacuum-size=500M trims by volume instead — pick one of the two
journalctl -u nginx -p err --since '30 min ago' --no-pager journalctl -u app -f -o short-iso journalctl --disk-usage
-f follows the file descriptor; after rotation it keeps reading the old file and the log looks frozen
-F follows by name and reopens after rotation — always use this when watching for a while
-n 100 shows the last 100 lines first; -n0 shows no history at all, only new output
-Fn0 is simply -F plus -n0, the usual pair for a live stream
--pid=PID stops following when that process exits
grep --line-buffered makes grep flush per line; without it the pipe stalls and you see nothing
Together: tail -Fn0 app.log | grep --line-buffered -i error
Multiple files: tail -F a.log b.log prints ==> separators; do not watch just one
tail -F /var/log/app/error.log tail -Fn0 /var/log/nginx/access.log | grep --line-buffered -i 500
OOM kills, disk IO errors, link-down events and driver faults all appear here
-T prints human-readable timestamps; -w follows
-l err,warn filters by level (-l takes a comma-separated list)
The ring buffer gets overwritten — capture early; journalctl -k is the persistent view of the same data
"Out of memory: Killed process" means the OOM killer did it, not a crash
With grep -iE: -i ignores case and -E enables extended regex (so | works, see the grep entry)
dmesg -T -l err,warn | tail -30 dmesg -T | grep -iE 'oom|killed process'
Confirm with dmesg -T | grep -i -A2 'killed process' — -A2 brings the next two lines, which hold the victim's memory footprint
journalctl -k carries the same event plus the memory state at the time
status=9/KILL or signal=KILL in systemctl status almost always means the OOM killer
--no-pager keeps the pager from taking over, which matters in scripts and screenshots
This is not a crash: the application log just stops, with no stack trace
Find the cause: free -h for available memory, then sort processes by RSS
Inside containers, check the cgroup limit at /sys/fs/cgroup/memory.max
dmesg -T | grep -i -A2 'killed process' systemctl status app --no-pager | head -20 free -h
Put configs in /etc/logrotate.d/, one file per service
daily / weekly for the period; rotate 7 keeps 7 archives
compress compresses old logs; delaycompress keeps the newest uncompressed
copytruncate copies then truncates — for programs that cannot reopen files, at the cost of a small loss window
postrotate signals the process to reopen, cleaner than copytruncate
-d is a dry run; always dry-run after editing to see the plan
-f forces one rotation now, to verify the config without waiting for the period
missingok silences "no such file", notifempty skips empty logs — both belong in nearly every config
# /etc/logrotate.d/app
/var/log/app/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
postrotate
systemctl reload app > /dev/null 2>&1 || true
endscript
}
# dry run first, then force once to verify
logrotate -d /etc/logrotate.d/app
logrotate -f /etc/logrotate.d/appAlmost every failed service is one of three things: wrong path, wrong user, missing dependency — status tells you which
start / stop / restart / reload — reload rereads config without restarting, only if the program supports it
status shows recent journal lines and the exit code; --no-pager keeps the pager out
enable for boot autostart; --now starts it at the same time
disable removes autostart but does not stop it; use stop separately
list-units --failed shows what is broken; --all includes inactive units
is-active / is-enabled exit codes work directly in scripts
cat prints the unit as systemd reads it (drop-ins included) — check it after editing
systemctl status app --no-pager systemctl enable --now app systemctl list-units --failed
Place it in /etc/systemd/system/NAME.service and always daemon-reload afterwards
ExecStart needs an absolute path and must not fork on its own (else use Type=forking)
Restart=on-failure plus RestartSec=5 prevents restart storms
User / Group decide permissions; switching off root usually means fixing home and file ownership too
EnvironmentFile keeps variables in a separate file instead of inlining them in ExecStart
WorkingDirectory affects relative paths; many programs cannot find their config without it
StandardOutput=journal routes logs to journalctl
After only orders startup and guarantees nothing; for networking add Wants=network-online.target
# /etc/systemd/system/app.service [Unit] Description=App After=network-online.target Wants=network-online.target [Service] Type=simple User=deploy WorkingDirectory=/srv/app EnvironmentFile=/etc/app/env ExecStart=/srv/app/bin/app Restart=on-failure RestartSec=5 StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target
1. systemctl status: check Active and the last error lines — this resolves most cases
2. journalctl -u SERVICE -n 50 --no-pager for the full trace (-n is the line count)
3. Confirm the ExecStart path exists and is executable: ls -l it
4. Run it by hand as the service user with sudo -u — the real error usually appears instantly
5. Check policy: ausearch -m avc -ts recent (-m selects the avc record type, -ts the start time)
AppArmor denials land in dmesg; inside containers also read docker logs
status=203/EXEC means path or permission; status=1 is the program exiting with code 1
systemctl status app --no-pager journalctl -u app -n 50 --no-pager sudo -u deploy /srv/app/bin/app # run it by hand to see the real error ausearch -m avc -ts recent # SELinux denial log
Edited a .service file: daemon-reload is mandatory, otherwise systemd keeps the old definition
Only changed values in EnvironmentFile: no reload needed, but a restart is required to pick them up
New unit: reload before it can be enabled
A running service does not adopt new config on reload — restart it
Classic symptom of a missed reload: ExecStart was changed but status still shows the old command
cat shows the file as systemd sees it, including drop-in overrides
systemctl daemon-reload && systemctl restart app systemctl cat app # see how systemd actually reads this file
After only orders startup, it does not guarantee success
Requires is a hard dependency: if the other unit fails, this one fails too
Wants is soft: this unit starts anyway — the right default in most cases
For networking, network.target only means the stack is initialised; addresses may not be ready
After=network-online.target only truly waits when paired with Wants=network-online.target
"Name resolution failed" right after boot is almost always this
Verify with systemctl list-dependencies SERVICE for the tree
[Unit] Wants=network-online.target After=network-online.target # inspect systemctl list-dependencies app
Check where a package comes from before installing, and what it takes with it before removing — especially on production
apt update refreshes indexes only and upgrades nothing — the most commonly reversed pair
apt upgrade upgrades installed packages; full-upgrade may remove packages to satisfy dependencies
install -y skips the prompt, which scripts need; without it apt waits for input
apt purge removes config files too; plain remove leaves them behind
apt list --installed shows what is present; apt list --upgradable filters to upgradable ones
apt-cache show PKG for version and description
apt-mark hold pins a version so bulk upgrades skip it
apt update && apt install -y nginx apt list --upgradable apt purge nginx-common
dnf replaces yum; on newer systems yum is usually a symlink
install / remove / update differ from apt — do not mix them up
install -y skips prompts, same idea as apt
provides reverse-looks-up a package by file or command
history lists transactions and history undo last rolls back the previous one — a lifesaver after a bad upgrade
--enablerepo enables a repo for one run; --nogpgcheck only when there is no other way
Modules (dnf module list) matter when several versions of one package coexist
dnf install -y nginx dnf provides /usr/sbin/ss dnf history undo last
RHEL: rpm -qf /usr/bin/ss — -q enters query mode and -f looks up by file
Debian: dpkg -S /usr/bin/ss; when that misses, run apt-file update then apt-file search
If you only know the command name: dnf provides '*/ss' or apt-file search bin/ss
dpkg -L PKG lists every file a package installed — useful before touching configs
rpm -qc PKG lists configuration files only (-qc is -q plus -c for config)
When a command is missing, confirm the package exists before blaming PATH
rpm -qf /usr/bin/ss dpkg -S /usr/bin/ss dpkg -L nginx-common | head -20
apt clean removes the whole .deb cache; apt autoclean only expired entries
dnf clean all clears cache and metadata
apt autoremove drops packages installed as dependencies that nothing needs now; --purge their configs too
-y is still required for autoremove, or it stops at the confirmation prompt
Run du -sh first to confirm the space is real (-s total, -h human readable)
Forgetting to clean after apt install in a Dockerfile wastes tens of MB per layer
du -sh /var/cache/apt /var/cache/dnf 2>/dev/null apt clean && apt autoremove --purge -y dnf clean all
Nine out of ten container problems are a mismatch between the config you think is running and the one actually running — align the facts with inspect first
-a include exited containers — listing only running ones hides the one that just crashed
-q print container IDs only, handy for piping into docker rm or docker logs
-s show writable-layer size, for hunting a container that filled the disk
--format custom columns; '{{.Names}} {{.Status}}' reads better than the default table and is script-friendly
--filter narrow by condition: status=exited for dead ones, name=web by name
--no-trunc do not truncate long IDs and full commands
-n 5 show only the 5 most recently created (short for --last)
Up 3 seconds (restarting) in Status means an endless restart loop — check docker logs --tail
# all containers, including exited
docker ps -a --format '{{.Names}} {{.Status}}'
# find exited or crash-looping ones
docker ps -a --filter status=exited -n 5
# who is eating the disk
docker ps -s --format '{{.Names}} {{.Size}}'-f follow the stream, same as tail -f
--tail 200 last 200 lines only — long-running containers have tens of thousands, and without this the useful part scrolls away
-t prefix every line with a timestamp, essential when correlating with host logs
--since 10m only the last 10 minutes; absolute times like 2026-10-29T10:00:00 also work
--until pair with --since to cut out a window
-i on grep means case-insensitive, so error / ERROR / Error are all caught
Logs live in /var/lib/docker/containers/<ID>/<ID>-json.log and vanish when the container is removed —
ship the important ones to journald or a collector instead of leaving them inside docker
docker logs -f --tail 200 -t myapp docker logs --since 10m myapp | grep -i error
-it interactive terminal; without -t there is no prompt, without -i you cannot type — pass both
-u root run as root, for images whose default user cannot read the config
-w /app set the working directory so you do not have to cd after entering
-e KEY=val inject an env var temporarily to check whether one was not passed correctly
-- stops docker from parsing further arguments; required when the command itself starts with -
-c on bash runs the rest as a command string, so bash -c 'a && b' does several steps at once
-l on ls gives the long format with permissions and owner
If exec fails, first check the container is running: exec into something Exited in docker ps -a always fails,
and the real problem is that the main process already died
No bash in the image? Try sh — alpine-based images usually only have sh
Distroless images have no shell at all; use docker cp to pull files out, or a temporary debug container
docker exec -it -u root myapp sh docker exec myapp env | grep -i database docker exec -it myapp bash -c 'ls -l /app'
--format evaluates a Go template; -f is the shorthand
Useful paths: .State.Status, .State.ExitCode,
.NetworkSettings.Networks for the container IP, .Mounts for bind mounts, .Config.Env for env vars,
.HostConfig.RestartPolicy for the restart policy
{{json .}} emits raw JSON, easier to pipe into jq
--type image inspects an image instead of a container
Values in compose can be overridden by command-line -e; only inspect shows what really took effect
.Config.Env often contains passwords and tokens — filter it before pasting anywhere
docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}}' myapp
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' myapp
docker inspect --format '{{json .Mounts}}' myapp | jq--no-stream take a single snapshot and exit, good for scripts and periodic logging
--format custom columns, easy to grep or load into a database
-a include stopped containers too (they show 0, but they are listed)
MEM LIMIT is that container's cap; MEM % near 100 means an OOM is imminent
CPU % is scaled by core count, so a single container above 100% on a multicore host is normal
When a container was OOMKilled stats shows nothing (the process is gone); check ExitCode 137 via docker inspect
and the oom-kill lines in dmesg
docker stats --no-stream --format '{{.Name}} {{.CPUPerc}} {{.MemUsage}}'
docker stats --no-stream -a-t tag the image; use name:version rather than only latest
-f point at a Dockerfile elsewhere than the build directory
--no-cache ignore the cache entirely — first thing to try when "I changed the code but the image did not"
--build-arg pass a build argument; only works if the Dockerfile declares it with ARG
--target stop at a specific stage, useful for debugging intermediate layers of a multi-stage build
--progress=plain print the full build log; the default TTY mode collapses parallel output into one line and hides errors
The trailing . is the build context, not the Dockerfile path — a large context slows builds down, so exclude node_modules and .git via .dockerignore
docker build -t myapp:1.2.3 . docker build -f docker/Dockerfile.prod --build-arg NODE_ENV=prod --progress=plain -t myapp:prod .
-f select a compose file; stack two -f flags for multiple environments, the later one wins
-p set the project name — it defaults to the directory name, so several stacks on one host must set it explicitly or their containers and networks collide
-d run detached; without it compose attaches and Ctrl+C stops the stack
--build rebuild images before starting
up create and start; down stop and remove containers plus networks
logs -f --tail 100 follow logs; append a service name to follow just that one
config validate and print the merged final config — run it after editing compose, safer than going straight to up
compose v2 is docker compose (space), v1 is docker-compose (hyphen); their flags differ slightly, so check which one the script has
docker compose -f docker-compose.yml -f docker-compose.prod.yml -p shop up -d --build docker compose -p shop logs -f --tail 100 api docker compose -p shop config
system df summarises usage by images / containers / volumes / build cache; -v breaks it down per object
prune removes unused objects; by default only dangling images (<none>), add -a to remove every image no container references
--volumes also clears volumes — a database may live in one, and that is unrecoverable
--filter until=24h only delete things older than 24 hours so nothing in use gets hit
Order of operations: system df -v first to see who is hogging space, then a targeted prune; do not open with system prune -a
Container logs fill disks too: /var/lib/docker/containers/*/*-json.log
The real fix is log-opts with max-size and max-file in daemon.json
docker system df -v docker image prune -a --filter until=24h docker system prune --volumes
Kubernetes troubleshooting follows a near-fixed order: status, then events, then logs, then a shell. Skipping a step means guessing
-A all namespaces — add this first when a resource "does not exist"; usually it is the wrong namespace
-n pick a namespace; omit it and the context default applies
-o wide adds IP and node columns, essential for network and scheduling problems
-o yaml print the full definition; export before editing anything
--show-labels show labels — when a Service cannot find its Pods it is almost always a label mismatch
-l app=api filter by label, more reliable than grepping names
--sort-by=.status.startTime order by start time, newest last
Short names exist: pod/po, svc, deploy, rs, no, cm, secret
kubectl get pod -A -o wide kubectl get pod -n shop -l app=api --show-labels kubectl get pod -A --sort-by=.status.startTime
The Events block at the bottom is the single most informative part of Kubernetes troubleshooting —
reasons for ImagePullBackOff, FailedScheduling, Unhealthy and OOMKilled all appear here
Omitting -n and hitting the wrong namespace gives a bare NotFound with no useful hint
Events are kept for about an hour by default, so capture them; do not count on finding them later
describe node ends with Allocated resources showing request / limit percentages —
when a Pod stays Pending, check whether requests are exhausted (not whether CPU is actually busy)
describe covers config and events only, never application logs — that is kubectl logs
kubectl describe pod -n shop api-7d9f-abc12 kubectl describe node k8s-node-2 | tail -30
-f follow the stream
-n select a namespace; omit it and the context default applies
--tail 200 last 200 lines; long-running Pods flood the terminal without it
-c select a container — required when a sidecar shares the Pod, otherwise you get an ambiguous error listing the containers
--previous read the previous container: CrashLoopBackOff restarts the container repeatedly,
so the current log only shows the "died right after start" fragment and the real cause is in the previous one
--since 10m only the last 10 minutes
--timestamps prefix each line with a timestamp
Resources accept the type/name form such as deploy/api to read a Deployment's Pod logs directly
kubectl logs -f --tail 200 -n shop deploy/api kubectl logs -n shop api-7d9f-abc12 --previous kubectl logs -n shop api-7d9f-abc12 -c istio-proxy --since 10m
-it interactive terminal; both flags together
-n select the namespace
-c select the container, required for multi-container Pods
-- stops kubectl from parsing the rest; required when the command starts with -
cp takes source and destination, with the Pod side written as pod:path
cp relies on tar inside the container, so distroless images fail outright
kubectl debug injects a temporary debug container sharing the network namespace;
--image picks the debug image (busybox is enough) and --target names the container being debugged —
this is the correct path when the production image has no shell, far faster than rebuilding it
kubectl exec -it -n shop api-7d9f-abc12 -- sh kubectl cp -n shop api-7d9f-abc12:/app/logs/app.log ./app.log kubectl debug -it -n shop api-7d9f-abc12 --image=busybox --target=api
Same source as the Events block in describe, but readable across resources at once — best when you do not yet know which object is broken
-n select the namespace
--sort-by=.lastTimestamp order by time, newest last
--field-selector type=Warning show warnings only, an order of magnitude less noise
--field-selector involvedObject.name=api-7d9f-abc12 narrow to one object
-A all namespaces; cluster-level trouble (node NotReady, evictions) only shows up with it
FailedScheduling carries the scheduler's own words, such as
insufficient cpu or node(s) had taint — it tells you directly why the Pod cannot be placed,
no need to guess between a resource shortage and a taint
Events have a TTL of roughly an hour, so copy them out promptly during a restart loop
kubectl get events -A --sort-by=.lastTimestamp | tail -20 kubectl get events -n shop --field-selector type=Warning kubectl get events -n shop --field-selector involvedObject.name=api-7d9f-abc12
-f points at a manifest file or directory; -f - reads from stdin
--dry-run=client does not submit, it only prints the final YAML — use it to generate templates instead of writing them by hand
--dry-run=server validates on the server (including admission webhooks), far stricter than client
-o yaml is what makes the dry-run output useful
rollout status waits for the rollout to finish and tells you exactly where it stalled
rollout undo reverts to the previous revision; --to-revision=3 picks a specific one, list them with history
rollout restart restarts Pods without changing config — the usual move after editing a ConfigMap or Secret
-n selects the namespace
apply is declarative and idempotent; create is imperative and fails with AlreadyExists the second time
kubectl apply -f deploy/api.yaml --dry-run=client -o yaml > api.gen.yaml kubectl rollout status deploy/api -n shop kubectl rollout undo deploy/api -n shop --to-revision=3 kubectl rollout restart deploy/api -n shop
Format is resource/name localport:remoteport; no NodePort or Ingress needed
-n selects the namespace
--address 0.0.0.0 lets other machines on the subnet connect; by default it binds only 127.0.0.1
It tunnels through kube-apiserver and bypasses the Service —
so "is the app healthy" and "is a NetworkPolicy blocking me" are separate questions, and this settles the first
Closing the terminal stops it; keep it in tmux or screen for long sessions
Troubleshooting order: port-forward works means the app is fine and the problem is Ingress / Service / NetworkPolicy;
port-forward fails means the Pod or the port itself is broken
kubectl port-forward -n shop svc/api 8080:80 kubectl port-forward -n shop --address 0.0.0.0 svc/api 8080:80 kubectl port-forward -n shop pod/api-7d9f-abc12 5432:5432
top node and top pod need metrics-server; without it you get Metrics API not available
--sort-by=cpu or --sort-by=memory sorts the list to surface the heavy consumers
--containers breaks a Pod down per container so you can see which sidecar eats the memory
-A all namespaces
Allocated resources in describe node is the request / limit ratio —
check it first when scheduling fails: oversized requests book the node out while the CPU sits idle
cordon marks a node unschedulable without touching Pods already running there
drain evicts Pods for maintenance; --ignore-daemonsets skips DaemonSets
(which should never be evicted) and --delete-emptydir-data allows removing Pods using emptyDir (their scratch data is lost)
uncordon restores scheduling afterwards
kubectl top node kubectl top pod -A --sort-by=memory --containers kubectl drain k8s-node-2 --ignore-daemonsets --delete-emptydir-data
top tells you which process burns CPU, not what it is waiting on. These tools answer the second question
-p attach to a running process; you can also prefix a command to trace it from startup
-f follow forked children — without it you only see the parent waiting while the real work happens elsewhere
-T print the duration of every call (<0.000012>); this is how you find the slow step
-tt microsecond timestamps
-s 200 show up to 200 characters per string; the default 32 truncates key paths into ...
-o out.txt write to a file; tens of thousands of lines are unreadable on a terminal
-c summarise call counts and total time per syscall on exit — run this first to find the expensive class,
then use -T to pin down a single slow call
Attaching with -p and seeing no output at all means the process is stuck in uninterruptible IO or a kernel lock;
look next at /proc/PID/stack and /proc/PID/wchan
Be careful in production: -f on every call can slow a process several times over, so lean on -c or filters
strace -p 1234 -T -tt -s 200 strace -f -p 1234 -o /tmp/st.txt strace -c -p 1234
-e trace= is the single most useful strace flag
-e trace=file only file-related calls (openat, stat, unlink) — for "file not found" or permission problems
-e trace=network only network calls (connect, sendto, recvfrom) — for "cannot connect" or timeouts
-e trace=openat,connect comma-separated, for a precise few
-e trace=desc only file-descriptor operations, to see which fd blocks
-e signal=SIGSEGV follow a signal only
Filtering turns tens of thousands of lines into dozens — the right order is: -c to find the expensive call class,
then -e trace= to follow just that class
openat returning ENOENT means the path is missing, EACCES means permission denied, EAGAIN means non-blocking with no data;
the three demand completely different fixes, so do not treat every error as a permission problem
Everything else matches the strace -p entry above: -f follows children, -p targets a process, -T shows per-call time, -tt gives microsecond stamps, -o writes to a file
strace -f -e trace=file -p 1234 -T strace -f -e trace=network -p 1234 -tt strace -o net.txt -e trace=openat,connect -f ./app
perf top is a live hot-function profile; no PID needed, it covers the whole machine
-p limit to one process
-F 99 sample at 99Hz — avoid round 100Hz so you do not alias with timer ticks and invent fake hotspots
-g show a call graph; the view is limited here, real call chains need record
unknown as a function name means missing debuginfo or stripped symbols —
perf is not broken, that binary just shipped without symbols
Needs root or a relaxed kernel.perf_event_paranoid; in containers usually --privileged or SYS_ADMIN
Difference from top: top answers "which process", perf top answers "which function" — the latter is where optimisation starts
perf top perf top -p 1234 -F 99 perf top -g
record captures samples, report browses them interactively, script dumps text — the trio
-F 99 sample 99 times per second
-g record call chains; required for flame graphs, without it you only get isolated function names
-p target a process; -a sample every CPU system-wide
--call-graph dwarf unwinds with DWARF info, more accurate than the default fp
at the cost of an order of magnitude more data; if the binary was built with -fomit-frame-pointer you must use it
-o output file, defaults to perf.data in the current directory
-i tells report or script which data file to read
-- prefixes a command so perf can launch and sample it from startup
Flame graphs need the folded output of script, fed into FlameGraph's stackcollapse helper
In containers perf_event_paranoid is usually 2, which hides kernel symbols;
sysctl kernel.perf_event_paranoid=-1 opens it up
perf record -F 99 -g -p 1234 -o /tmp/perf.data perf report -i /tmp/perf.data perf script -i /tmp/perf.data > /tmp/perf.txt
pidstat 1 prints once per second and keeps refreshing; a second number caps how many iterations
-p ALL shows every process; -p 1234 one specific one
-d disk IO (kB_rd/s reads, kB_wr/s writes, iodelay in milliseconds)
-r memory (page faults, RSS)
-u CPU utilisation (the default mode)
-t per-thread breakdown, to find which thread burns the CPU — essential for multithreaded programs
-w context switches: cswch/s is voluntary (waiting on a resource), nvcswch/s involuntary (preempted)
Part of the sysstat package, so minimal installs need it added first
Better than top for long sampling at a fixed interval into a file, so you can average or plot afterwards
pidstat -p 1234 1 10 pidstat -d -p ALL 1 5 pidstat -t -p 1234 1
iostat -x 1 extended stats, refreshed every second
-x prints the extended metrics; without it you get a different, much thinner table
%util near 100% means the disk is saturated; await is the average wait per IO in milliseconds
rkB/s and wkB/s are throughput — high await with low throughput means a slow disk,
high await with high throughput means genuine load; the two call for different capacity work
vmstat 1 one line per second: r is run-queue length (persistently above core count means CPU shortage),
b blocked processes, si / so swap in and out (non-zero means memory pressure)
wa is the share of CPU time waiting on IO; high wa points at the disk, not the CPU
-d here is disk statistics, showing only the disk columns
The first line is the average since boot; real-time values start on the second line — never draw conclusions from line one
iostat -x 1 5 vmstat 1 5 vmstat -d 1
When a process hangs and strace prints nothing (uninterruptible state), you need the user-space stack
gdb -p pauses the process while attached — you must detach before quitting or it stays frozen, which is an outage
-ex runs one gdb command and continues; chain several to attach, dump stacks and detach in one shot
thread apply all bt dumps full stacks for every thread; plain bt is enough for single-threaded programs
gcore writes a core file for offline analysis, safer than live debugging
-o sets the core output path (no .PID suffix appended)
Do not use gdb on a JVM — it cannot read Java stacks, use jstack instead
perf --call-graph samples points over time; a hang needs a full stack, and neither replaces the other
gdb -p 1234 -ex 'thread apply all bt' -ex detach gcore -o /tmp/core.1234 1234
Servers often only ship vi, so editing config means using it. Twenty keys are enough; you do not need all of it
vim +120 file jumps straight to line 120 — the fastest way to follow a line number from an error log
-u NONE loads no config and no plugins, for ruling out a broken vimrc
-w /tmp/keys.log records every keystroke, for figuring out what you actually pressed
:w save; :w! force-save (read-only file or insufficient permission)
:q quit; refused when there are unsaved changes
:q! quit and discard changes, the fast escape hatch
:wq or :x save and quit; :x skips the write when nothing changed
:e! discard changes and reload — when you have mangled the file and want a fresh start
:saveas /tmp/a.conf save as, the standard backup step before editing config
Accidentally edited a read-only file? :q! is the quickest recovery — do not look for a save button
vim +120 /etc/nginx/nginx.conf vim -u NONE /etc/nginx/nginx.conf :w 保存 :q! quit, discarding changes :wq 保存退出 :x write only if changed :e! 重新加载 :saveas /tmp/a.conf save as
gg first line; G last line; 42G jump to line 42
0 start of line; ^ first non-blank character (use this when aligning indentation); $ end of line
w next word start; b previous word start; e end of current word
Ctrl+f page down; Ctrl+b page up; Ctrl+d half page down; Ctrl+u half page up
% jump to the matching bracket — the quick way to find where a config block closes
'' back to the previous position (two single quotes); Enter works too
Skip the arrow keys on big files; hjkl with the motions above is far faster
:set nu before editing so the line numbers in error messages mean something
gg 文件首行 G last line of file 42G 第 42 行 0 start of line ^ 行首首个非空白 $ end of line w / b / e 下词首 / 上词首 / 词尾 Ctrl+f 下一页 Ctrl+b page up % 跳到匹配的括号 '' back to last position
dd delete a line — this is really a cut, so p pastes it back
yy copy a line; 5yy copies five lines
p paste below the current line; P pastes above
u undo; Ctrl+r redo (for when you undid too far)
x delete one character; r replace the character under the cursor without entering insert mode
cc rewrite a whole line, clearing content but keeping indentation
d$ delete to end of line; d0 delete to start; dw delete a word
. repeat the last change — the most valuable key in Vim; delete 20 lines with dd then 19 presses of .
i insert before the cursor; a insert after; o open a line below; O open a line above
Esc returns to normal mode, and forgetting it is how a line of junk gets typed
dd / yy / p delete line / yank line / paste 5yy yank 5 lines u / Ctrl+r undo / redo x / r delete char / replace one char cc change the whole line (keeps indent) . repeat last change (the most valuable key) i / a / o / O insert before / after / new line below / above
/pattern search forward; ?pattern search backward
n next match; N previous match
* searches for the word under the cursor without typing it; # searches backwards
:set hlsearch highlight all matches; :noh clears the highlight (for when the screen is all yellow)
:%s/old/new/g replace everywhere; g means every match on a line, without it only the first
:%s/old/new/gc ask before each replacement — always add c for config files; one wrong replace takes a service down
:10,20s/old/new/g restrict to lines 10 through 20
:.,$s/old/new/g from the current line to end of file
The separator is swappable: :s#/usr/local/old#/usr/local/new#g avoids escaping every slash in a path
Count matches with /pattern before replacing so you know what you are about to change
:%s/old/new/g replace everywhere (all matches per line) :%s/old/new/gc confirm each replace (always add c for configs) :10,20s/old/new/g replace only lines 10-20 :.,$s/old/new/g current line to end of file :s#/usr/local/old#/usr/local/new#g swap delimiter so paths need no escaping :noh clear search highlight
Ctrl+v starts block selection; move the cursor to sweep a column — the tool for bulk comments and alignment
With a block selected, I inserts simultaneously at the start of every line and only applies when you press Esc (forget Esc and it looks like nothing happened)
A appends at the end of every selected line
c replaces the whole block, d deletes it
:g/pattern/d deletes every matching line
:g/pattern/s/old/new/g replaces only inside matching lines, safer than a global replace
:sort sorts the selected lines; :sort u sorts and removes duplicates
:normal @a runs the macro in register a once per selected line, for repetitive work
:set nu first so you cannot misjudge the range during block edits
Ctrl+v I Esc after block select, insert at start of every line (bulk comment) Ctrl+v A Esc after block select, append at end of every line :g/^#/d delete all lines starting with # :g/ERROR/s/old/new/g replace only on lines containing ERROR :sort u sort and dedupe the selected lines
Pasting code from a browser or Windows and the indentation marches right: :set paste before pasting, :set nopaste after
(or :set pastetoggle=<F2> to toggle with F2)
A file edited on Windows shows ^M at line ends: :set ff=unix then :w
:set ff alone reports whether it is currently dos or unix
Mojibake in Chinese text: :set fileencoding=utf-8 and save; if you only want to read it correctly without touching the file,
use :e ++enc=gbk to reinterpret it as GBK
:w !sudo tee % solves "I forgot sudo, now I cannot save" —
it writes the buffer back through sudo under the current filename, faster than reopening and redoing the edits
All of these are per-session; put them in ~/.vimrc to make them stick
:set paste turn off autoindent before pasting :set nopaste restore it afterwards :set ff=unix strip Windows ^M :set fileencoding=utf-8 fix the encoding, then write :w !sudo tee % force write when you forgot sudo
"The server is slow" is not one problem but four: CPU, memory, IO, network. One command tells you which
uptime shows 1/5/15 minute load averages — above the core count means queuing
High load with idle CPU (high %id in top) points at IO wait — watch the wa column
In free -h read the available column, not the free column
Check df -h (capacity) and df -i (inodes) together; a full disk produces strange failures
vmstat 1 5 samples once per second, five times — watch r (run queue), b (blocked), wa (IO wait), si/so (swapping)
Identify the layer before acting — far more effective than restarting services
uptime free -h df -h; df -i vmstat 1 5
Look at the wa column in vmstat: sustained above 10 means IO is the drag
Then the b column (blocked processes): above the core count means many are stuck
Find the culprit with pidstat -d 1 5 (-d reports IO stats only, every second, five times)
In ps -eo, -e selects all processes and -o picks columns; the state column reveals D (uninterruptible)
awk '$1=="D"' filters processes stuck in D state — those are the ones blocked on IO
Typical causes: full table scans, synchronous log writes, a running backup, flaky NFS or cloud disks
Processes in D state cannot be killed — wait for IO or reboot
vmstat 1 5 pidstat -d 1 5 ps -eo state,pid,cmd | awk '$1=="D"'
In free -h, available is what new processes can still get; a low free value is normal — Linux uses idle memory as cache
High buff/cache is not a problem; it is reclaimed on demand
Heavy swap use means real pressure; sustained non-zero si/so in vmstat is thrashing
Sort processes: ps -eo pid,rss,comm --sort=-rss (-e all, -o columns, minus means descending)
RSS is real usage; VSZ is address space and mostly meaningless
In containers the cgroup limit matters (/sys/fs/cgroup/memory.max), not the host total
For a suspected leak, compare two RSS snapshots ten minutes apart and see which one only grows
free -h vmstat 1 5 # watch si / so ps -eo pid,rss,comm --sort=-rss | head
Layer 1: curl -w splits DNS, connect, time-to-first-byte and total
curl -o /dev/null drops the body, -s silences progress, -w prints your own format — together you see numbers only
Layer 2: curl 127.0.0.1 on the server itself, ruling out network and gateway
Layer 3: compare app log timings with nginx $request_time and $upstream_response_time
A large gap means the delay is in an intermediate proxy or connection pool
Do not forget the database slow query log — many "slow endpoints" are slow queries
High total but low upstream time usually means throttling or queuing
curl -o /dev/null -s -w 'dns:%{time_namelookup} conn:%{time_connect} ttfb:%{time_starttransfer} total:%{time_total}\n' https://example.com/api
# add to nginx log_format:
# $request_time $upstream_response_timeping -c 50 sends 50 packets and reports loss% plus mdev (jitter); a single ping proves nothing
ping -M do -s 1472 tests path MTU: -M do forbids fragmentation and -s sets the payload size
mtr -r -c 50: -r prints a report instead of a live screen, -c sets the packet count, and it beats traceroute by showing per-hop loss
ICMP rate limiting can mimic loss; TCP probes (nc -z, curl) are more reliable
ethtool -S NIC shows rx/tx error and drop counters (-S is statistics)
grep -iE means ignore case plus extended regex
Loss within one rack points at NIC or cabling; across regions, at the carrier
ping -c 50 10.0.0.5 mtr -r -c 50 10.0.0.5 ethtool -S eth0 | grep -iE 'err|drop'
On an unfamiliar box, run this first for the full picture, then decide what to dig into
Each line is labelled with the question it answers — do not copy it as a black box
Flag cheat sheet (all covered above): --sort=-%cpu descending CPU, -h human readable, -i inodes
grep -vE means invert plus extended regex, used here to drop virtual filesystems such as tmpfs
ss -tulnp unpacks as t/TCP, u/UDP, l/listening, n/numeric, p/process
For systemctl, --failed shows broken units only and --no-pager avoids paging
For dmesg, -T gives readable time and -l filters by level
Everything here is read-only, safe on production, and paste-ready for a ticket
echo "== 负载 =="; uptime echo "== CPU 前 5 =="; ps aux --sort=-%cpu | head -6 echo "== 内存前 5 =="; ps aux --sort=-%mem | head -6 echo "== 内存 =="; free -h echo "== 磁盘 =="; df -h | grep -vE 'tmpfs|overlay' echo "== inode =="; df -i | grep -vE 'tmpfs|overlay' echo "== 监听端口 =="; ss -tulnp echo "== 失败服务 =="; systemctl list-units --failed --no-pager echo "== 内核错误 =="; dmesg -T -l err | tail -10