2.2 File Management

Linux Directory Structure

Essential Directories for DevOps:

/                      # Root filesystem
├── bin/              # Essential user commands (ls, cat, cp)
├── sbin/             # System administration commands
├── etc/              # System configuration files
│   ├── systemd/      # Service definitions
│   ├── nginx/        # Web server config
│   ├── ssh/          # SSH configuration
│   └── ssl/          # SSL certificates
├── home/             # User home directories
├── opt/              # Optional software (Docker, third-party apps)
├── tmp/              # Temporary files (cleared on reboot)
├── usr/              # User programs and data
│   ├── bin/          # User commands
│   ├── lib/          # Libraries
│   └── local/        # Locally installed software
├── var/              # Variable data
│   ├── log/          # System and application logs
│   ├── lib/          # Application data (databases)
│   ├── cache/        # Application cache files
│   └── spool/        # Print queues, mail, cron jobs
└── root/             # Root user home directory

Container and Cloud Considerations:

# Container-specific paths
/var/lib/docker/      # Docker data (images, containers, volumes)
/var/lib/kubelet/     # Kubernetes node data
/etc/docker/          # Docker daemon configuration
/etc/kubernetes/      # Kubernetes configuration

# Cloud provider agents
/opt/aws/             # AWS CLI and tools
/etc/waagent.conf     # Azure agent (Azure VMs)
/etc/google/          # Google Cloud agent configuration

# Configuration management
/etc/ansible/         # Ansible configuration
/etc/puppet/          # Puppet configuration
/etc/salt/            # Salt configuration

File Operations

Navigation and Discovery:

# Enhanced listing with details
ls -la                           # List all files with permissions, dates
ls -lah                          # Human-readable file sizes
ls -lt                           # Sort by modification time
ls -lS                           # Sort by file size
tree /path/                      # Show directory structure (install: apt install tree)

# Modern alternatives
exa -la                          # Modern ls replacement (install: apt install exa)
bat filename                     # Syntax-highlighted cat (install: apt install bat)
fd pattern /path/                # Fast find alternative (install: apt install fd-find)

File Operations with Safety:

# Safe copying with backups
cp -i source destination         # Interactive mode (confirm overwrites)
cp -a source/ destination/       # Archive mode (preserves all attributes)
rsync -av source/ destination/   # Better for large files/directories

# Safe moving and renaming
mv -i source destination         # Interactive mode
rename 's/old/new/' *.txt        # Batch rename using regex

# Safe deletion
rm -i filename                   # Interactive deletion
rm -rf dirname                   # Recursive force delete (DANGEROUS!)
trash filename                   # Move to trash instead (install: apt install trash-cli)

File Content Operations:

# View file content
cat filename                     # Display entire file
less filename                    # Paginated view (q to quit)
head -n 20 filename              # First 20 lines
tail -n 20 filename              # Last 20 lines
tail -f /var/log/syslog          # Follow log file changes

# Modern alternatives
bat filename                     # Syntax highlighting + line numbers
delta file1 file2                # Better diff tool
jq . config.json                 # Pretty-print JSON files

File Permissions and Security:

# Permission format: rwxrwxrwx (owner group other)
# r=read(4), w=write(2), x=execute(1)

# Common permission patterns
chmod 644 file.txt               # rw-r--r-- (readable config files)
chmod 600 private.key            # rw------- (private keys, secrets)
chmod 755 script.sh              # rwxr-xr-x (executable scripts)
chmod 700 ~/.ssh/                # rwx------ (SSH directory)
chmod +x script.py               # Add execute permission

# Ownership changes
chown user:group file            # Change user and group
chown -R user:group directory/   # Recursive ownership change

# Advanced permissions
chmod u+s executable             # Set SUID (run as owner)
chmod g+s directory/             # Set SGID (inherit group)
chmod +t /tmp/                   # Sticky bit (only owner can delete)

DevOps-Specific Permissions:

# Configuration files
chmod 640 /etc/nginx/nginx.conf          # Readable by group
chmod 600 /etc/ssl/private/server.key    # Private keys
chmod 644 /etc/systemd/system/app.service # Service files

# Application directories
chown -R app:app /var/lib/myapp/
chmod 755 /var/lib/myapp/
chmod 644 /var/lib/myapp/config/*

# Log files
chmod 640 /var/log/myapp.log
chown app:adm /var/log/myapp.log

Search and Find Operations:

# Find files and directories
find /path -name "*.py"                    # Find Python files
find /path -type f -size +100M             # Files larger than 100MB
find /path -mtime -7                       # Modified in last 7 days
find /path -user appuser                   # Files owned by specific user
find /etc -name "*.conf" -exec chmod 644 {} \;  # Find and fix permissions

# Modern find alternatives
fd "\.py$" /path/                          # Faster find (install: apt install fd-find)
fd --type f --size +100m                   # Large files with fd

# Content search with grep
grep -r "pattern" /path/                   # Search in files recursively
grep -n "error" /var/log/app.log           # Show line numbers
grep -i "warning" /var/log/*               # Case-insensitive search
grep -v "debug" /var/log/app.log           # Exclude lines containing "debug"
grep -A 5 -B 5 "error" /var/log/app.log   # Show 5 lines before/after match

# Modern grep alternatives
rg "pattern" /path/                        # ripgrep (faster, install: apt install ripgrep)
ag "pattern" /path/                        # silver searcher (install: apt install silversearcher-ag)

Advanced File Operations:

# Symbolic and hard links
ln -s /path/to/file symlink                # Create symbolic link
ln /path/to/file hardlink                  # Create hard link
readlink -f symlink                        # Show target of symbolic link

# Archives and compression
tar -czf backup.tar.gz directory/          # Create compressed archive
tar -xzf backup.tar.gz                     # Extract archive
tar -tzf backup.tar.gz                     # List archive contents
zip -r archive.zip directory/              # Create ZIP archive
unzip archive.zip                          # Extract ZIP archive

# Disk usage analysis
du -sh directory/                          # Directory size (human readable)
du -ah directory/ | sort -rh | head -20    # Top 20 largest files/dirs
df -h                                      # Filesystem usage
lsblk                                      # List block devices

# Modern disk usage tools
ncdu /path/                                # Interactive disk usage (install: apt install ncdu)
dust /path/                                # Fast du replacement (install: cargo install du-dust)

DevOps File Operations:

# Configuration management
diff -u original.conf modified.conf        # Show configuration differences
rsync -av --delete source/ dest/           # Sync directories (delete removed files)

# Log file operations
tail -f /var/log/{syslog,auth.log,nginx/*.log}  # Follow multiple logs
journalctl -f -u nginx                     # Follow systemd service logs
logrotate -f /etc/logrotate.d/nginx        # Force log rotation

# Security and compliance
find /etc -type f -perm /o+w               # Find world-writable files
find /home -name ".ssh" -type d            # Find SSH directories
stat filename                             # Show detailed file information

# Container and cloud operations
docker cp container:/path/file ./          # Copy from container
kubectl cp pod:/path/file ./file           # Copy from Kubernetes pod
aws s3 sync ./local/ s3://bucket/prefix/   # Sync to cloud storage

Warning

Dangerous Commands: Never run rm -rf / or similar destructive commands. Always double-check paths before deleting.

Permission Classes

  • User (u): The owner of a file/directory

  • Group (g): The members of the file/directory’s group

  • Others (o): Any users that are not part of the user or group classes

Permission Types

  • Read (r/4): view/copy file/directory contents

  • Write (w/2): view/copy/move/delete file/directory

  • Execute (x/1): execute file, access directory

Access Permissions

Octal

Symbolic

Description

0

—

no permissions

1

–x

execute only

2

-w-

write only

3

-wx

write and execute

4

r–

read only

5

r-x

read and execute

6

rw-

read and write

7

rwx

read, write and execute

Access rights commands

# The chown (change owner) command alters the user that a file or directory belongs to
chown --help

# The chgrp (change group) command alters the group that a file or directory belongs to
chgrp --help

# The chmod (change mode) command alters the file permissions
chmod --help

chmod 700 file

chmod u=rwx file

Special permissions

Linux offers three other types of permissions, called special permission bits that may be set on executable files or directories to allow them to respond differently for certain operations.

  • setuid bit: affects only on files, provides non-owners the ability to run executables with the privileges of the owner

  • setgid bit: has an effect on files and directories, used for group collaboration (alters the standard behavior so that the group of the files created inside the directory, will not be that of the user who created them, but that of the parent directory)

  • sticky bit: When the sticky bit is set on a directory, only the owner of a file within that directory can delete or modify it, even if other users have write permissions to the directory. A typical case is the /tmp directory, which is writable by all users on the system, but users cannot delete files owned by others.

Default permissions

Linux assigns default permissions to a file or directory at the time of its creation. Default permissions are calculated based on the umask (user mask) value subtracted from a preset value called initial permissions (777 for directories, 666 for files).

initial permissions

777 -

666 -

umask

022

022

default permissions

775

644

Control attributes

There are certain attributes that may be set on a file or directory in order to control what can or cannot be done to it. For example, you can enable attributes on a file or directory so that no users, including root, can delete, modify, rename, or compress it.

The commands to list or change attributes are lsattr and chattr.

Useful commands

# Shows information about file system disk usage
df -h

# Shows information about directory and file sizes on the disk
du -h /var

# Shows information about directory and file sizes on the disk
# Only on the first level of directories
# Sort output in reversed order and human-readable format
du -h --max-depth=1 /usr | sort -rh

# Find and print all directories in /usr
find /usr -type d

# Find and print all files with .log extension in /var/log
find /var/log -type f -name "*.log"

# Find directories and files owned by Alice
find / -user alice

# Find directories and files owned by the billing group
find / -group billing

# Find files larger than 10MB and list them in long format
find / -type f -size +10M -exec ls -lh {} \;

TODO

  1. create /opt/billing directory

  2. create /opt/billing/clients and /opt/billing/invoices files

  3. configure alice user as owner for /opt/billing directory and its contents

  4. configure billing group as owner for /opt/billing directory and its contents

  5. configure access rights for /opt/billing directory:

    1. read, write and execute for alice user

    2. read and execute for billing group

    3. read and execute for everyone else

  6. remove access rights to others for all files in the /opt/billing directory

  7. add write permissions on /opt/billing/invoices for billing group

  8. make john as owner for /opt/billing/clients and assign him read-only rights

  9. make alice as owner for /opt/billing/invoices and assign her read-only rights

Warning

To err is human … to really f*ck up requires the root password.

Solution to TODO

  1. mkdir /opt/billing

  2. touch /opt/billing/clients /opt/billing/invoices

  3. chown alice /opt/billing

  4. chgrp billing /opt/billing

  5. chmod 755 /opt/billing

  6. chmod o-r /opt/billing/*

  7. chmod g+w /opt/billing/invoices

  8. chown john /opt/billing/clients chmod u-w /opt/billing/clients

  9. chown alice /opt/billing/invoices chmod u-w /opt/billing/invoices