Hetzner VPS

Initial Situation

• App code disappeared
• File .pwned found
• Indications of brute‑force/compromise

Goal: Treat the server as compromised, rebuild it completely, and operate it hardened afterwards (Least Privilege).

---

1. Immediate Actions (Incident Response)

1. Classify the server as compromised

   • No further trust in the system state
   • No repair attempts on the running server

2. Briefly inspect evidence (optional, short)

   • stat .pwned
   • last, who, w
   • /var/log/auth.log
   • crontab -l, /etc/crontab
   • Goal: confirmation, not forensics

3. Delete the server in the Hetzner dashboard

   • ❗ Do not stop it, actually delete it
   • Remove old disks, processes, backdoors

---

2. Local Preparation (before reinstalling)

1. Generate new SSH keys

   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_hetzner

   • With a strong passphrase

   Copy public key

   cat ~/.ssh/id_ed25519_hetzner.pub

2. Generate a separate key for GitHub Actions.

   ssh-keygen -t ed25519 -f ~/.ssh/github-actions-hetzner

   • ❌ No passphrase (CI‑compatible)

3. Sort out old / unclear keys

   • Locally and in the Hetzner dashboard

---

3. Create a New Server (Hetzner)

1. Server type

   • Cost‑Optimized
   • e.g. CX23

2. OS

   • Ubuntu 22.04 LTS or 24.04 LTS

3. Region

   • e.g. Nuremberg

4. Networking

   • Public IPv4 + IPv6: ✅
   • Private networks: ❌

5. SSH key

   • Add only the new personal key
   • Do not use the old compromised key
   • Add as standard ssh-key can be activated

---

4. First Login & Basic Setup

1. Login as root

   ssh root@SERVER_IP

   In case of warning: “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!”

   ssh-keygen -R <SERVER_IP> // remove old entry for the same ip

2. Update system

   apt update && apt upgrade -y

3. Create deploy user

   adduser deploy // add secure password

   usermod -aG sudo deploy // add deploy user to sudo group

4. Set SSH key for deploy

   If the user has a different name, replace /deploy with that name

   mkdir -p /home/deploy/.ssh

   Copy ssh key used for Hetzner and paste it into authorized_keys

   nano /home/deploy/.ssh/authorized_keys

5. Set proper rights

chown -R deploy:deploy /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys

Test login with ssh deploy@<SERVER_IP>

---

5. Harden SSH

1. SSHD configuration

   nano /etc/ssh/sshd_config

   Important values:

PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
AllowUsers deploy # only user deploy can log in via ssh

2. Test & reload

   sshd -t // test configuration, ok if no response

   systemctl reload ssh

---

6. Install & Configure Firewall (UFW)

1. Install & enable UFW

sudo apt install ufw -y
sudo ufw default deny incoming
sudo ufw default allow outgoing

2. Allow required ports

sudo ufw allow OpenSSH
sudo ufw allow 80
sudo ufw allow 443

3. Enable firewall

sudo ufw enable
sudo ufw status verbose

4. Change firewall settings

# List rules with numbers
sudo ufw status numbered

# Delete rule
sudo ufw delete < number >

# Allow ssh access only for certain ips (IPv4)
sudo ufw allow from < public-ip > to any port 22 proto tcp

# Check for public ip
curl -4 https://ipinfo.io/ip
curl -6 https://ipinfo.io/ip

# If no IPv6 available, disable it
sudo nano /etc/default/ufw
IPV6=no
sudo ufw reload

---

7. Install & Configure Fail2ban

1. Installation

   apt install fail2ban -y

2. Configure jail

# Fail2ban loads config files in this order:

/etc/fail2ban/jail.conf # default values (never change them)
/etc/fail2ban/jail.local # global config, overrides jail.conf
/etc/fail2ban/jail.d/< service >.local # modular, specific, overrides global values only for ssh

sudo nano /etc/fail2ban/jail.d/sshd.local

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = %(sshd_log)s
backend = systemd

maxretry = 3
findtime = 10m
bantime = 24h

banaction = iptables-multiport
banaction_allports = iptables-allports

3. Check status

   sudo fail2ban-client status sshd

   After changes reload

   sudo fail2ban-client reload sshd

   Or restart

   sudo systemctl restart fail2ban

---

8. Securely Connect GitHub Actions

1. Create GitHub deploy key (local)

   github-actions-hetzner

2. Add key to server

   nano /home/deploy/.ssh/authorized_keys

3. Restrict key (Least Privilege)

   command="/usr/bin/git-shell -c \"$SSH_ORIGINAL_COMMAND\"",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAA... github-actions@hetzner

4. Set permissions

   chmod 600 authorized_keys

   chown deploy:deploy authorized_keys

---

9. Application Setup (one‑time, manual)

• Login as deploy
• git clone
• npm install / npm ci
• Set .env
• Configure PM2
• Configure Nginx

➡️ Only during this phase are broad permissions allowed

---

10. Temporary Sudo Access (Setup Phase Only) ⚠️

WARNING: This is TEMPORARY and must be removed after deployment!

During initial app deployment, to avoid the clunky Hetzner console:

# As root (via Hetzner Console - ONCE)
sudo visudo

# Add this line at the end:
deploy ALL=(ALL) ALL

# Save and exit (Ctrl+X, Y, Enter)

✅ SET A REMINDER to remove this after all apps are deployed!

Better Alternative (Limited Sudo):

sudo visudo

# Add instead:
deploy ALL=(root) NOPASSWD: \
    /usr/bin/pm2 restart *, \
    /usr/bin/pm2 reload *, \
    /usr/bin/pm2 status, \
    /usr/bin/systemctl restart nginx, \
    /usr/bin/systemctl reload nginx, \
    /usr/bin/systemctl status nginx, \
    /usr/bin/nginx -t, \
    /usr/bin/certbot

---

11. Install Required Software

# Update system
sudo apt update && sudo apt upgrade -y

# Install Node.js (LTS via NodeSource)
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs

# Install PM2 globally
sudo npm install -g pm2

# Install Nginx
sudo apt install -y nginx

# Install Certbot for SSL
sudo apt install -y certbot python3-certbot-nginx

# Install Git (if not already installed)
sudo apt install -y git

# Verify installations
node --version
npm --version
pm2 --version
nginx -v
certbot --version

---

12. Directory Structure Setup

# Create app directories (as deploy user)
sudo mkdir -p /var/www/app
sudo chown -R deploy:deploy /var/www/app

# Create individual app folders
cd /var/www/app
mkdir -p myapp1.domain.com
mkdir -p myapp2.domain.com

---

13. GitHub SSH Key Setup (Per Server)

# As deploy user
ssh-keygen -t ed25519 -C "deploy@production-server"
# Press Enter for default location
# Press Enter twice for no passphrase (needed for automation)

# Display public key
cat ~/.ssh/id_ed25519.pub
# Copy this entire output

Add to GitHub:

1. GitHub.com → Settings → SSH and GPG keys
2. New SSH key → Paste key
3. Title: Production Server - [Date]

Test connection:

ssh -T git@github.com

⚠️ IMPORTANT: Delete old SSH keys from compromised servers!

---

14. Deploy Your First App

Clone Repository

cd /var/www/app/myapp.domain.com
git clone git@github.com:username/myapp.git .

Install Dependencies

# For backend
cd backend
npm install

# For frontend (if separate)
cd ../frontend
npm install

Configure Environment Variables

# Backend .env
cd /var/www/app/myapp.domain.com/backend
nano .env

Add your production variables:

NODE_ENV=production
MONGODB_URI=mongodb://...
JWT_SECRET=your-secret-here
PORT=5000

Secure the file:

chmod 600 .env
chown deploy:deploy .env

Frontend .env.production (if applicable):

cd /var/www/app/myapp.domain.com/frontend
nano .env.production

# Only non-secret values! These become PUBLIC in the bundle
VITE_API_URL=/api

---

15. Build Strategy

Option A: Build on Server (Small Apps)

cd /var/www/app/myapp.domain.com/frontend
npm run build

If you get permission errors:

# Fix binary permissions
chmod +x node_modules/.bin/vite
chmod +x node_modules/@esbuild/linux-x64/bin/esbuild

# Or nuclear option:
rm -rf node_modules package-lock.json
npm install
npm run build

Option B: Build Locally & Upload (Large Apps, Recommended)

On your local machine:

cd /path/to/myapp
npm run build

# Upload dist folder
scp -r frontend/dist/* deploy@SERVER_IP:/var/www/app/myapp.domain.com/frontend/dist/

Create deployment script (local machine):

# deploy-myapp.sh
#!/bin/bash
echo "🔨 Building..."
npm run build

echo "📤 Uploading to server..."
scp -r frontend/dist/* deploy@SERVER_IP:/var/www/app/myapp.domain.com/frontend/dist/

echo "🔄 Restarting app..."
ssh deploy@SERVER_IP "cd /var/www/app/myapp.domain.com/backend && pm2 restart myapp"

echo "✅ Deployment complete!"

chmod +x deploy-myapp.sh
./deploy-myapp.sh

---

16. Start App with PM2

cd /var/www/app/myapp.domain.com/backend
pm2 start npm --name "myapp" -- start

# Verify it's running
pm2 list
pm2 logs myapp --lines 50

# Save PM2 process list
pm2 save

# Setup PM2 to start on boot (FIRST TIME ONLY)
pm2 startup
# Copy and run the sudo command it outputs (as root or with sudo)

PM2 Management Commands:

pm2 list                   # Show all processes
pm2 restart myapp          # Restart app
pm2 stop myapp             # Stop app
pm2 logs myapp             # Show logs
pm2 logs myapp --lines 100 # Show more logs
pm2 delete myapp           # Remove from PM2
pm2 save                   # Save current process list

---

17. Configure Nginx

Backend-Only App (Node.js API, Next.js)

sudo nano /etc/nginx/sites-available/myapp.domain.com

server {
    listen 80;
    server_name myapp.domain.com;

    location / {
        proxy_pass http://localhost:5000;  # Your app's port
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }
}

Frontend + Backend (SPA with API)

server {
    listen 80;
    server_name myapp.domain.com;

    # Serve static frontend
    root /var/www/app/myapp.domain.com/frontend/dist;
    index index.html;

    # Frontend routes (SPA)
    location / {
        try_files $uri $uri/ /index.html;
    }

    # API proxy to backend
    location /api/ {
        proxy_pass http://localhost:5000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }
}

Enable the Site

# Create symlink
sudo ln -s /etc/nginx/sites-available/myapp.domain.com /etc/nginx/sites-enabled/myapp.domain.com

# Test configuration
sudo nginx -t

# Should output:
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful

# Reload nginx
sudo systemctl reload nginx

# Check status
sudo systemctl status nginx

Common Nginx Issues:

❌ Bad symlink:

# Wrong:
ln -s /etc/nginx/sites-available/myapp /etc/nginx/sites-enabled/sites-available-myapp

# Correct:
ln -s /etc/nginx/sites-available/myapp.domain.com /etc/nginx/sites-enabled/myapp.domain.com

❌ Config errors:

# Always test before reload:
sudo nginx -t

# If errors, check:
sudo nginx -t 2>&1 | grep -i error

---

18. DNS Configuration (Namecheap)

1. Go to Namecheap Dashboard

2. Select Domain → Advanced DNS

3. Add A Record:

   For subdomain: myapp.domain.com → Host = myapp

   For root domain: domain.com → Host = @

   Type	Host	Value	TTL
   A Record	myapp	YOUR_SERVER_IP	Automatic

4. Wait 5-15 minutes for DNS propagation

   Check DNS propagation:

# From server or local machine
nslookup myapp.domain.com 8.8.8.8
# or
host myapp.domain.com 8.8.8.8
# or
dig @8.8.8.8 myapp.domain.com

# Or use online tool:
# https://dnschecker.org

Flush local DNS cache (if needed):

# Windows
ipconfig /flushdns

# Mac
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

# Linux
sudo systemd-resolve --flush-caches

---

19. SSL/HTTPS with Let’s Encrypt

Allow HTTPS through firewall:

sudo ufw allow 'Nginx Full'
sudo ufw status

Get SSL certificate:

sudo certbot --nginx -d myapp.domain.com

# Follow prompts:
# 1. Enter email address
# 2. Agree to terms of service
# 3. Choose whether to redirect HTTP to HTTPS (choose YES/2)

Certbot automatically:

• ✅ Gets certificate from Let’s Encrypt
• ✅ Modifies nginx config
• ✅ Sets up auto-renewal

Verify certificate:

sudo certbot certificates

Test auto-renewal:

sudo certbot renew --dry-run

Manual renewal (if needed):

sudo certbot renew
sudo systemctl reload nginx

Check nginx config after certbot:

sudo cat /etc/nginx/sites-available/myapp.domain.com

Should now include SSL configuration:

server {
    listen 443 ssl;
    server_name myapp.domain.com;
    
    ssl_certificate /etc/letsencrypt/live/myapp.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/myapp.domain.com/privkey.pem;
    # ... rest of config
}

server {
    listen 80;
    server_name myapp.domain.com;
    return 301 https://$server_name$request_uri;
}

---

20. Update Nginx Config With Extended Security

sudo nano /etc/nginx/sites-available/myapp.domain.com

From the original nginx config, only two lines created by certbot are needed. Beginning with:

ssl_certificate… and ssl_certificate_key…

Backend Only App (Node.js API, Next.js)

Replace “my-app” with custom app name and “my-domain.de” with custom domain

# Rate limiting (add at the TOP of the file, outside server blocks)
limit_req_zone $binary_remote_addr zone=my-app_limit:10m rate=10r/s;

# HTTP to HTTPS redirect (Certbot creates this)
server {
    listen 80;
    server_name my-domain.de;
    return 301 https://$server_name$request_uri;
}

# HTTPS Server (Certbot creates the SSL lines)
server {
    listen 443 ssl http2;
    server_name my-domain.de;

    # SSL Certificate (ONLY keep these 2 lines from certbot)
    ssl_certificate /etc/letsencrypt/live/my-domain.de/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/my-domain.de/privkey.pem;
    
    # Add these SSL settings manually
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    # Security basics
    server_tokens off;
    client_max_body_size 5M;
    
    # Rate limiting
    limit_req zone=my-app_limit burst=20 nodelay;
    limit_req_status 429;

    location / {
        proxy_pass http://localhost:5000;
        proxy_http_version 1.1;
        
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        
        # Security headers
        add_header X-Frame-Options "DENY" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header X-XSS-Protection "1; mode=block" always;
        add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    }
}

Frontend + Backend (SPA with API)

Replace “my-app” with custom app name and “my-domain.de” with custom domain

# Rate limiting
limit_req_zone $binary_remote_addr zone=my-app_limit:10m rate=10r/s;

# HTTP to HTTPS redirect
server {
    listen 80;
    server_name my-domain.de;
    return 301 https://$server_name$request_uri;
}

# HTTPS Server
server {
    listen 443 ssl http2;
    server_name my-domain.de;

    # SSL Certificate (ONLY keep these 2 lines from certbot)
    ssl_certificate /etc/letsencrypt/live/my-domain.de/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/my-domain.de/privkey.pem;
    
    # Your SSL settings (these replace certbot's include file)
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    # Security basics
    server_tokens off;
    client_max_body_size 5M;
    
    # Rate limiting
    limit_req zone=my-app_limit burst=20 nodelay;
    limit_req_status 429;

    # Serve static frontend
    root /var/www/app/my-domain.de/frontend/dist;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
        
        add_header X-Frame-Options "DENY" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header X-XSS-Protection "1; mode=block" always;
        add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    }

    location /api/ {
        proxy_pass http://localhost:5000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }
}

sudo nginx -t

sudo systemctl reload nginx

---

21. Verify Everything Works

# Check PM2 processes
pm2 list

# Check nginx
sudo systemctl status nginx

# Check SSL
sudo certbot certificates

# Check firewall
sudo ufw status

# Check app logs
pm2 logs myapp --lines 50

# Check nginx access logs
sudo tail -f /var/log/nginx/access.log

# Check nginx error logs
sudo tail -f /var/log/nginx/error.log

Test in browser:

1. http://myapp.domain.com → Should redirect to HTTPS
2. https://myapp.domain.com → Should show your app
3. Check browser console for errors
4. Check Network tab – API calls should work

---

22. Deploy Additional Apps

For each new app, repeat:

1. Create directory: /var/www/app/app2.domain.com
2. Clone repo: git clone ...
3. Install dependencies: npm install
4. Configure .env
5. Build (if frontend)
6. Start with PM2 (use unique name and port)
7. Create nginx config (unique server_name and port)
8. Configure DNS (new A record)
9. Get SSL certificate: certbot --nginx -d app2.domain.com

Port allocation:

• App 1: 5000
• App 2: 5001
• App 3: 5002
• etc.

---

23. REMOVE TEMPORARY SUDO ACCESS ⚠️

After all apps are deployed and working:

sudo visudo

# Remove or comment out this line:
# deploy ALL=(ALL) ALL

# Save and exit

Verify deploy can’t use sudo anymore:

sudo ls
# Should show: deploy is not in the sudoers file

From now on:

• System changes → Use Hetzner Console (as root)
• App deployments → SSH as deploy (no sudo needed)

---

24. Monitoring & Maintenance

Daily Checks

pm2 list                          # All apps running?
sudo systemctl status nginx       # Nginx ok?
df -h                             # Disk space ok?
free -h                           # Memory ok?

Weekly Checks

sudo apt update && sudo apt upgrade -y  # System updates
pm2 logs --lines 100                    # Check for errors
sudo tail -100 /var/log/auth.log        # Check for intrusions

Monthly Checks

sudo certbot certificates         # SSL expiry (should auto-renew)
sudo fail2ban-client status sshd  # Check banned IPs

---

25. Deployment Workflow (After Initial Setup)

For Code Updates:

Option A: Manual

# SSH to server as deploy
ssh deploy@SERVER_IP
cd /var/www/app/myapp.domain.com
git pull origin main
cd backend && npm install  # If package.json changed
pm2 restart myapp

Option B: Local Build + Upload

# On local machine
./deploy-myapp.sh  # Your deployment script

Option C: GitHub Actions (See section 8 in your existing docs)

For Nginx Config Changes:

# Via Hetzner Console (as root)
nano /etc/nginx/sites-available/myapp.domain.com
nginx -t
systemctl reload nginx

For Environment Variable Changes:

# As deploy
nano /var/www/app/myapp.domain.com/backend/.env
pm2 restart myapp

---

Troubleshooting Common Issues

App Not Accessible

# Check if app is running
pm2 list
pm2 logs myapp --lines 50

# Check if port is listening
sudo netstat -tlnp | grep :5000

# Check nginx config
sudo nginx -t
sudo systemctl status nginx

# Check firewall
sudo ufw status

502 Bad Gateway

• ❌ App not running → pm2 restart myapp
• ❌ Wrong port in nginx config → Check proxy_pass matches app port
• ❌ App crashed → Check pm2 logs myapp

SSL Certificate Issues

sudo certbot renew
sudo systemctl reload nginx

Permission Denied Errors

# Check file ownership
ls -la /var/www/app/myapp.domain.com

# Should be owned by deploy:deploy or deploy:www-data
sudo chown -R deploy:deploy /var/www/app/myapp.domain.com

---

Security Checklist (Post-Deployment)

• deploy ALL=(ALL) ALL removed from sudoers ⚠️
• All .env files are chmod 600
• Old SSH keys from compromised servers removed from GitHub
• All apps using HTTPS (not HTTP)
• UFW only allows ports 22, 80, 443
• Fail2ban is active and monitoring
• PM2 startup script configured
• Certbot auto-renewal tested
• No passwords in git repos
• MongoDB not publicly accessible
• Strong JWT secrets in use

---

Final Notes

✅ DO:

• Keep system updated: sudo apt update && sudo apt upgrade
• Monitor logs regularly: pm2 logs, nginx logs
• Use deployment scripts/GitHub Actions
• Test nginx config before reload: nginx -t
• Keep .env files secure

❌ DON’T:

• Leave deploy ALL=(ALL) ALL in sudoers permanently
• Commit .env files to git
• Run apps as root
• Expose MongoDB/databases publicly
• Reuse compromised SSH keys
• Skip nginx -t before reload

When in doubt: Use Hetzner Console as root for system changes, SSH as deploy for app changes.

Target State

• ✅ New, clean server
• ✅ No passwords, only SSH keys
• ✅ Root login disabled
• ✅ Fail2ban active
• ✅ CI/CD without shell access
• ✅ Clear separation: setup vs. operations

---

Mnemonic:

If a server is compromised, it gets replaced — not repaired.