How to Set Up a Linux Server from Scratch: A Practical Walkthrough

Most "how to set up a Linux server" guides walk you through the commands without explaining why you're running them. This one tries to do both. We'll go from a fresh Ubuntu 24.04 server to a locked-down, monitored, production-ready machine—and explain what each step is actually doing.

We're assuming you just spun up a VPS (DigitalOcean, Linode, Hetzner, wherever), you have root access via SSH, and you've never done this before or want a clean reference. Everything here applies equally to bare metal.

Step 1: First Login and Immediate Hygiene

When you first SSH in as root, you're on borrowed time. The server is already being scanned by bots, and root with password auth is a target. The first thing to do is change the root password to something strong—or better, disable password auth entirely once you have a key set up.

# Check who's already tried to log in (you might be surprised)
cat /var/log/auth.log | grep "Failed password" | tail -20

On a fresh server that's been up for 10 minutes, you'll often already see hundreds of failed SSH attempts. This isn't a crisis—it's just bots doing their thing—but it's a good reminder that hardening SSH is not optional.

Step 2: Create a Non-Root User

Running everything as root is dangerous because mistakes have no safety net. Create a user for your actual work:

adduser deploy
usermod -aG sudo deploy

Now copy your SSH public key to the new user. From your local machine:

ssh-copy-id deploy@your-server-ip

Test that you can log in as the new user before doing anything else. Do not lock yourself out.

Step 3: Harden SSH

Edit /etc/ssh/sshd_config. The changes that matter most:

PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
Port 22  # Consider changing to a non-standard port to reduce log noise

Disabling password authentication is the single most effective security change you can make. Without it, brute-force attacks against your server are essentially impossible. After saving, restart SSH—but keep your current session open until you confirm the new session works.

sudo systemctl restart sshd

Step 4: Configure the Firewall

UFW (Uncomplicated Firewall) is the easiest way to manage iptables rules on Ubuntu. The default should be deny everything, then explicitly allow what you need:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh        # or your custom SSH port
sudo ufw allow 80/tcp     # HTTP
sudo ufw allow 443/tcp    # HTTPS
sudo ufw enable

Check the status with sudo ufw status verbose. If you changed the SSH port, make sure you allow the new port before enabling the firewall—or you'll lock yourself out.

Step 5: Keep Software Updated

Unpatched software is how most servers get compromised. Configure automatic security updates:

sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

This automatically applies security updates. You'll still want to handle major version upgrades manually, but security patches will be applied without you having to think about it.

Step 6: Fail2ban for Intrusion Prevention

Even with password auth disabled, you might want to reduce noise. Fail2ban monitors log files and temporarily bans IPs that show signs of attack:

sudo apt install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

The default configuration protects SSH. You can extend it to protect web applications, mail servers, and anything else that writes auth logs.

Step 7: Monitoring and Alerts

A server you can't see is a server you can't trust. At minimum, set up something that tells you when the server is down or behaving unexpectedly. For simple needs, a free uptime monitoring service (UptimeRobot, BetterStack) pinging your server is enough. For production workloads, consider a proper monitoring stack.

Also worth installing: htop for process monitoring, ncdu for disk usage, and logwatch for daily log summaries.

Common Mistakes

A few things that trip people up:

  • Locking yourself out of SSH. Always test the new SSH configuration from a second terminal window before closing the first. If something goes wrong, you still have access.
  • Forgetting to open the firewall for your application. You set up a web server and the site doesn't load—the firewall is probably blocking port 80 or 443.
  • Running everything as root. It works until it doesn't. A bug in your deployment script becomes a lot less catastrophic when it runs as a limited user.
  • No backups. Snapshots are not backups. Set up actual off-server backups before you put anything important on the machine.

That's the foundation. From here you'd install your application stack—Nginx, Node.js, Python, whatever you need—but the server underneath is solid. Security isn't a one-time setup; it's something you revisit as your setup evolves. But starting from this baseline puts you in a much better position than the default fresh install.