dev-resources.site

for different kinds of informations.

How to recover and update Proxmox 8 firewall configuration in SQLite when you locked yourself out

Published at

11/16/2024

TLDR

The firewall config is not in /etc/pve/firewall/cluster.fw but in a SQLite Database in /var/lib/pve-cluster/config.db. You need to reboot your system into rescue mode, edit the value enable: 1 to enable: 0 and reboot into Proxmox.

Context

I made a noob mistake and locked myself out of my server. Luckily Hetzner allows me to reboot into rescue mode. This is what happened and how I managed to get my access back.

In other words, this tutorial is for situations where you've accidentally locked yourself out of your Proxmox server due to a firewall misconfiguration (like I did). In my case, I enabled the firewall (enable: 1) with an incorrect configuration, preventing access to the server. The solution involves booting into a rescue system, mounting the Proxmox partition, and manually editing the firewall configuration in the SQLite database.

Prerequisites

Access to a rescue system (e.g., Hetzner Rescue System)
Basic knowledge of Linux commands and SQLite, although you can copy and paste these commands and it should work.

Disclaimer : I am not responsible for data loss or anything else for that matter. The following commands worked for me and nothing bad happened. I out them here in case they help someone else, as I had to research a few hour before solving this (specially the issue of not finding the config).

Step 1: Boot into Rescue System

Boot your server into the rescue system provided by your hosting provider (e.g., Hetzner Rescue System).

Step 2: Identify the Proxmox Partition

Use the lsblk command to list all block devices:

lsblk

Identify the partition where Proxmox is installed. It's often part of a RAID array or LVM setup.

In my case the output was like this:

loop07:003.1G1 loop
nvme1n1259:00 476.9G0 disk
├─nvme1n1p1259:10256M0 part
│ └─md09:00 255.9M0 raid1 
├─nvme1n1p2259:201G0 part
│ └─md19:101022M0 raid1 
└─nvme1n1p3259:30 475.7G0 part
└─md29:20 475.6G0 raid1 
├─vg0-root 253:0064G0 lvm
├─vg0-swap 253:108G0 lvm
└─vg0-data 253:20402G0 lvm
nvme0n1259:40 476.9G0 disk
├─nvme0n1p1259:50256M0 part
│ └─md09:00 255.9M0 raid1 
├─nvme0n1p2259:601G0 part
│ └─md19:101022M0 raid1 
└─nvme0n1p3259:70 475.7G0 part
└─md29:20 475.6G0 raid1 
├─vg0-root 253:0064G0 lvm
├─vg0-swap 253:108G0 lvm
└─vg0-data 253:20402G0 lvm

There I saw that I should mount vg0, and that is was in a raid md2

Step 3: Assemble RAID Array (if applicable)

If your Proxmox partition is part of a RAID array, assemble it:

mdadm --assemble --scan

Step 4: Activate Volume Group

Activate the volume group (usually named vg0 in Proxmox):

vgchange -ay vg0

Step 5: Mount the Proxmox Partition

Create a mount point and mount the Proxmox root partition:

mkdir /mnt/proxmox
mount /dev/vg0/root /mnt/proxmox

Verify the mount:

ls /mnt/proxmox/

Here you should see some files and directories.

Step 6: Locate the Configuration Database

The Proxmox configuration is stored in an SQLite database. Locate it:

ls -la /mnt/proxmox/var/lib/pve-cluster

You should see a file named config.db.

Step 7: Access the SQLite Database

Open the SQLite database:

sqlite3 /mnt/proxmox/var/lib/pve-cluster/config.db

sqlite3 is already installed in the rescue system of Hetzner. You need to install it if it's not available in your system.

Step 8: Check the Current Firewall Configuration

View the current firewall configuration:

SELECT \* FROM tree WHERE name = 'cluster.fw';

Note : Initially I didn't know where this was, so I used the following to find where the entry was and if there was any.

SELECT \* FROM tree WHERE name = 'cluster.fw';

Step 9: Update the `enable` Option

Change the enable option from 1 to 0 to disable the firewall:

UPDATE tree 
SET data = replace(data, 'enable: 1', 'enable: 0') 
WHERE name = 'cluster.fw';

Step 10: Verify the Change

Confirm that the change was made successfully:

SELECT \* FROM tree WHERE name = 'cluster.fw';

Step 11: Exit SQLite

Exit the SQLite prompt:

.quit

Step 12: Unmount and Reboot

Unmount the Proxmox partition and reboot the server:

umount /mnt/proxmox
reboot

Important Notes

Disabling the Firewall: This process disables the firewall cluster-wide. Re-enable it after properly configuring it once you regain access.
Security Risks: A disabled firewall may expose your system to security risks. You have been warned.
Backup: Always create backups before making significant changes. I have my proxmox configs in a git repository for reference.
Alternative Methods: When possible, use the Proxmox web interface or CLI tools for configuration changes. At least that's what I've read. I like to use config files, but I also locked myself out of my server.

References

Several sites, but I cannot longer remember all of them.

Some of the sites I visited are:

firewall Article's

30 articles in total

Configuring network access with Cisco ASA via minicom utility