Container fails to open a SQLite3 file

diasdm · December 24, 2024, 11:36pm

I recently ran a Zabbix Proxy (7.0) container on my HAPax3 router (rOS 7.16.2). Then decided to update the container (to 7.2), so I removed the old container and added the new one with the new tag version. Now it won’t start anymore.
I tried to add the old version back, but it fails as well.

The container was set with the following configuration:

/container mounts
add name=zabbix_proxy_db  src=/usb1/zabbix_proxy/db_data dst=/var/lib/zabbix/db_data comment="Zabbix Proxy container SQLite3 database volume"
add name=zabbix_proxy_tls src=/usb1/zabbix_proxy/enc     dst=/var/lib/zabbix/enc     comment="Zabbix Proxy container TLS encryption volume"

/file
add type=directory name=usb1/zabbix_proxy
add type=file      name=usb1/zabbix_proxy/enc/zabbix_proxy.psk contents="key"

/container envs
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_HOSTNAME"              key=ZBX_HOSTNAME              value=mikrotik
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_SERVER_HOST"           key=ZBX_SERVER_HOST           value=[zabbix-server]
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_PROXYMODE"             key=ZBX_PROXYMODE             value=0
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_TIMEOUT"               key=ZBX_TIMEOUT               value=4
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_ENABLEREMOTECOMMANDS"  key=ZBX_ENABLEREMOTECOMMANDS  value=1
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_PROXYOFFLINEBUFFER"    key=ZBX_PROXYOFFLINEBUFFER    value=720
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_STARTVMWARECOLLECTORS" key=ZBX_STARTVMWARECOLLECTORS value=1
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_TLSCONNECT"            key=ZBX_TLSCONNECT            value=psk
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_TLSPSKFILE"            key=ZBX_TLSPSKFILE            value=/var/lib/zabbix/enc/zabbix_proxy.psk
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_TLSPSKIDENTITY"        key=ZBX_TLSPSKIDENTITY        value=[id]
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_PROXYCONFIGFREQUENCY"  key=ZBX_PROXYCONFIGFREQUENCY  value=60
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_PROXYBUFFERMODE"       key=ZBX_PROXYBUFFERMODE       value=hybrid
add name=Zabbix_Proxy_Env comment="Zabbix Proxy container env ZBX_PROXYMEMORYBUFFERSIZE" key=ZBX_PROXYMEMORYBUFFERSIZE value=8M

/container
add remote-image="zabbix/zabbix-proxy-sqlite3:ol-7.2-latest" interface=veth3_zproxy hostname=zproxy domain-name=ierece.diasdm.com.br dns=192.168.8.5 envlist=Zabbix_Proxy_Env mounts=zabbix_proxy_db,zabbix_proxy_tls root-dir=usb1/zabbix start-on-boot=yes workdir=/var/lib/zabbix comment="Zabbix Proxy container"

When I start the container, it stops right after.
I enabled the “Logging” option and found that the proxy fails to initialize because it cannot open the SQLite3 database.

20:13:05 container,info,debug      1:20241224:231305.579 Starting Zabbix Proxy (active) [mikrotik]. Zabbix 7.2.1 (revision 67f3c83).
20:13:05 container,info,debug      1:20241224:231305.579 **** Enabled features ****
20:13:05 container,info,debug      1:20241224:231305.579 SNMP monitoring:       YES
20:13:05 container,info,debug      1:20241224:231305.579 IPMI monitoring:       YES
20:13:05 container,info,debug      1:20241224:231305.579 Web monitoring:        YES
20:13:05 container,info,debug      1:20241224:231305.579 VMware monitoring:     YES
20:13:05 container,info,debug      1:20241224:231305.579 ODBC:                  YES
20:13:05 container,info,debug      1:20241224:231305.579 SSH support:           YES
20:13:05 container,info,debug      1:20241224:231305.579 IPv6 support:          YES
20:13:05 container,info,debug      1:20241224:231305.579 TLS support:           YES
20:13:05 container,info,debug      1:20241224:231305.579 **************************
20:13:05 container,info,debug      1:20241224:231305.579 using configuration file: /etc/zabbix/zabbix_proxy.conf
20:13:05 container,info,debug      1:20241224:231305.580 cannot open database file "/var/lib/zabbix/db_data/mikrotik.sqlite": [2] No such file or directory
20:13:05 container,info,debug      1:20241224:231305.580 creating database ...
20:13:05 container,info,debug      1:20241224:231305.580 [Z3001] connection to database '/var/lib/zabbix/db_data/mikrotik.sqlite' failed: [0] unable to open atabase file
20:13:05 container,info,debug      1:20241224:231305.580 cannot initialize database: cannot open database

The actual SQLite3 file (/var/lib/zabbix/db_data/mikrotik.sqlite) should be created dynamically when the proxy starts, as it is not a permanent database.
Usually, when the Zabbix Proxy fails to open the database file, it is due to lack of permissions in the directory. However, I could not find any permission configuration for the target directory (/usb1/zabbix_proxy/db_data).

This is strange because the database volume should be writeable in the container.
So, I’m not sure why it is failing to open the SQLite file.

Any thoughts on this scenario?

optio · December 25, 2024, 11:28am

You can temporally set container config cmd=“tail -f /dev/null”, this should start container without starting Zabbix and then from running container shell you can examine what’s the issue with fs permissions.

diasdm · December 26, 2024, 10:23pm

I followed your suggestion and started the container with the “tail” command. This allowed me to access the shell of the container.
Doing this seems to overwrite the entrypoint script though.
However, there seems to be nothing wrong. The database volume should be user-writable.

bash-5.1# ls -lah /var/lib/zabbix/
drwxrwxr-x  2 zabbix root 4.0K Dec 26 00:54 db_data

To start the proxy from within the container, I exported some variables so that the Proxy binary could read them, but got the same error.
The SQLite3 database file is not created despite the “creating database” message.

# export ZBX_SERVER_HOST=[zabbix.server]
# export ZBX_DB_NAME=/var/lib/zabbix/db_data/mikrotik.sqlite
# zabbix_proxy -f

Starting Zabbix Proxy (active) [zproxy]. Zabbix 7.2.1 (revision 67f3c83).
Press Ctrl+C to exit.

    11:20241226:012832.550 Starting Zabbix Proxy (active) [zproxy]. Zabbix 7.2.1 (revision 67f3c83).
    11:20241226:012832.550 **** Enabled features ****
    11:20241226:012832.550 SNMP monitoring:       YES
    11:20241226:012832.550 IPMI monitoring:       YES
    11:20241226:012832.550 Web monitoring:        YES
    11:20241226:012832.550 VMware monitoring:     YES
    11:20241226:012832.550 ODBC:                  YES
    11:20241226:012832.550 SSH support:           YES
    11:20241226:012832.550 IPv6 support:          YES
    11:20241226:012832.550 TLS support:           YES
    11:20241226:012832.550 **************************
    11:20241226:012832.550 using configuration file: /etc/zabbix/zabbix_proxy.conf
    11:20241226:012832.551 cannot open database file "/var/lib/zabbix/db_data/mikrotik.sqlite": [2] No such file or directory
    11:20241226:012832.551 creating database ...
    11:20241226:012832.551 [Z3001] connection to database '/var/lib/zabbix/db_data/mikrotik.sqlite' failed: [0] unable to open database file
    11:20241226:012832.551 cannot initialize database: cannot open database

I tested with different Zabbix Proxy versions, including older ones, but they all fail in the same way.
I even tried to create the “mikrotik.sqlite” file manually, but the Proxy sees that it “is not a Zabbix database”.

I think there might be something wrong with the mounts configured, but I just can’t see what it is.

tangent · December 30, 2024, 12:35am

Try “touch”ing a file in that directory.

If it fails, there’s your problem, reduced to a simpler level than SQLite.

If not, is there a sqlite3 binary in that image? If so, use it to try creating a DB in that directory. A simple query like “CREATE TABLE a(a)” will force a file write.