When the updater container is gracefully stopped (by docker or kubernetes), it should remove the lock/update_daemon.lock file, so that another instance of this container can continue the updates.
It’s currently not always the case. I suppose it happens when you stop the container during an update.
This happens to me under kubernetes (if the corresponding pod is gracefully killed by k8s during an update, for any good reason), but the same issue probably affects the updater when run by docker-compose (I did not check).
I see there is already code to handle that in update_daemon2.php. I suppose this could be fixed by handling process signals in updater.sh, but I’m not 100% sure.
unless i’m missing something here lockfile simply being there shouldn’t prevent updates, unless it is flock()ed. if it’s not, updater assumes it crashed or something and goes on updating.
I slightly disagree on the vision of k8s.
It’s true that tt-rss is not cloud-native, and that it needs persistence.
It’s also true that k8s has been initially designed for stateless workloads.
However, k8s can bring many advantages over docker-compose (at least ability to spread pods among several nodes), and works fine with stateful workloads and/or non-cloud-native apps, as long as you keep only one replica for each deployment, and don’t expect to have all the nicest k8s advantages (like zero-downtime if you stop a pod or a node, ability to scale to handle many requests etc)
I currently self-host many things on a home k8s cluster (actually, it’s k3s, but it does not matter), with all data (PVs) on an NFS server. Wordpress, Nextcloud, Kanboard, Matomo, Piwigo (to cite only the ones that use a similar language/architecture): none of them are cloud-native, they are clearly stateful, and most of them even need to put the PHP files on NFS.
Nextcloud, for example, provides a Helm chart.
However, my problem indeed probably comes from NFS. In this case, it’s highly probable that I would have the same problem with docker-compose+NFS, but I did not check.
(I wanted to post a new message to give ideas I have to make a graceful shutdown remove the lock files, but I don’t have the right to post to 2 successive messages here, yet)
around here (speaking as a senior infrastructure admin) first thing we ask from developer teams when they want to migrate their stuff into k8s is whether it has any persistence. if they do, we tell them to come back after they remove it.
you can of course run whatever you want, however you want, in your homelab cluster, but you’re getting used to objective antipatterns which are not used anywhere in the real world.
and if you come anywhere near any semi-competent devops team suggesting to use NFS of all things (to store PHP files on, oh god…) you’ll get laughed out of the building.
exactly. that was me, a year ago, playing around with my homelab cluster while having zero understanding of how those things are actually used. a lot has changed since then. many things i had to unlearn.
p.s. even way back then i wouldn’t ever consider using NFS for PVs unless this was an actual disposable lab environment.
any issue with the NFS server makes disk i/o for the containers indefinitely stuck, which could affect your entire cluster.
also, in general, trying to force stateful services into k8s is a bad idea. i’m running tt-rss on k8s but it is effectively stateless. if your services require PVCs you’re objectively doing k8s wrong. if you need ReadWriteMany you’re doing it wrong times 10.
p.s. at work we do not allow PVs at all. you can have an emptyDir, that’s it.
Thanks. I’m running docker but I guess it’s the same problem. But at the end of the day services need some sort of persistence, how do you handle it?
In my case I’ll most probably just create a disk device that points to a zvol (using TrueNAS) instead of mounting datasets through NFS.
if it needs persistence or access to file storage, it doesn’t go on k8s.
a lot of times though persistence is not actually needed. i.e. take some kind of garden variety php fpm + nginx app like roundcube. it needs common filesystem between fpm and nginx containers so that nginx can serve static files, but does it actually need a PVC?
well, it doesn’t. you can bake everything you need to run it (i.e. local plugins) into container image and share an emptydir between nginx and fpm. then you can scale out using replicas which would provide proper HA in case one of your k8s worker nodes goes down (if you have cluster ingress properly setup, i’m using a rather simple consul + haproxy + istio setup).
it’s not perfect nor it is cloud native but it’s better than trying to bring shared storage into k8s for no reason or deal with docker compose on multiple hosts.
for tt-rss deployment i have an init container which pulls local plugins from S3 onto shared emptydir on startup. then with cache_s3 plugin tt-rss doesn’t need anything on local filesystem and scales normally using replicasets.
add argo cd and vault and you have a mostly automatic highly available way to run almost anything you need with pretty much zero bullshit. you also get free rolling rollouts as opposed to “watchtower decided to restart this thing while you were using it”.
p.s. there’s a helm chart for jellyfin. an application that has no ability to scale nor function without direct access to a huge library of media files. its existence proves that some people just don’t understand jack shit about k8s.
Thanks, good info. I guess when it comes to databases you’d use a database server not hosted on kubernetes, docker or something like that, but rather on a dedicated server.
i’m running a small three VM (on separate baremetal hypervisors) patroni cluster which all apps use, both on k8s and otherwise (for a real prod deployment you’d want separate DCS for patroni, so six VM).
in general my goal is to have stuff mostly working if one baremetal host is turned off for whatever reason.
so there’s a lot of threes - three consul nodes, three k3s control nodes with haproxy, three vault nodes, three load balancers for out-of-k8s stuff, you get the idea. consul cluster provides redundant entrypoints for everything because it’s essentially a dns server with active healthchecks.
