Обсуждение: PostgreSQL - HA Cluster / Etcd Issue
Hello,
In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
Although many things in this architecture work as expected, Etcd encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once every 2 or 3 months. The root cause is that the database size reaches the specified limit and Etcd does not clean it up through rotation. I am aware of this, but despite reviewing the maintenance documentation, I could not find the most appropriate parameter values for this rotation. I am sharing an example of my own Etcd configuration below. I kindly ask you to share your experience on this matter. I am kinda stuck on this issue.
In particular, is it possible to schedule the parameter that performs this cleanup with a time setting? For example, can I tell it to run the compact operation at around 01:00 AM?
In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
Although many things in this architecture work as expected, Etcd encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once every 2 or 3 months. The root cause is that the database size reaches the specified limit and Etcd does not clean it up through rotation. I am aware of this, but despite reviewing the maintenance documentation, I could not find the most appropriate parameter values for this rotation. I am sharing an example of my own Etcd configuration below. I kindly ask you to share your experience on this matter. I am kinda stuck on this issue.
In particular, is it possible to schedule the parameter that performs this cleanup with a time setting? For example, can I tell it to run the compact operation at around 01:00 AM?
Best regards.
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
[Service]
User=etcd
Type=notify
ExecStart=/usr/local/sbin/etcd \
--name etcd1 \
--data-dir=/etc/etcd \
--initial-advertise-peer-urls=http://10.115.208.168:2380 \
--listen-peer-urls=http://192.168.1.1:2380 \
--listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
--advertise-client-urls=192.168.1.1:2379 \
--initial-cluster-token="etcd-cluster" \
--initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=http://192.168.1.2:2380,etcd3=http://3:2380" \
--initial-cluster-state="new" \
--log-level="error"
[Install]
WantedBy=multi-user.target
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
[Service]
User=etcd
Type=notify
ExecStart=/usr/local/sbin/etcd \
--name etcd1 \
--data-dir=/etc/etcd \
--initial-advertise-peer-urls=http://10.115.208.168:2380 \
--listen-peer-urls=http://192.168.1.1:2380 \
--listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
--advertise-client-urls=192.168.1.1:2379 \
--initial-cluster-token="etcd-cluster" \
--initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=http://192.168.1.2:2380,etcd3=http://3:2380" \
--initial-cluster-state="new" \
--log-level="error"
[Install]
WantedBy=multi-user.target
This is a PostgreSQL mailing list, and your question has nothing to do with PostgreSQL. -- Scott Ribe scott_ribe@elevated-dev.com https://www.linkedin.com/in/scottribe/ > On Aug 20, 2025, at 1:40 AM, SOzcn <selahattinozcnma@gmail.com> wrote: > > Hello, > > In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived. > > Although many things in this architecture work as expected, Etcd encounters the error "memberID:5960773920904691185 alarm:NOSPACE"once every 2 or 3 months. The root cause is that the database size reaches the specified limit and Etcd doesnot clean it up through rotation. I am aware of this, but despite reviewing the maintenance documentation, I could notfind the most appropriate parameter values for this rotation. I am sharing an example of my own Etcd configuration below.I kindly ask you to share your experience on this matter. I am kinda stuck on this issue. > > In particular, is it possible to schedule the parameter that performs this cleanup with a time setting? For example, canI tell it to run the compact operation at around 01:00 AM? > > Best regards. > > > [Unit] > Description=etcd service > Documentation=https://github.com/etcd-io/etcd > After=network.target > > [Service] > User=etcd > Type=notify > ExecStart=/usr/local/sbin/etcd \ > --name etcd1 \ > --data-dir=/etc/etcd \ > --initial-advertise-peer-urls=http://10.115.208.168:2380 \ > --listen-peer-urls=http://192.168.1.1:2380 \ > --listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \ > --advertise-client-urls=192.168.1.1:2379 \ > --initial-cluster-token="etcd-cluster" \ > --initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=http://192.168.1.2:2380,etcd3=http://3:2380" \ > --initial-cluster-state="new" \ > --log-level="error" > > [Install] > WantedBy=multi-user.target
This is high availability environment and etcd is one management tools of them . Many PostgreSQL managed by this architec. I advice to you research. Etcs - Patrono - Postgresql- Haproxy- Keepalived. Anyway if you doesn't have experience of it, that's ok.
20 Ağu 2025 Çar 15:46 tarihinde Scott Ribe <scott_ribe@elevated-dev.com> şunu yazdı:
This is a PostgreSQL mailing list, and your question has nothing to do with PostgreSQL.
--
Scott Ribe
scott_ribe@elevated-dev.com
https://www.linkedin.com/in/scottribe/
> On Aug 20, 2025, at 1:40 AM, SOzcn <selahattinozcnma@gmail.com> wrote:
>
> Hello,
>
> In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
>
> Although many things in this architecture work as expected, Etcd encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once every 2 or 3 months. The root cause is that the database size reaches the specified limit and Etcd does not clean it up through rotation. I am aware of this, but despite reviewing the maintenance documentation, I could not find the most appropriate parameter values for this rotation. I am sharing an example of my own Etcd configuration below. I kindly ask you to share your experience on this matter. I am kinda stuck on this issue.
>
> In particular, is it possible to schedule the parameter that performs this cleanup with a time setting? For example, can I tell it to run the compact operation at around 01:00 AM?
>
> Best regards.
>
>
> [Unit]
> Description=etcd service
> Documentation=https://github.com/etcd-io/etcd
> After=network.target
>
> [Service]
> User=etcd
> Type=notify
> ExecStart=/usr/local/sbin/etcd \
> --name etcd1 \
> --data-dir=/etc/etcd \
> --initial-advertise-peer-urls=http://10.115.208.168:2380 \
> --listen-peer-urls=http://192.168.1.1:2380 \
> --listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
> --advertise-client-urls=192.168.1.1:2379 \
> --initial-cluster-token="etcd-cluster" \
> --initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=http://192.168.1.2:2380,etcd3=http://3:2380" \
> --initial-cluster-state="new" \
> --log-level="error"
>
> [Install]
> WantedBy=multi-user.target
Are you saying that you already have a compaction job running (e.g.
etcdctl compact $(etcdctl endpoint status --write-out=json | jq -r '.[0].Status.header.revision - 1000')
or similar), but you’re hoping there’s a parameter in etcd itself to handle that scheduling?
On Wed, Aug 20, 2025 at 8:53 AM SOzcn <selahattinozcnma@gmail.com> wrote:
This is high availability environment and etcd is one management tools of them . Many PostgreSQL managed by this architec. I advice to you research. Etcs - Patrono - Postgresql- Haproxy- Keepalived. Anyway if you doesn't have experience of it, that's ok.20 Ağu 2025 Çar 15:46 tarihinde Scott Ribe <scott_ribe@elevated-dev.com> şunu yazdı:This is a PostgreSQL mailing list, and your question has nothing to do with PostgreSQL.
--
Scott Ribe
scott_ribe@elevated-dev.com
https://www.linkedin.com/in/scottribe/
> On Aug 20, 2025, at 1:40 AM, SOzcn <selahattinozcnma@gmail.com> wrote:
>
> Hello,
>
> In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
>
> Although many things in this architecture work as expected, Etcd encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once every 2 or 3 months. The root cause is that the database size reaches the specified limit and Etcd does not clean it up through rotation. I am aware of this, but despite reviewing the maintenance documentation, I could not find the most appropriate parameter values for this rotation. I am sharing an example of my own Etcd configuration below. I kindly ask you to share your experience on this matter. I am kinda stuck on this issue.
>
> In particular, is it possible to schedule the parameter that performs this cleanup with a time setting? For example, can I tell it to run the compact operation at around 01:00 AM?
>
> Best regards.
>
>
> [Unit]
> Description=etcd service
> Documentation=https://github.com/etcd-io/etcd
> After=network.target
>
> [Service]
> User=etcd
> Type=notify
> ExecStart=/usr/local/sbin/etcd \
> --name etcd1 \
> --data-dir=/etc/etcd \
> --initial-advertise-peer-urls=http://10.115.208.168:2380 \
> --listen-peer-urls=http://192.168.1.1:2380 \
> --listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
> --advertise-client-urls=192.168.1.1:2379 \
> --initial-cluster-token="etcd-cluster" \
> --initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=http://192.168.1.2:2380,etcd3=http://3:2380" \
> --initial-cluster-state="new" \
> --log-level="error"
>
> [Install]
> WantedBy=multi-user.target
> On Aug 20, 2025, at 7:52 AM, SOzcn <selahattinozcnma@gmail.com> wrote: > > This is high availability environment and etcd is one management tools of them . Many PostgreSQL managed by this architec.I advice to you research. Etcs - Patrono - Postgresql- Haproxy- Keepalived. Anyway if you doesn't have experienceof it, that's ok. I am perfectly well aware of all that. However, you have an etcd problem, not a postgres problem. You'd have a much betterchance of getting answers to etcd issues on an etcd mailing list.
Hi there,
Etcd database should be extended from 2 gb to 8gb. On the other hand, you can use defrag or again created the etcd db.
Have a nice day,
20 Ağu 2025 Çar 10:40 tarihinde SOzcn <selahattinozcnma@gmail.com> şunu yazdı:
Hello,
In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
Although many things in this architecture work as expected, Etcd encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once every 2 or 3 months. The root cause is that the database size reaches the specified limit and Etcd does not clean it up through rotation. I am aware of this, but despite reviewing the maintenance documentation, I could not find the most appropriate parameter values for this rotation. I am sharing an example of my own Etcd configuration below. I kindly ask you to share your experience on this matter. I am kinda stuck on this issue.
In particular, is it possible to schedule the parameter that performs this cleanup with a time setting? For example, can I tell it to run the compact operation at around 01:00 AM?Best regards.[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
[Service]
User=etcd
Type=notify
ExecStart=/usr/local/sbin/etcd \
--name etcd1 \
--data-dir=/etc/etcd \
--initial-advertise-peer-urls=http://10.115.208.168:2380 \
--listen-peer-urls=http://192.168.1.1:2380 \
--listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
--advertise-client-urls=192.168.1.1:2379 \
--initial-cluster-token="etcd-cluster" \
--initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=http://192.168.1.2:2380,etcd3=http://3:2380" \
--initial-cluster-state="new" \
--log-level="error"
[Install]
WantedBy=multi-user.target
Hello Ozgur,
Thanks for the reply.
If i up to etcd size from 2 to 8 GB it's make much problem but if defrag is fix the "no space" issue can you share your experience? By example is it spent too much rss etc. and are you doing that with cronjob or manuel. As i know if etcd went to error with "no space" defrag methot also doesn't work.
Have a nice day.
If i up to etcd size from 2 to 8 GB it's make much problem but if defrag is fix the "no space" issue can you share your experience? By example is it spent too much rss etc. and are you doing that with cronjob or manuel. As i know if etcd went to error with "no space" defrag methot also doesn't work.
Have a nice day.
Ozgur Kulu <kuluozgur@gmail.com>, 20 Ağu 2025 Çar, 19:51 tarihinde şunu yazdı:
Hi there,Etcd database should be extended from 2 gb to 8gb. On the other hand, you can use defrag or again created the etcd db.Have a nice day,20 Ağu 2025 Çar 10:40 tarihinde SOzcn <selahattinozcnma@gmail.com> şunu yazdı:Hello,
In my database architecture, I use Etcd, Patroni, HAProxy, and Keepalived.
Although many things in this architecture work as expected, Etcd encounters the error "memberID:5960773920904691185 alarm:NOSPACE" once every 2 or 3 months. The root cause is that the database size reaches the specified limit and Etcd does not clean it up through rotation. I am aware of this, but despite reviewing the maintenance documentation, I could not find the most appropriate parameter values for this rotation. I am sharing an example of my own Etcd configuration below. I kindly ask you to share your experience on this matter. I am kinda stuck on this issue.
In particular, is it possible to schedule the parameter that performs this cleanup with a time setting? For example, can I tell it to run the compact operation at around 01:00 AM?Best regards.[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
[Service]
User=etcd
Type=notify
ExecStart=/usr/local/sbin/etcd \
--name etcd1 \
--data-dir=/etc/etcd \
--initial-advertise-peer-urls=http://10.115.208.168:2380 \
--listen-peer-urls=http://192.168.1.1:2380 \
--listen-client-urls=http://192.168.1.1:2379,http://127.0.0.1:2379 \
--advertise-client-urls=192.168.1.1:2379 \
--initial-cluster-token="etcd-cluster" \
--initial-cluster="etcd1=http://192.168.1.1:2380,etcd2=http://192.168.1.2:2380,etcd3=http://3:2380" \
--initial-cluster-state="new" \
--log-level="error"
[Install]
WantedBy=multi-user.target