xDev Asia

Download, install and configure etcd cluster 3 nodes, create systemd service and check health with etcdctl commands.

PostgreSQL High Availability with Patroni & etcd

Part 2: Installation & Configuration

xdev.asia

Aim_

After this lesson, you will:

Understand the role of etcd in Patroni architecture_
Download and install etcd on 3 nodes
Configure etcd cluster with Raft consensus
Create systemd service for etcd_
Check health of etcd cluster_
Use basic etcdctl commands version

1. Introducing etcd

1.1. What is etcd?

etcd is distributed, reliable key-value store using Raft consensus algorithm. Developed by CoreOS and currently a project of CNCF (Cloud Native Computing Foundation).

Main features:

🔐 Strongly consistent: Ensure consistency with Raft
🚀 Fast: Sub-millisecond latency for reads
🔄 Distributed: Running multi-node cluster with quorum
📡 Watch mechanism: Real-time notifications for changes
🔒 TTL support: Automatic key expiration (for leader locks)
🌐 gRPC + HTTP API: Easy integration

1.2. etcd in Patroni Architecture

┌──────────────────────────────────┐
│      etcd Cluster (3 nodes)      │
│  ┌─────┐   ┌─────┐   ┌─────┐    │
│  │etcd1│───│etcd2│───│etcd3│    │
│  └──┬──┘   └──┬──┘   └──┬──┘    │
│     │         │         │         │
│     └─────────┴─────────┘         │
│        Raft Consensus             │
└──────────────────────────────────┘
│        │        │
┌────┴────┐  │  ┌─────┴─────┐
▼         ▼  ▼  ▼           ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│Patroni 1│ │Patroni 2│ │Patroni 3│
└─────────┘ └─────────┘ └─────────┘

etcd saved archive:

/service/postgres/leader: Leader lock (TTL 30s)
/service/postgres/members/: Node information_
/service/postgres/config: Cluster configuration
/service/postgres/initialize: Bootstrap state_
/service/postgres/failover: Failover instructions

2. Download and install etcd

2.1. Architecture considerations

Cluster size recommendations:

3 nodes: Recommendations for production, tolerance 1 failure
5 nodes: High availability, tolerate 2 failures
7+ nodes: Overkill for most uses cases

Deployment topology:

Option 1: etcd on separate servers (Recommended) ┌──────────┐ ┌──────────┐ ┌──────────┐ │ etcd1 │ │ etcd2 │ │ etcd3 │ └──────────┘ └──────────┘ └──────────┘ ▲ ▲ ▲ └─────────────┴─────────────┘ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │Patroni 1 │ │Patroni 2 │ │Patroni 3 │ │ + PG │ │ + PG │ │ + PG │ └──────────┘ └──────────┘ └──────────┘

Option 2: etcd co-located (For labs/dev) ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ etcd1 │ │ etcd2 │ │ etcd3 │ │ Patroni 1 │ │ Patroni 2 │ │ Patroni 3 │ │ PG │ │ PG │ │ PG │ └──────────────┘ └──────────────┘ └──────────────┘

This lab uses Option 2 (co-located) to save resources.

2.2. Install etcd on Ubuntu/Debian

Perform on ALL 3 nodes.

Step 1: Download etcd binary

# Set version
ETCD_VER=v3.5.11
Download
wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz
Extract
tar xzvf etcd-${ETCD_VER}-linux-amd64.tar.gz
Move binaries to PATH
sudo mv etcd-${ETCD_VER}-linux-amd64/etcd /usr/local/bin/
sudo mv etcd-${ETCD_VER}-linux-amd64/etcdctl /usr/local/bin/
sudo mv etcd-${ETCD_VER}-linux-amd64/etcdutl /usr/local/bin/
Verify
etcd --version
etcdctl version

Output:

etcd Version: 3.5.11
Git SHA: ...
Go Version: go1.20.12

Step 2: Create etcd user and directories_

# Tạo user sudo useradd -r -s /bin/false etcd Tạo directories sudo mkdir -p /var/lib/etcd sudo mkdir -p /etc/etcd Set ownership

sudo chown -R etcd:etcd /var/lib/etcd sudo chown -R etcd:etcd /etc/etcd

2.3. Install on CentOS/RHEL

# Download (same as Ubuntu)
ETCD_VER=v3.5.11
wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf etcd-${ETCD_VER}-linux-amd64.tar.gz
sudo mv etcd-${ETCD_VER}-linux-amd64/etcd* /usr/local/bin/
Create user and directories
sudo useradd -r -s /sbin/nologin etcd
sudo mkdir -p /var/lib/etcd /etc/etcd
sudo chown -R etcd:etcd /var/lib/etcd /etc/etcd

3. Configure etcd cluster 3 nodes

3.1. Network topology

node1 (etcd1): 10.0.1.11:2379,2380
node2 (etcd2): 10.0.1.12:2379,2380
node3 (etcd3): 10.0.1.13:2379,2380
Port 2379: Client communication (Patroni connects here)
Port 2380: Peer communication (etcd cluster internal)

3.2. Create configuration file_

Node 1 (10.0.1.11) - /etc/etcd/etcd.conf

# Member name ETCD_NAME="etcd1" Data directory ETCD_DATA_DIR="/var/lib/etcd/etcd1.etcd" Listen URLs ETCD_LISTEN_PEER_URLS="http://10.0.1.11:2380" ETCD_LISTEN_CLIENT_URLS="http://10.0.1.11:2379,http://127.0.0.1:2379" Advertise URLs (what other nodes use to connect) ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.1.11:2380" ETCD_ADVERTISE_CLIENT_URLS="http://10.0.1.11:2379" Cluster configuration ETCD_INITIAL_CLUSTER="etcd1=http://10.0.1.11:2380,etcd2=http://10.0.1.12:2380,etcd3=http://10.0.1.13:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-patroni" Logging

ETCD_LOG_LEVEL="info"

Node 2 (10.0.1.12) - /etc/etcd/etcd.conf

ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/etcd2.etcd"
ETCD_LISTEN_PEER_URLS="http://10.0.1.12:2380"
ETCD_LISTEN_CLIENT_URLS="http://10.0.1.12:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.1.12:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.1.12:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://10.0.1.11:2380,etcd2=http://10.0.1.12:2380,etcd3=http://10.0.1.13:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-patroni"
ETCD_LOG_LEVEL="info"

Node 3 (10.0.1.13) - /etc/etcd/etcd.conf

ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/etcd3.etcd"
ETCD_LISTEN_PEER_URLS="http://10.0.1.13:2380"
ETCD_LISTEN_CLIENT_URLS="http://10.0.1.13:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.1.13:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.1.13:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://10.0.1.11:2380,etcd2=http://10.0.1.12:2380,etcd3=http://10.0.1.13:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-patroni"
ETCD_LOG_LEVEL="info"

3.3. Giải thích các parameters

Parameter	Italian meaning
`ETCD_NAME`	Unique name of member in cluster_
`_ETCD_DATA_DIR`	Save directory data
`ETCD_LISTEN_PEER_URLS`	URL listen for peer communication (port 2380)
`ETCD_LISTEN_CLIENT_URLS`	URL listen for client connections (port 2379)
`ETCD_INITIAL_ADVERTISE_PEER_URLS`	URL for other peers to connect to
`_ETCD_ADVERTISE_CLIENT_URLS`	URL for clients to connect to_
`_ETCD_INITIAL_CLUSTER_`	List of all members when bootstrap
`ETCD_INITIAL_CLUSTER_STATE___HTML TAG_275___`	`new` (first time) or `existing` (add member)
`_ETCD_INITIAL_CLUSTER_TOKEN`	Token for unique cluster (avoid confusion) mixed)

4. Create systemd service

Create file /etc/systemd/system/etcd.service on ALL 3 nodes:

[Unit]
Description=etcd distributed reliable key-value store
Documentation=https://etcd.io/docs/
After=network.target
Wants=network-online.target
[Service]
Type=notify
User=etcd
Group=etcd
Load environment variables from config file
EnvironmentFile=/etc/etcd/etcd.conf
Start etcd with config
ExecStart=/usr/local/bin/etcd
Restart on failure
Restart=on-failure
RestartSec=5
Limits
LimitNOFILE=65536
LimitNPROC=65536
Security
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/etcd
[Install]
WantedBy=multi-user.target

Reload systemd and enable service:

sudo systemctl daemon-reload
sudo systemctl enable etcd

5. Start etcd cluster

5.1. Start etcd on nodes

Important: Start SIMULTANEOUSLY or within 30 seconds so the cluster can form.

Terminal 1 (node1):

sudo systemctl start etcd
sudo systemctl status etcd

Terminal 2 (node2):

sudo systemctl start etcd
sudo systemctl status etcd

Terminal 3 (node3):

sudo systemctl start etcd
sudo systemctl status etcd

5.2. Check logs_

sudo journalctl -u etcd -f

Successful startup logs:_

... etcd1 became leader at term 2
... established a TCP streaming connection with peer etcd2
... established a TCP streaming connection with peer etcd3
... ready to serve client requests

6. Check the health of etcd cluster

6.1. Check cluster members

# Từ bất kỳ node nào etcdctl member list Output: 8e9e05c52164694d, started, etcd1, http://10.0.1.11:2380, http://10.0.1.11:2379, false 91bc3c398fb3c146, started, etcd2, http://10.0.1.12:2380, http://10.0.1.12:2379, false fd422379fda50e48, started, etcd3, http://10.0.1.13:2380, http://10.0.1.13:2379, false

6.2. Check cluster health

etcdctl endpoint health --cluster Output: http://10.0.1.11:2379 is healthy: successfully committed proposal: took = 2.345678ms http://10.0.1.12:2379 is healthy: successfully committed proposal: took = 1.234567ms http://10.0.1.13:2379 is healthy: successfully committed proposal: took = 2.123456ms

6.3. Check endpoint status

etcdctl endpoint status --cluster --write-out=table
Output:
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.0.1.11:2379  | 8e9e05c52164694d |  3.5.11 |   20 kB |      true |      false |         2 |          8 |                  8 |        |
| 10.0.1.12:2379  | 91bc3c398fb3c146 |  3.5.11 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
| 10.0.1.13:2379  | fd422379fda50e48 |  3.5.11 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Explanation output:

IS LEADER_: etcd1 is currently leader
RAFT TERM: Election term (increases each election)
RAFT INDEX: Number of log entries

7. etcdctl basic commands

7.1. Set environment (optional)

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=http://10.0.1.11:2379,http://10.0.1.12:2379,http://10.0.1.13:2379
Thêm vào ~/.bashrc để persistent
echo 'export ETCDCTL_API=3' >> ~/.bashrc
echo 'export ETCDCTL_ENDPOINTS=http://10.0.1.11:2379,http://10.0.1.12:2379,http://10.0.1.13:2379' >> ~/.bashrc

7.2. Basic operations_

_Put/Get/Delete keys

# Write a key etcdctl put /test/key1 "Hello etcd" Read a key etcdctl get /test/key1 Output: /test/key1 Hello etcd Get with details etcdctl get /test/key1 --write-out=json Delete a key

etcdctl del /test/key1

List keys with prefix_

# Put some test keys etcdctl put /service/postgres/test1 "value1" etcdctl put /service/postgres/test2 "value2" List all keys under /service/postgres/ etcdctl get /service/postgres/ --prefix Output: /service/postgres/test1 value1 /service/postgres/test2 value2

Watch for changes

# Terminal 1: Watch for changes etcdctl watch /service/postgres/ --prefix Terminal 2: Make changes etcdctl put /service/postgres/leader "node1" Terminal 1 sẽ hiển thị: PUT /service/postgres/leader node1

TTL keys (used for leader locks)

# Create a lease with 30 seconds TTL etcdctl lease grant 30 Output: lease 7587869125995748410 granted with TTL(30s) Put key with lease etcdctl put /test/ttl-key "value" --lease=7587869125995748410 Key sẽ tự động xóa sau 30 giây Keep lease alive

etcdctl lease keep-alive 7587869125995748410

7.3. Advanced operations_

Transaction (atomic operations)_

# Atomic compare-and-swap
etcdctl txn <<< '
compare:
value("/test/key1") = "old_value"
success requests:
put /test/key1 "new_value"
failure requests:
get /test/key1
'

Snapshot backup_

# Create snapshot etcdctl snapshot save /tmp/etcd-backup.db Verify snapshot

etcdctl snapshot status /tmp/etcd-backup.db --write-out=table

8. Lab: Setup etcd cluster completely

8.1. Lab objectives

✅ Install etcd on 3 nodes
✅ Cluster configuration
✅ Verify cluster health
✅ Test basic operations
✅ Simulate node failure

8.2. Step-by-step lab guide

1. Install etcd on all nodes

Done in Section 2.

2. Create config files

Done in Section 3.

3. Create systemd service

Implemented in Section 4.

4. Start cluster

# Trên cả 3 nodes (đồng thời) sudo systemctl start etcd Check status

sudo systemctl status etcd

5. Verify cluster

# Member list etcdctl member list Health check etcdctl endpoint health --cluster Status

etcdctl endpoint status --cluster --write-out=table

6. Test write/read

# On node1: Write etcdctl put /test/mykey "Hello from etcd cluster" On node2: Read etcdctl get /test/mykey Should see: Hello from etcd cluster On node3: Verify etcdctl get /test/mykey Should see: Hello from etcd cluster

7. Test leader election

# Identify current leader etcdctl endpoint status --cluster --write-out=table Note which node IS LEADER = true Stop leader node sudo systemctl stop etcd # On leader node Wait 5-10 seconds Check from another node etcdctl endpoint status --cluster --write-out=table New leader should be elected Restart stopped node sudo systemctl start etcd # On stopped node Verify rejoined

etcdctl member list

8. Test data persistence_

# Write some data etcdctl put /persistent/key "This should survive restart" Restart ALL nodes (one by one) sudo systemctl restart etcd Verify data etcdctl get /persistent/key Should still see: This should survive restart

8.3. Troubleshooting common issues

Issue 1: Cluster won't form

# Symptom journalctl -u etcd -n 50 Error: "request cluster ID mismatch" Solution: Clear data and restart

sudo systemctl stop etcd sudo rm -rf /var/lib/etcd/* sudo systemctl start etcd

Issue 2: Cannot connect to etcd

# Check if etcd is listening sudo netstat -tlnp | grep etcd Should see ports 2379 and 2380 Check firewall sudo firewall-cmd --list-all # CentOS/RHEL sudo ufw status # Ubuntu Add firewall rules if needed

sudo ufw allow 2379/tcp sudo ufw allow 2380/tcp

Issue 3: Node won't join cluster

# Check ETCD_INITIAL_CLUSTER in config cat /etc/etcd/etcd.conf | grep INITIAL_CLUSTER Verify network connectivity

ping 10.0.1.11 telnet 10.0.1.11 2380

Issue 4: Split-brain or multiple leaders

# Check cluster status etcdctl endpoint status --cluster --write-out=table If multiple leaders (shouldn't happen with proper setup): 1. Stop all etcd instances sudo systemctl stop etcd # On all nodes 2. Clear data on all nodes sudo rm -rf /var/lib/etcd/* 3. Restart cluster (bootstrap again) Start all nodes within 30 seconds

9. Performance tuning

9.1. etcd tuning parameters

# Add to /etc/etcd/etcd.conf
Heartbeat interval (default: 100ms)
ETCD_HEARTBEAT_INTERVAL="100"
Election timeout (default: 1000ms)
ETCD_ELECTION_TIMEOUT="1000"
Snapshot count (default: 10000)
Compact and snapshot after this many transactions
ETCD_SNAPSHOT_COUNT="10000"
Quota backend bytes (default: 2GB)
Max database size
ETCD_QUOTA_BACKEND_BYTES="2147483648"

9.2. Monitoring etcd

Key metrics to monitor:

Latency (99th percentile < 50ms)
Disk fsync duration (< 10ms)
Leader changes (should be rare)
Database size
Failed proposals

Check metrics:_

curl http://10.0.1.11:2379/metrics Key metrics: etcd_server_has_leader etcd_server_leader_changes_seen_total etcd_disk_backend_commit_duration_seconds etcd_network_peer_round_trip_time_seconds

10. Summary

Key Takeaways

✅ etcd cluster: 3-node cluster for production HA

✅ Ports: 2379 (client), 2380 (peer)

✅ Raft consensus: Automatic leader election and data replication

✅ Quorum: Need 2/3 nodes for cluster to operate dynamic

✅ TTL keys: Used for Patroni leader locks

✅ etcdctl: CLI tool to manage and troubleshoot

Checklist later Lab

etcd cluster 3 nodes running
etcdctl member list full display 3 members
etcdctl endpoint health --cluster all healthy
There is 1 leader and 2 followers
etcd service enabled and will auto-start upon reboot
Firewall allows ports 2379 and 2380_

_Current architecture at

✅ 3 VMs prepared (Bài 4)
✅ PostgreSQL 15 installed (Bài 5)
✅ etcd cluster running (Bài 6)
Next: Cài đặt Patroni và bootstrap HA cluster

Preparing for Lesson 7

The next lesson will install Patroni and integrate with the setup etcd cluster.

Lesson 6: Install and configure etcd cluster

Aim_

1. Introducing etcd

1.1. What is etcd?

1.2. etcd in Patroni Architecture

2. Download and install etcd

2.1. Architecture considerations

2.2. Install etcd on Ubuntu/Debian

Step 1: Download etcd binary

Download

Extract

Move binaries to PATH

Verify

Step 2: Create etcd user and directories_

Tạo directories

Set ownership

2.3. Install on CentOS/RHEL

Create user and directories

3. Configure etcd cluster 3 nodes

3.1. Network topology

3.2. Create configuration file_

Node 1 (10.0.1.11) - /etc/etcd/etcd.conf

Data directory

Listen URLs

Advertise URLs (what other nodes use to connect)

Cluster configuration

Logging

Node 2 (10.0.1.12) - /etc/etcd/etcd.conf

Node 3 (10.0.1.13) - /etc/etcd/etcd.conf

3.3. Giải thích các parameters

4. Create systemd service

Load environment variables from config file

Start etcd with config

Restart on failure

Limits

Security

5. Start etcd cluster

5.1. Start etcd on nodes

5.2. Check logs_

6. Check the health of etcd cluster

6.1. Check cluster members

Output:

8e9e05c52164694d, started, etcd1, http://10.0.1.11:2380, http://10.0.1.11:2379, false

91bc3c398fb3c146, started, etcd2, http://10.0.1.12:2380, http://10.0.1.12:2379, false

fd422379fda50e48, started, etcd3, http://10.0.1.13:2380, http://10.0.1.13:2379, false

6.2. Check cluster health

Output:

http://10.0.1.11:2379 is healthy: successfully committed proposal: took = 2.345678ms

http://10.0.1.12:2379 is healthy: successfully committed proposal: took = 1.234567ms

http://10.0.1.13:2379 is healthy: successfully committed proposal: took = 2.123456ms

6.3. Check endpoint status

Output:

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

| 10.0.1.11:2379 | 8e9e05c52164694d | 3.5.11 | 20 kB | true | false | 2 | 8 | 8 | |

| 10.0.1.12:2379 | 91bc3c398fb3c146 | 3.5.11 | 20 kB | false | false | 2 | 8 | 8 | |

| 10.0.1.13:2379 | fd422379fda50e48 | 3.5.11 | 20 kB | false | false | 2 | 8 | 8 | |

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

7. etcdctl basic commands

7.1. Set environment (optional)

Thêm vào ~/.bashrc để persistent

7.2. Basic operations_

_Put/Get/Delete keys

Read a key

Output:

/test/key1

Hello etcd

Get with details

Delete a key

List keys with prefix_

List all keys under /service/postgres/

Output:

/service/postgres/test1

value1

/service/postgres/test2

value2

Watch for changes

Terminal 2: Make changes

Terminal 1 sẽ hiển thị: