xDev Asia

Cài đặt Python dependencies, setup Patroni qua pip, phân tích cấu trúc patroni.yml và tạo systemd service trên 3 nodes.

PostgreSQL High Availability với Patroni & etcd

Phần 2: Cài đặt & Cấu hình

xdev.asia

Mục tiêu

Sau bài học này, bạn sẽ:

Cài đặt Python và các dependencies cần thiết
Cài đặt Patroni qua pip
Hiểu cấu trúc file patroni.yml
Tạo systemd service cho Patroni
Cài đặt Patroni trên 3 nodes

1. Giới thiệu

Patroni là một Python application, do đó cần Python runtime và các dependencies. Trong bài này, chúng ta sẽ:

Cài đặt Python 3 và pip
Cài đặt Patroni package
Cấu hình Patroni
Tạo systemd service để quản lý Patroni daemon

Kiến trúc mục tiêu:

┌──────────────────────────────────┐
│      etcd Cluster (3 nodes)      │
│         ✅ RUNNING                │
└──────────────────────────────────┘
         │        │        │
    ┌────┴────┐  │  ┌─────┴─────┐
    ▼         ▼  ▼  ▼           ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│Patroni 1│ │Patroni 2│ │Patroni 3│ ← Installing now
│  + PG   │ │  + PG   │ │  + PG   │
└─────────┘ └─────────┘ └─────────┘

2. Cài đặt Python Dependencies

2.1. Cài đặt Python trên Ubuntu/Debian

Thực hiện trên TẤT CẢ 3 nodes.

Bước 1: Cài đặt Python 3 và pip

# Update package list
sudo apt update
Cài Python 3, pip, và development tools
sudo apt install -y python3 python3-pip python3-dev python3-venv
Kiểm tra version
python3 --version
Output: Python 3.10.x (hoặc 3.11.x hoặc 3.12.x)
pip3 --version
Output: pip 22.x.x

Bước 2: Cài đặt system dependencies

# Cài các libraries cần thiết cho Patroni và PostgreSQL Python modules sudo apt install -y libpq-dev gcc python3-psycopg2 libpq-dev: PostgreSQL client library headers gcc: Compiler cho building Python packages python3-psycopg2: PostgreSQL adapter cho Python

2.2. Cài đặt Python trên CentOS/RHEL

# Python 3 thường đã có sẵn, nhưng cần pip và dev tools sudo dnf install -y python3 python3-pip python3-devel System dependencies sudo dnf install -y postgresql18-devel gcc python3-psycopg2 Kiểm tra

python3 --version pip3 --version

2.3. Upgrade pip (khuyến nghị)

# Upgrade pip to latest version sudo pip3 install --upgrade pip Verify

pip3 --version

3. Cài đặt Patroni qua pip

3.1. Cài đặt Patroni

Thực hiện trên TẤT CẢ 3 nodes.

# Cài Patroni với etcd support sudo pip3 install patroni[etcd] Hoặc specify version cụ thể sudo pip3 install patroni[etcd]==3.2.0 Kiểm tra installation patroni --version Output: patroni 3.2.0 patronictl --version Output: patronictl 3.2.0

Giải thích [etcd]:

Patroni hỗ trợ nhiều DCS backends (etcd, consul, zookeeper)
[etcd] install thêm python-etcd client library
Có thể cài nhiều backends: patroni[etcd,consul,zookeeper]

3.2. Các packages được cài đặt

# Liệt kê các packages liên quan
pip3 list | grep -E "(patroni|etcd|psycopg)"
Output:
patroni          3.2.0
python-etcd      0.4.5
psycopg2         2.9.x
psycopg2-binary  2.9.x

3.3. Verify Patroni commands

# Check patroni binary which patroni Output: /usr/local/bin/patroni Check patronictl binary which patronictl Output: /usr/local/bin/patronictl List patronictl commands

patronictl --help

Output:

Usage: patronictl [OPTIONS] COMMAND [ARGS]...

Commands: list Show cluster members switchover Perform planned switchover failover Perform manual failover reinit Reinitialize cluster member restart Restart cluster member reload Reload cluster member configuration pause Disable auto-failover resume Enable auto-failover edit-config Edit cluster configuration ...

4. Cấu trúc file patroni.yml

4.1. Tổng quan về patroni.yml

File patroni.yml là file cấu hình chính của Patroni, bao gồm:

Scope: Tên cluster
Namespace: Prefix cho keys trong DCS
Node information: Tên node, REST API config
DCS connection: etcd endpoints
Bootstrap: Initial cluster setup
PostgreSQL: Database configuration
Tags: Node metadata
Watchdog: Optional hardware watchdog

4.2. Cấu trúc cơ bản

tags: nofailover: false noloadbalance: false clonefrom: false nosync: false

4.3. Giải thích các sections chính

Section: Global

scope: postgres          # Cluster name (unique identifier)
namespace: /service/     # DCS key prefix
name: node1             # Unique node name trong cluster

Section: REST API

restapi: listen: 0.0.0.0:8008 # Listen trên tất cả interfaces connect_address: 10.0.1.11:8008 # Address để nodes khác connect authentication: # Optional: Basic auth username: admin password: secret

REST API endpoints:

GET /: Cluster info
GET /health: Health check (200 = healthy)
GET /primary: Check if node is primary
GET /replica: Check if node is replica
POST /restart: Restart PostgreSQL
POST /reload: Reload configuration

Section: etcd (DCS)

etcd: hosts: 10.0.1.11:2379,10.0.1.12:2379,10.0.1.13:2379 # etcd endpoints protocol: http # http hoặc https username: user # Optional: etcd authentication password: pass

Section: Bootstrap

bootstrap: dcs: ttl: 30 # Leader lock TTL (seconds) loop_wait: 10 # Check interval (seconds) retry_timeout: 10 # DCS operation timeout maximum_lag_on_failover: 1048576 # Max lag (bytes) để eligible for failover postgresql: use_pg_rewind: true # Enable pg_rewind for fast recovery parameters: # PostgreSQL parameters wal_level: replica hot_standby: "on" wal_keep_size: "1GB" max_wal_senders: 10 max_replication_slots: 10 wal_log_hints: "on" # Required for pg_rewind initdb: # pg_initdb options - encoding: UTF8 - data-checksums # Enable page checksums - locale: en_US.UTF-8 pg_hba: # pg_hba.conf entries - host replication replicator 10.0.1.0/24 scram-sha-256 - host all all 0.0.0.0/0 scram-sha-256

users: # Create users during bootstrap admin: password: admin_password options: - createrole - createdb

Lưu ý: Bootstrap section chỉ áp dụng khi khởi tạo cluster lần đầu.

Section: PostgreSQL

postgresql: listen: 0.0.0.0:5432 # PostgreSQL listen address connect_address: 10.0.1.11:5432 # Address để connect từ bên ngoài data_dir: /var/lib/postgresql/18/data # Data directory bin_dir: /usr/lib/postgresql/18/bin # PostgreSQL binaries pgpass: /tmp/pgpass # Optional: pgpass file authentication: replication: username: replicator password: replicator_password superuser: username: postgres password: postgres_password rewind: # Optional: dedicated user for pg_rewind username: rewind_user password: rewind_password parameters: # PostgreSQL runtime parameters max_connections: 100 shared_buffers: 2GB effective_cache_size: 6GB maintenance_work_mem: 512MB checkpoint_completion_target: 0.9 wal_buffers: 16MB default_statistics_target: 100 random_page_cost: 1.1 effective_io_concurrency: 200 work_mem: 16MB min_wal_size: 1GB max_wal_size: 2GB pg_hba: # Additional pg_hba.conf entries - host all all 10.0.2.0/24 scram-sha-256

callbacks: # Optional: callback scripts on_start: /etc/patroni/scripts/on_start.sh on_stop: /etc/patroni/scripts/on_stop.sh on_role_change: /etc/patroni/scripts/on_role_change.sh

Section: Tags

tags: nofailover: false # false = có thể become primary noloadbalance: false # false = có thể nhận read traffic clonefrom: false # false = có thể clone từ node này nosync: false # false = có thể become synchronous standby Custom tags

datacenter: dc1 environment: production

Section: Watchdog (Optional)

watchdog:
mode: required         # required, automatic, hoặc off
device: /dev/watchdog  # Hardware watchdog device
safety_margin: 5       # Seconds before watchdog triggers

5. Tạo Patroni configuration files

5.1. Tạo thư mục cấu hình

Trên TẤT CẢ 3 nodes:

# Tạo directory cho Patroni config sudo mkdir -p /etc/patroni Tạo directory cho callback scripts (optional) sudo mkdir -p /etc/patroni/scripts Set ownership

sudo chown -R postgres:postgres /etc/patroni

5.2. Node 1 - /etc/patroni/patroni.yml

scope: postgres namespace: /service/ name: node1 restapi: listen: 0.0.0.0:8008 connect_address: 10.0.1.11:8008 etcd: hosts: 10.0.1.11:2379,10.0.1.12:2379,10.0.1.13:2379 bootstrap: dcs: ttl: 30 loop_wait: 10 retry_timeout: 10 maximum_lag_on_failover: 1048576 postgresql: use_pg_rewind: true parameters: wal_level: replica hot_standby: "on" wal_keep_size: "1GB" max_wal_senders: 10 max_replication_slots: 10 wal_log_hints: "on" initdb: - encoding: UTF8 - data-checksums pg_hba: - host replication replicator 10.0.1.11/32 scram-sha-256 - host replication replicator 10.0.1.12/32 scram-sha-256 - host replication replicator 10.0.1.13/32 scram-sha-256 - host all all 10.0.1.0/24 scram-sha-256 - host all all 0.0.0.0/0 md5 users: admin: password: admin123 options: - createrole - createdb postgresql: listen: 0.0.0.0:5432 connect_address: 10.0.1.11:5432 data_dir: /var/lib/postgresql/18/data bin_dir: /usr/lib/postgresql/18/bin authentication: replication: username: replicator password: replicator_password superuser: username: postgres password: postgres_password parameters: max_connections: 100 shared_buffers: 2GB effective_cache_size: 6GB maintenance_work_mem: 512MB checkpoint_completion_target: 0.9

tags: nofailover: false noloadbalance: false clonefrom: false nosync: false

5.3. Node 2 - /etc/patroni/patroni.yml

scope: postgres namespace: /service/ name: node2 restapi: listen: 0.0.0.0:8008 connect_address: 10.0.1.12:8008 etcd: hosts: 10.0.1.11:2379,10.0.1.12:2379,10.0.1.13:2379 bootstrap: dcs: ttl: 30 loop_wait: 10 retry_timeout: 10 maximum_lag_on_failover: 1048576 postgresql: use_pg_rewind: true parameters: wal_level: replica hot_standby: "on" wal_keep_size: "1GB" max_wal_senders: 10 max_replication_slots: 10 wal_log_hints: "on" initdb: - encoding: UTF8 - data-checksums pg_hba: - host replication replicator 10.0.1.11/32 scram-sha-256 - host replication replicator 10.0.1.12/32 scram-sha-256 - host replication replicator 10.0.1.13/32 scram-sha-256 - host all all 10.0.1.0/24 scram-sha-256 - host all all 0.0.0.0/0 md5 users: admin: password: admin123 options: - createrole - createdb postgresql: listen: 0.0.0.0:5432 connect_address: 10.0.1.12:5432 data_dir: /var/lib/postgresql/18/data bin_dir: /usr/lib/postgresql/18/bin authentication: replication: username: replicator password: replicator_password superuser: username: postgres password: postgres_password parameters: max_connections: 100 shared_buffers: 2GB effective_cache_size: 6GB maintenance_work_mem: 512MB checkpoint_completion_target: 0.9

tags: nofailover: false noloadbalance: false clonefrom: false nosync: false

5.4. Node 3 - /etc/patroni/patroni.yml

scope: postgres namespace: /service/ name: node3 restapi: listen: 0.0.0.0:8008 connect_address: 10.0.1.13:8008 etcd: hosts: 10.0.1.11:2379,10.0.1.12:2379,10.0.1.13:2379 bootstrap: dcs: ttl: 30 loop_wait: 10 retry_timeout: 10 maximum_lag_on_failover: 1048576 postgresql: use_pg_rewind: true parameters: wal_level: replica hot_standby: "on" wal_keep_size: "1GB" max_wal_senders: 10 max_replication_slots: 10 wal_log_hints: "on" initdb: - encoding: UTF8 - data-checksums pg_hba: - host replication replicator 10.0.1.11/32 scram-sha-256 - host replication replicator 10.0.1.12/32 scram-sha-256 - host replication replicator 10.0.1.13/32 scram-sha-256 - host all all 10.0.1.0/24 scram-sha-256 - host all all 0.0.0.0/0 md5 users: admin: password: admin123 options: - createrole - createdb postgresql: listen: 0.0.0.0:5432 connect_address: 10.0.1.13:5432 data_dir: /var/lib/postgresql/18/data bin_dir: /usr/lib/postgresql/18/bin authentication: replication: username: replicator password: replicator_password superuser: username: postgres password: postgres_password parameters: max_connections: 100 shared_buffers: 2GB effective_cache_size: 6GB maintenance_work_mem: 512MB checkpoint_completion_target: 0.9

tags: nofailover: false noloadbalance: false clonefrom: false nosync: false

5.5. Set permissions

Trên TẤT CẢ 3 nodes:

# Set ownership sudo chown postgres:postgres /etc/patroni/patroni.yml Secure permissions (file chứa passwords) sudo chmod 600 /etc/patroni/patroni.yml Verify ls -l /etc/patroni/patroni.yml Output: -rw------- 1 postgres postgres ... patroni.yml

6. Tạo systemd service cho Patroni

6.1. Tạo systemd unit file

Tạo file /etc/systemd/system/patroni.service trên TẤT CẢ 3 nodes:

[Unit]
Description=Patroni PostgreSQL HA manager
Documentation=https://patroni.readthedocs.io/
After=syslog.target network.target etcd.service
[Service]
Type=simple
User=postgres
Group=postgres
Start Patroni
ExecStart=/usr/local/bin/patroni /etc/patroni/patroni.yml
Reload configuration
ExecReload=/bin/kill -HUP $MAINPID
Restart behavior
Restart=on-failure
RestartSec=5
Limits
LimitNOFILE=65536
LimitNPROC=65536
Logging
StandardOutput=journal
StandardError=journal
Working directory
WorkingDirectory=/var/lib/postgresql
Environment
Environment=PATH=/usr/lib/postgresql/18/bin:/usr/local/bin:/usr/bin:/bin
[Install]
WantedBy=multi-user.target

6.2. Giải thích systemd unit file

Directive	Giải thích
`After=etcd.service`	Khởi động sau khi etcd đã sẵn sàng
`Type=simple`	Process chạy foreground
`User=postgres`	Chạy với user postgres
`ExecStart`	Command để start Patroni
`ExecReload`	Send HUP signal để reload config
`Restart=on-failure`	Auto restart nếu fail
`RestartSec=5`	Đợi 5 giây trước khi restart
`LimitNOFILE=65536`	Max open files
`StandardOutput=journal`	Log vào systemd journal

6.3. Enable và verify service

Trên TẤT CẢ 3 nodes:

# Reload systemd sudo systemctl daemon-reload Enable Patroni service (auto-start on boot) sudo systemctl enable patroni Verify service file systemctl cat patroni Check status (should be inactive/dead - not started yet)

systemctl status patroni

7. Verify installation

7.1. Check Patroni installation

Trên TẤT CẢ 3 nodes:

# Check Patroni version
patroni --version
Check patronictl
patronictl --help
Verify config file
sudo -u postgres cat /etc/patroni/patroni.yml
Validate YAML syntax
python3 -c "import yaml; yaml.safe_load(open('/etc/patroni/patroni.yml'))"
No output = valid YAML

7.2. Check etcd connectivity

# Test etcd từ mỗi node etcdctl endpoint health --cluster Should see all 3 etcd nodes healthy

7.3. Pre-flight checklist

Trước khi khởi động Patroni, verify:

# ✅ PostgreSQL installed nhưng NOT running systemctl status postgresql Should be: inactive (dead) ✅ etcd cluster healthy etcdctl endpoint health --cluster ✅ Patroni config file exists và có permissions đúng ls -l /etc/patroni/patroni.yml ✅ Data directory có ownership đúng ls -ld /var/lib/postgresql/18/data Owner: postgres:postgres ✅ systemd service enabled systemctl is-enabled patroni Output: enabled ✅ Firewall rules (nếu có firewall) sudo ufw status | grep -E "(5432|8008)" Hoặc

sudo firewall-cmd --list-ports | grep -E "(5432|8008)"

8. Troubleshooting

8.1. Issue: pip install fails

# Error: "No module named 'setuptools'" Solution: sudo apt install python3-setuptools Or upgrade pip

sudo pip3 install --upgrade pip setuptools

8.2. Issue: Cannot find PostgreSQL binaries

# Error: "could not find postgres binary" Solution: Check bin_dir in patroni.yml Find PostgreSQL bin directory which postgres Output: /usr/lib/postgresql/18/bin/postgres Update patroni.yml

postgresql: bin_dir: /usr/lib/postgresql/18/bin

8.3. Issue: Permission denied on data directory

# Error: "FATAL: data directory ... has wrong ownership" Solution:

sudo chown -R postgres:postgres /var/lib/postgresql/18/data sudo chmod 700 /var/lib/postgresql/18/data

8.4. Issue: YAML syntax error

# Validate YAML
python3 -c "import yaml; yaml.safe_load(open('/etc/patroni/patroni.yml'))"
Common issues:
- Mixed tabs and spaces (use spaces only)
- Incorrect indentation
- Missing quotes around special characters
- Duplicate keys

9. Tổng kết

Key Takeaways

✅ Patroni: Python application, cài qua pip

✅ Dependencies: Python 3, pip, psycopg2, python-etcd

✅ Configuration: patroni.yml chứa tất cả settings

✅ systemd service: Quản lý Patroni daemon

✅ Security: File config có permissions 600 (chứa passwords)

Checklist sau Lab

Python 3 và pip đã cài trên cả 3 nodes
Patroni 3.2.0+ đã cài trên cả 3 nodes
File /etc/patroni/patroni.yml đã tạo trên mỗi node với config riêng
systemd service patroni.service đã tạo và enabled
Permissions đúng cho config file (600, owner postgres)
etcd cluster đang running và healthy

Kiến trúc hiện tại

✅ 3 VMs prepared (Bài 4)
✅ PostgreSQL 18 installed (Bài 5)
✅ etcd cluster running (Bài 6)
✅ Patroni installed and configured (Bài 7)
Next: Bootstrap cluster lần đầu

Chuẩn bị cho Bài 8

Bài 8 sẽ đi sâu vào cấu hình Patroni chi tiết:

Phân tích từng section trong patroni.yml
Bootstrap configuration options
PostgreSQL parameters tuning
Authentication setup
Tags và constraints

Bài 9: Bootstrap cluster

Sau khi Patroni đã cài và cấu hình xong, Bài 9 sẽ hướng dẫn:

Khởi động Patroni lần đầu
Quá trình bootstrap tự động
Kiểm tra cluster status
Troubleshootin

Bài 7: Cài đặt Patroni

Mục tiêu

1. Giới thiệu

2. Cài đặt Python Dependencies

2.1. Cài đặt Python trên Ubuntu/Debian

Bước 1: Cài đặt Python 3 và pip

Cài Python 3, pip, và development tools

Kiểm tra version

Output: Python 3.10.x (hoặc 3.11.x hoặc 3.12.x)

Output: pip 22.x.x

Bước 2: Cài đặt system dependencies

libpq-dev: PostgreSQL client library headers

gcc: Compiler cho building Python packages

python3-psycopg2: PostgreSQL adapter cho Python

2.2. Cài đặt Python trên CentOS/RHEL

System dependencies

Kiểm tra

2.3. Upgrade pip (khuyến nghị)

Verify

3. Cài đặt Patroni qua pip

3.1. Cài đặt Patroni

Hoặc specify version cụ thể

Kiểm tra installation

Output: patroni 3.2.0

Output: patronictl 3.2.0

3.2. Các packages được cài đặt

Output:

patroni 3.2.0

python-etcd 0.4.5

psycopg2 2.9.x

psycopg2-binary 2.9.x

3.3. Verify Patroni commands

Output: /usr/local/bin/patroni

Check patronictl binary

Output: /usr/local/bin/patronictl

List patronictl commands

4. Cấu trúc file patroni.yml

4.1. Tổng quan về patroni.yml

4.2. Cấu trúc cơ bản

4.3. Giải thích các sections chính

Section: Global

Section: REST API

authentication: # Optional: Basic auth

username: admin

password: secret

Section: etcd (DCS)

protocol: http # http hoặc https

username: user # Optional: etcd authentication

password: pass

Section: Bootstrap

Section: PostgreSQL

Section: Tags

Custom tags

Section: Watchdog (Optional)

5. Tạo Patroni configuration files

5.1. Tạo thư mục cấu hình

Tạo directory cho callback scripts (optional)

Set ownership

5.2. Node 1 - /etc/patroni/patroni.yml

5.3. Node 2 - /etc/patroni/patroni.yml

5.4. Node 3 - /etc/patroni/patroni.yml

5.5. Set permissions

Secure permissions (file chứa passwords)

Verify

Output: -rw------- 1 postgres postgres ... patroni.yml

6. Tạo systemd service cho Patroni

6.1. Tạo systemd unit file

Start Patroni

Reload configuration

Restart behavior

Limits

Logging

Working directory

Environment

6.2. Giải thích systemd unit file

6.3. Enable và verify service

Enable Patroni service (auto-start on boot)

Verify service file

Check status (should be inactive/dead - not started yet)

7. Verify installation