目標
このレッスンを終えると、次のことができるようになります:
- Patroni REST API とエンドポイントを理解する_
- ヘルスチェックに REST API を使用する
- ロードバランサー (HAProxy、HAProxy、 Nginx)
- クラスターのステータスと構成のクエリ
- カスタム監視の実装
- REST API エンドポイントの保護
1。 REST API の概要
1.1。 REST API とは何ですか?
Patroni は、HTTP REST API を各ノードで次の目的で公開します:
- 🔍 ヘルスチェック: ロードバランサーはノードの健全性をチェック
- 📊 監視: 外部システムがクラスターの状態をクエリ
- ⚙️ 管理: 構成の読み取り、クラスタートポロジー
- 🔄 オートメーション: CI/CD、オーケストレーションツールとの統合
1.2。 API 構成
patroni.yml 内:
restapi: listen: 0.0.0.0:8008 # Listen address and port connect_address: 10.0.1.11:8008 # Advertised addressOptional: Basic authentication
authentication:
username: admin
password: secret_password
Optional: SSL/TLS
certfile: /etc/patroni/certs/server.crt
keyfile: /etc/patroni/certs/server.key
cafile: /etc/patroni/certs/ca.crt
デフォルトポート: 8008
1.3。 API エンドポイントの概要
| _エンドポイント_ | _方法 | _目的_ | 使用ケース_ |
|---|---|---|---|
| __ _HTMLTAG_140___/ | GET | 基本ノード情報_ | クイックヘルスチェック |
/プライマリ_ または _/master | _GET | ノードがプライマリかどうかを確認 | LB プライマリルーティング_ |
/レプリカ_ | GET | ノードがレプリカかどうかを確認 | LB読み取りルーティング_ |
/読み書き_ | GET | 書き込み可能かどうかを確認する(プライマリ) | LB書き込みルーティング_ |
/読み取り専用 または_ /スタンバイ | GET | 読み取り専用 (レプリカ) かどうかを確認 | LB 読み取りルーティング_ |
/同期 | GET | 同期レプリカかどうかを確認 | 同期レプリカ検出 |
/非同期_ | GET | 非同期レプリカかどうかを確認_ | 非同期レプリカ検出_ |
/健康_ | GET | 詳細な健康状態check_ | モニタリング |
| __ _HTMLTAG_248___/patroni | GET | 詳細なクラスターとノードinfo_ | 高度な監視_ |
/config_ | GET | クラスター構成DCS_ | 構成検査 |
/クラスター_ | GET | すべてのクラスターメンバーinfo_ | トポロジビュー |
/history_ | GET | フェイルオーバー履歴_ | 監査ログ |
2.ヘルスチェックエンドポイント
2.1。基本的なヘルスチェック: GET /
目的: ノードが実行されているかどうかを簡単にチェックします。
curl -s http://10.0.1.11:8008/Response on PRIMARY:
HTTP 200 OK
{
"state": "running",
"postmaster_start_time": "2024-11-25 10:30:00.123456+00:00",
"role": "master",
"server_version": 180000,
"cluster_unlocked": false,
"xlog": {
"location": 67108864
},
"timeline": 1,
"database_system_identifier": "7001234567890123456",
"patroni": {
"version": "3.2.0",
"scope": "postgres"
}
}
Response on REPLICA:
HTTP 200 OK
{
"state": "running",
"postmaster_start_time": "2024-11-25 10:31:15.789012+00:00",
"role": "replica",
"server_version": 180000,
"cluster_unlocked": false,
"xlog": {
"received_location": 67108864,
"replayed_location": 67108864
},
"timeline": 1,
"database_system_identifier": "7001234567890123456",
"patroni": {
"version": "3.2.0",
"scope": "postgres"
}
}
応答コード_:
- 200 OK: ノードは正常で実行中_
- 503 サービスは利用できません: ノードは異常です (PostgreSQL下など)
2.2。プライマリ チェック: GET /primary または /master
目的: ノードが現在のプライマリ/リーダーであるかどうかを確認します。
curl -s http://10.0.1.11:8008/primaryOn PRIMARY:
HTTP 200 OK
{
"state": "running",
"role": "master",
"xlog": {
"location": 67108864
}
}
On REPLICA:
HTTP 503 Service Unavailable
(empty body or error message)
使用例: ロード バランサーのヘルス チェック書き込みトラフィック ルーティングの場合。
2.3。レプリカ チェック: GET /replica
目的: ノードがレプリカ (スタンバイ) かどうかを確認します。
curl -s http://10.0.1.12:8008/replicaOn REPLICA:
HTTP 200 OK
{
"state": "running",
"role": "replica",
"xlog": {
"received_location": 67108864,
"replayed_location": 67108864
}
}
On PRIMARY:
HTTP 503 Service Unavailable
: ロード バランサーのヘルス チェック読み取りトラフィック ルーティングの場合。
2.4。読み取り/書き込みチェック: GET /read-write__HTMLTAG_344___目的: ノードが書き込みを受け入れるかどうかを確認します (プライマリ + メンテナンス中ではない)。
curl -s http://10.0.1.11:8008/read-write
Returns 200 if:
- Node is primary
- Cluster is not paused
- No maintenance mode
2.5。読み取り専用チェック: GET /read-only または /standby
目的: Cノードが読み取り専用レプリカかどうかを確認してください。
curl -s http://10.0.1.12:8008/read-only
Returns 200 if:
- Node is replica
- PostgreSQL is running
- Replication lag < threshold (optional)
詳細: ラグ許容範囲:
# Check replica with max 1MB lag tolerance
curl -s "http://10.0.1.12:8008/read-only?lag=1048576"
Returns 503 if lag > 1MB
2.6。同期レプリカのチェック: GET /synchronous
目的: ノードが同期レプリカであるかどうかを確認します。
curl -s http://10.0.1.12:8008/synchronous
Returns 200 if:
- Node is replica
- sync_state = 'sync' (from pg_stat_replication)
2.7。非同期レプリカのチェック: GET /asynchronous
目的: ノードが非同期レプリカであるかどうかを確認します。
curl -s http://10.0.1.13:8008/asynchronous
Returns 200 if:
- Node is replica
- sync_state != 'sync'
2.8。健康エンドポイント: GET /health
目的: 詳細な健康情報。
curl -s http://10.0.1.11:8008/health | jq
Response:
{
"state": "running",
"role": "master",
"server_version": 180000,
"cluster_unlocked": false,
"timeline": 1,
"database_system_identifier": "7001234567890123456",
"postmaster_start_time": "2024-11-25 10:30:00.123456+00:00",
"patroni": {
"version": "3.2.0",
"scope": "postgres",
"name": "node1"
},
"replication": [
{
"usename": "replicator",
"application_name": "node2",
"client_addr": "10.0.1.12",
"state": "streaming",
"sync_state": "sync",
"sync_priority": 1
},
{
"usename": "replicator",
"application_name": "node3",
"client_addr": "10.0.1.13",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
]
}
3。クラスター情報エンドポイント
3.1。詳細なノード情報: GET /patroni
目的: 包括的なノードおよびクラスター情報。
curl -s http://10.0.1.11:8008/patroni | jq
Response (truncated):
{
"state": "running",
"postmaster_start_time": "2024-11-25 10:30:00.123456+00:00",
"role": "master",
"server_version": 180000,
"xlog": {
"location": 67108864
},
"timeline": 1,
"cluster_unlocked": false,
"database_system_identifier": "7001234567890123456",
"patroni": {
"version": "3.2.0",
"scope": "postgres",
"name": "node1"
},
"dcs": {
"last_seen": 1700912345,
"ttl": 30
},
"tags": {
"nofailover": false,
"noloadbalance": false,
"clonefrom": false,
"nosync": false
},
"pending_restart": false,
"replication": [...],
"timeline_history": [...]
}
3.2。クラスター構成: GET /config
目的: DCS からクラスター全体の構成を取得します。
curl -s http://10.0.1.11:8008/config | jq
Response:
{
"ttl": 30,
"loop_wait": 10,
"retry_timeout": 10,
"maximum_lag_on_failover": 1048576,
"synchronous_mode": true,
"synchronous_mode_strict": false,
"postgresql": {
"parameters": {
"max_connections": 100,
"shared_buffers": "256MB",
"wal_level": "replica",
"max_wal_senders": 10,
"max_replication_slots": 10,
"hot_standby": "on"
},
"use_pg_rewind": true,
"use_slots": true
}
}
3.3。クラスター メンバー: GET /cluster
目的: すべてのクラスター メンバーに関する情報を取得します。
curl -s http://10.0.1.11:8008/cluster | jq
Response:
{
"members": [
{
"name": "node1",
"role": "leader",
"state": "running",
"api_url": "http://10.0.1.11:8008/patroni",
"host": "10.0.1.11",
"port": 5432,
"timeline": 1,
"lag": 0
},
{
"name": "node2",
"role": "sync_standby",
"state": "running",
"api_url": "http://10.0.1.12:8008/patroni",
"host": "10.0.1.12",
"port": 5432,
"timeline": 1,
"lag": 0
},
{
"name": "node3",
"role": "replica",
"state": "running",
"api_url": "http://10.0.1.13:8008/patroni",
"host": "10.0.1.13",
"port": 5432,
"timeline": 1,
"lag": 0
}
],
"scope": "postgres"
}
3.4。フェイルオーバー履歴: GET /history
目的: クラスターのフェイルオーバー/スイッチオーバー履歴を取得します。
curl -s http://10.0.1.11:8008/history | jq
Response:
[
[
1, // Timeline
67108864, // LSN
"no recovery target specified",
"2024-11-25T10:30:00+00:00"
],
[
2,
134217728,
"no recovery target specified",
"2024-11-25T11:45:30+00:00"
]
]
4。ロード バランサーの統合
4.1。 HAProxy 構成
haproxy.cfg:
global
log /dev/log local0
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
Stats page
listen stats
bind *:7000
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:password
Primary/Write endpoint
listen postgres-primary
bind *:5000
mode tcp
option tcplog
option tcp-check
# Health check via Patroni REST API
tcp-check connect port 8008
tcp-check send GET\ /primary\ HTTP/1.0\r\n\r\n
tcp-check expect string HTTP/1.1\ 200
default-server inter 3s fall 3 rise 2
server node1 10.0.1.11:5432 check port 8008
server node2 10.0.1.12:5432 check port 8008
server node3 10.0.1.13:5432 check port 8008
Replicas/Read-only endpoint
listen postgres-replicas
bind *:5001
mode tcp
option tcplog
option tcp-check
balance roundrobin
# Health check via Patroni REST API
tcp-check connect port 8008
tcp-check send GET\ /replica\ HTTP/1.0\r\n\r\n
tcp-check expect string HTTP/1.1\ 200
default-server inter 3s fall 3 rise 2
server node1 10.0.1.11:5432 check port 8008
server node2 10.0.1.12:5432 check port 8008
server node3 10.0.1.13:5432 check port 8008
Read-write endpoint (primary only)
listen postgres-read-write
bind *:5002
mode tcp
option tcplog
option tcp-check
tcp-check connect port 8008
tcp-check send GET\ /read-write\ HTTP/1.0\r\n\r\n
tcp-check expect string HTTP/1.1\ 200
default-server inter 3s fall 3 rise 2
server node1 10.0.1.11:5432 check port 8008
server node2 10.0.1.12:5432 check port 8008
server node3 10.0.1.13:5432 check port 8008
Read-only endpoint (replicas only)
listen postgres-read-only
bind *:5003
mode tcp
option tcplog
option tcp-check
balance leastconn
tcp-check connect port 8008
tcp-check send GET\ /read-only\ HTTP/1.0\r\n\r\n
tcp-check expect string HTTP/1.1\ 200
default-server inter 3s fall 3 rise 2
server node1 10.0.1.11:5432 check port 8008
server node2 10.0.1.12:5432 check port 8008
server node3 10.0.1.13:5432 check port 8008
インストールして開始HAProxy:
# Install
sudo apt install -y haproxy
Configure
sudo nano /etc/haproxy/haproxy.cfg
(paste config above)
Validate config
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
Start
sudo systemctl restart haproxy
sudo systemctl enable haproxy
Check status
sudo systemctl status haproxy
HAProxy_:
# Connect to primary (port 5000)
psql -h haproxy_host -p 5000 -U app_user -d myapp -c "SELECT pg_is_in_recovery();"
Should return: f (false = primary)
Connect to replica (port 5001)
psql -h haproxy_host -p 5001 -U app_user -d myapp -c "SELECT pg_is_in_recovery();"
Should return: t (true = replica)
View HAProxy stats
curl http://haproxy_host:7000/stats
Or open in browser: http://haproxy_host:7000/stats
4.2 をテストします。 Nginx (ストリーム モジュールあり)
nginx.conf:
stream {
# Upstream for primary
upstream postgres_primary {
least_conn;
server 10.0.1.11:5432 max_fails=3 fail_timeout=10s;
server 10.0.1.12:5432 max_fails=3 fail_timeout=10s backup;
server 10.0.1.13:5432 max_fails=3 fail_timeout=10s backup;
}
# Upstream for replicas
upstream postgres_replicas {
least_conn;
server 10.0.1.11:5432 max_fails=3 fail_timeout=10s;
server 10.0.1.12:5432 max_fails=3 fail_timeout=10s;
server 10.0.1.13:5432 max_fails=3 fail_timeout=10s;
}
# Primary endpoint
server {
listen 5000;
proxy_pass postgres_primary;
proxy_connect_timeout 5s;
proxy_timeout 300s;
}
# Replicas endpoint
server {
listen 5001;
proxy_pass postgres_replicas;
proxy_connect_timeout 5s;
proxy_timeout 300s;
}
}
注: Nginx ストリーム モジュール ではありませんHTTP ヘルスチェック を直接サポートします。外部スクリプトが必要か、代わりに HAProxy を使用してください。
4.3。外部 LB のヘルスチェック スクリプト
クラウド ロード バランサーのスクリプト (AWS ALB、GCP LB、など):_
#!/bin/bash
/usr/local/bin/patroni_health_check.sh
set -e
NODE_IP="$1"
PORT="${2:-8008}"
ENDPOINT="${3:-/primary}" # or /replica
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "http://${NODE_IP}:${PORT}${ENDPOINT}")
if [ "$RESPONSE" = "200" ]; then
echo "Healthy"
exit 0
else
echo "Unhealthy (HTTP $RESPONSE)"
exit 1
fi
使用法:_
# Check if node is primary
./patroni_health_check.sh 10.0.1.11 8008 /primary
Check if node is replica
./patroni_health_check.sh 10.0.1.12 8008 /replica
5。モニタリングの統合
5.1。 Prometheus エクスポーター_
カスタム クエリで postgres_exporter を使用:HTMLTAG_448_
# Install postgres_exporter
wget https://github.com/prometheus-community/postgres_exporter/releases/download/v0.15.0/postgres_exporter-0.15.0.linux-amd64.tar.gz
tar -xzf postgres_exporter-0.15.0.linux-amd64.tar.gz
sudo mv postgres_exporter-0.15.0.linux-amd64/postgres_exporter /usr/local/bin/
Create systemd service
sudo tee /etc/systemd/system/postgres_exporter.service > /dev/null << EOF
[Unit]
Description=PostgreSQL Exporter
After=network.target
[Service]
Type=simple
User=postgres
Environment="DATA_SOURCE_NAME=postgresql://exporter:password@localhost:5432/postgres?sslmode=disable"
ExecStart=/usr/local/bin/postgres_exporter
Restart=always
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl start postgres_exporter
sudo systemctl enable postgres_exporter
Patroni のカスタム クエリメトリクス_:_
# /etc/postgres_exporter/queries.yaml
patroni_info:
query: |
SELECT
CASE WHEN pg_is_in_recovery() THEN 'replica' ELSE 'primary' END as role,
1 as value
metrics:
- role:
usage: "LABEL"
description: "PostgreSQL role"
- value:
usage: "GAUGE"
description: "Node role indicator"
5.2。カスタム監視スクリプト
REST API を使用した Python スクリプト:
#!/usr/bin/env python3
/usr/local/bin/patroni_monitor.py
import requests
import json
import sys
NODES = [
"http://10.0.1.11:8008",
"http://10.0.1.12:8008",
"http://10.0.1.13:8008"
]
def check_cluster():
results = []
for node_url in NODES:
try:
response = requests.get(f"{node_url}/patroni", timeout=5)
data = response.json()
results.append({
"node": data["patroni"]["name"],
"role": data["role"],
"state": data["state"],
"timeline": data["timeline"],
"lag": data.get("xlog", {}).get("replayed_location", 0)
})
except Exception as e:
print(f"Error checking {node_url}: {e}", file=sys.stderr)
results.append({
"node": node_url,
"role": "unknown",
"state": "unreachable",
"error": str(e)
})
return results
def main():
cluster_status = check_cluster()
print(json.dumps(cluster_status, indent=2))
# Check if we have a leader
leaders = [n for n in cluster_status if n.get("role") == "master"]
if len(leaders) != 1:
print(f"ERROR: Expected 1 leader, found {len(leaders)}", file=sys.stderr)
sys.exit(1)
# Check all nodes reachable
unreachable = [n for n in cluster_status if n.get("state") == "unreachable"]
if unreachable:
print(f"WARNING: {len(unreachable)} nodes unreachable", file=sys.stderr)
sys.exit(1)
print("Cluster is healthy")
sys.exit(0)
if name == "main":
main()
実行モニタリング:
python3 /usr/local/bin/patroni_monitor.py
Output:
[
{
"node": "node1",
"role": "master",
"state": "running",
"timeline": 1,
"lag": 0
},
{
"node": "node2",
"role": "replica",
"state": "running",
"timeline": 1,
"lag": 0
},
{
"node": "node3",
"role": "replica",
"state": "running",
"timeline": 1,
"lag": 0
}
]
Cluster is healthy
5.3。グラファナダッシュボードd クエリの例_
PromQL クエリ:
# Node role
patroni_info{role="primary"}
Replication lag
pg_stat_replication_replay_lag_seconds
Timeline
patroni_timeline
Number of replicas
count(patroni_info{role="replica"})
Synchronous replica status
patroni_sync_state{sync_state="sync"}
6。安全な REST API_
6.1。認証を有効にする_
patroni.yml内:
restapi:
listen: 0.0.0.0:8008
connect_address: 10.0.1.11:8008
Basic authentication
authentication:
username: admin
password: secure_password_here
でアクセス認証_:_
# Using curl
curl -u admin:secure_password_here http://10.0.1.11:8008/patroni
Or with header
curl -H "Authorization: Basic $(echo -n admin:secure_password_here | base64)"
http://10.0.1.11:8008/patroni
6.2。 SSL/TLS を有効にする
証明書を生成:
# Create CA
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt
-subj "/CN=Patroni-CA"
Create server certificate
openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr
-subj "/CN=node1.example.com"
Sign with CA
openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key
-set_serial 01 -out server.crt
Set permissions
sudo chown postgres:postgres server.key server.crt ca.crt
sudo chmod 600 server.key
で構成するpatroni.yml_:
restapi:
listen: 0.0.0.0:8008
connect_address: 10.0.1.11:8008
certfile: /etc/patroni/certs/server.crt
keyfile: /etc/patroni/certs/server.key
cafile: /etc/patroni/certs/ca.crt
Optional: Require client certificates
verify_client: required
authentication:
username: admin
password: secure_password_here
HTTPS によるアクセス_:
curl -k -u admin:secure_password_here https://10.0.1.11:8008/patroni
Or with CA certificate
curl --cacert /etc/patroni/certs/ca.crt
-u admin:secure_password_here
https://10.0.1.11:8008/patroni
6.3。ファイアウォール ルール_
# Allow REST API only from specific IPs
sudo ufw allow from 10.0.1.0/24 to any port 8008
sudo ufw allow from <load_balancer_ip> to any port 8008
sudo ufw allow from <monitoring_server_ip> to any port 8008
Deny from everywhere else
sudo ufw deny 8008
7。高度な REST API の使用法_
7.1。スクリプト化されたフェイルオーバー チェック
#!/bin/bash
Check if failover is safe
CLUSTER_URL="http://10.0.1.11:8008/cluster"
Get cluster info
CLUSTER_DATA=$(curl -s "$CLUSTER_URL")
Count healthy replicas
HEALTHY_REPLICAS=$(echo "$CLUSTER_DATA" | jq '[.members[] | select(.role != "leader" and .state == "running")] | length')
if [ "$HEALTHY_REPLICAS" -ge 1 ]; then
echo "Safe to failover: $HEALTHY_REPLICAS healthy replicas"
exit 0
else
echo "NOT safe to failover: only $HEALTHY_REPLICAS healthy replicas"
exit 1
fi
7.2。プライマリ エンドポイントを動的に取得
#!/bin/bash
Get current primary IP:port
get_primary() {
for NODE in 10.0.1.11 10.0.1.12 10.0.1.13; do
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "http://${NODE}:8008/primary")
if [ "$RESPONSE" = "200" ]; then
echo "${NODE}:5432"
return 0
fi
done
echo "No primary found" >&2
return 1
}
PRIMARY=$(get_primary)
echo "Current primary: $PRIMARY"
Use in connection string
psql "host=$(echo $PRIMARY | cut -d: -f1) port=5432 user=app_user dbname=myapp"
7.3。レプリケーションの遅延_
#!/bin/bash
Alert if replication lag > threshold
THRESHOLD_MB=100
for NODE in 10.0.1.11 10.0.1.12 10.0.1.13; do
LAG=$(curl -s "http://${NODE}:8008/patroni" | jq '.replication[]? | select(.sync_state != "sync") | .replay_lag' | wc -l)
if [ "$LAG" -gt "$THRESHOLD_MB" ]; then
echo "ALERT: Node $NODE replication lag > ${THRESHOLD_MB}MB"
# Send notification
fi
done
8を監視します。ラボ演習
ラボ 1: REST API エンドポイントを調べる
タスク:
- それぞれのすべてのエンドポイントをクエリするノード_
- プライマリとレプリカ間の応答を比較
- プライマリとレプリカでどちらのエンドポイントが 200 を返すかを特定
# Test script
for ENDPOINT in / /primary /replica /read-write /read-only /health /patroni; do
echo "=== $ENDPOINT ==="
for NODE in 10.0.1.11 10.0.1.12 10.0.1.13; do
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "http://${NODE}:8008${ENDPOINT}")
echo " Node $NODE: $HTTP_CODE"
done
done
ラボ 2: セットアップHAProxy
タスク:
- HAProxy をインストール_
- Patroni ヘルスで構成するチェック_
- 書き込みトラフィックをプライマリのみに送信
- レプリカに分散した読み取りトラフィックをテスト_
- フェイルオーバーをトリガーし、HAProxy リダイレクトが自動的に確認する
ラボ 3: モニタリングを作成するダッシュボード
タスク:
- すべてのノードをクエリするPythonスクリプトの作成_
- クラスタートポロジの表示_
- レプリケーションの表示ラグ_
- 現在のプライマリを強調表示_
- 5 秒ごとに実行
ラボ 4: 安全な REST API_
タスク:
- 基本認証を有効にする
- SSL証明書を生成
- 構成HTTPS_
- auth + SSL を使用するようにcurlコマンドを更新_
- ファイアウォールルールを構成
9。 REST API
9.1 のトラブルシューティング。 REST API が応答しません
Check:
# 1. Verify Patroni is running
sudo systemctl status patroni
2. Check if port is listening
sudo netstat -tlnp | grep 8008
3. Check firewall
sudo ufw status | grep 8008
4. Test locally
5. Check logs
sudo journalctl -u patroni -n 50 | grep -i rest
9.2。間違った HTTP コードが返されました
デバッグ:
コードブロック_369.3。 SSL/TLS エラー
Check:
# Verify certificate
openssl x509 -in /etc/patroni/certs/server.crt -text -noout
Check certificate matches key
openssl x509 -modulus -noout -in server.crt | md5sum
openssl rsa -modulus -noout -in server.key | md5sum
Should match
Test SSL connection
openssl s_client -connect 10.0.1.11:8008 -CAfile ca.crt
10。概要
主要エンドポイントの概要
_エンドポイント_ 使用時に200を返す_ 使用ケース_ /プライマリ_ノードはプライマリ LB書き込みルーティング_ /レプリカ_ノードはレプリカ LB読み取りルーティング /読み取り/書き込みノードは書き込みを受け入れる 書き込みエンドポイント_ /読み取り専用ノードは読み取り専用レプリカです 読み取りエンドポイント_ /healthノードは正常です 詳細モニタリング /patroni常に (詳細情報) 詳細モニタリング /クラスター_常に (すべてのメンバー) トポロジ表示_
統合チェックリスト
- すべてのノードからアクセス可能な REST API_
- ヘルスチェックが設定された HAProxy
- システム クエリ REST のモニタリングAPI
- 認証有効
- SSL/TLS 設定済み (本番環境)
- ファイアウォールルール設定済み
- ヘルスチェックスクリプトテスト済み
現在のアーキテクチャ
✅ 3 VMs prepared (Bài 4)
✅ PostgreSQL 18 installed (Bài 5)
✅ etcd cluster running (Bài 6)
✅ Patroni installed (Bài 7)
✅ Patroni configured (Bài 8)
✅ Cluster bootstrapped (Bài 9)
✅ Replication configured (Bài 10)
✅ Callbacks implemented (Bài 11)
✅ REST API integrated (Bài 12)
Next: Failover management
レッスン 13 の準備
レッスン 13 ではフェイルオーバーとスイッチオーバー_:
- 自動フェイルオーバープロセス
- 手動スイッチオーバー_
- フェイルオーバーシナリオとテスト
- リーダーにおけるDCSの役割選挙_
- ダウンタイムを最小限に抑える戦略
curl -s http://10.0.1.11:8008/read-write
Returns 200 if:
- Node is primary
- Cluster is not paused
- No maintenance mode
curl -s http://10.0.1.12:8008/read-only
Returns 200 if:
- Node is replica
- PostgreSQL is running
- Replication lag < threshold (optional)
# Check replica with max 1MB lag tolerance
curl -s "http://10.0.1.12:8008/read-only?lag=1048576"
Returns 503 if lag > 1MB
curl -s http://10.0.1.12:8008/synchronous
Returns 200 if:
- Node is replica
- sync_state = 'sync' (from pg_stat_replication)
curl -s http://10.0.1.13:8008/asynchronous
Returns 200 if:
- Node is replica
- sync_state != 'sync'
curl -s http://10.0.1.11:8008/health | jq
Response:
{
"state": "running",
"role": "master",
"server_version": 180000,
"cluster_unlocked": false,
"timeline": 1,
"database_system_identifier": "7001234567890123456",
"postmaster_start_time": "2024-11-25 10:30:00.123456+00:00",
"patroni": {
"version": "3.2.0",
"scope": "postgres",
"name": "node1"
},
"replication": [
{
"usename": "replicator",
"application_name": "node2",
"client_addr": "10.0.1.12",
"state": "streaming",
"sync_state": "sync",
"sync_priority": 1
},
{
"usename": "replicator",
"application_name": "node3",
"client_addr": "10.0.1.13",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
]
}
curl -s http://10.0.1.11:8008/patroni | jq
Response (truncated):
{
"state": "running",
"postmaster_start_time": "2024-11-25 10:30:00.123456+00:00",
"role": "master",
"server_version": 180000,
"xlog": {
"location": 67108864
},
"timeline": 1,
"cluster_unlocked": false,
"database_system_identifier": "7001234567890123456",
"patroni": {
"version": "3.2.0",
"scope": "postgres",
"name": "node1"
},
"dcs": {
"last_seen": 1700912345,
"ttl": 30
},
"tags": {
"nofailover": false,
"noloadbalance": false,
"clonefrom": false,
"nosync": false
},
"pending_restart": false,
"replication": [...],
"timeline_history": [...]
}
curl -s http://10.0.1.11:8008/config | jq
Response:
{
"ttl": 30,
"loop_wait": 10,
"retry_timeout": 10,
"maximum_lag_on_failover": 1048576,
"synchronous_mode": true,
"synchronous_mode_strict": false,
"postgresql": {
"parameters": {
"max_connections": 100,
"shared_buffers": "256MB",
"wal_level": "replica",
"max_wal_senders": 10,
"max_replication_slots": 10,
"hot_standby": "on"
},
"use_pg_rewind": true,
"use_slots": true
}
}
curl -s http://10.0.1.11:8008/cluster | jq
Response:
{
"members": [
{
"name": "node1",
"role": "leader",
"state": "running",
"api_url": "http://10.0.1.11:8008/patroni",
"host": "10.0.1.11",
"port": 5432,
"timeline": 1,
"lag": 0
},
{
"name": "node2",
"role": "sync_standby",
"state": "running",
"api_url": "http://10.0.1.12:8008/patroni",
"host": "10.0.1.12",
"port": 5432,
"timeline": 1,
"lag": 0
},
{
"name": "node3",
"role": "replica",
"state": "running",
"api_url": "http://10.0.1.13:8008/patroni",
"host": "10.0.1.13",
"port": 5432,
"timeline": 1,
"lag": 0
}
],
"scope": "postgres"
}
curl -s http://10.0.1.11:8008/history | jq
Response:
[
[
1, // Timeline
67108864, // LSN
"no recovery target specified",
"2024-11-25T10:30:00+00:00"
],
[
2,
134217728,
"no recovery target specified",
"2024-11-25T11:45:30+00:00"
]
]
global
log /dev/log local0
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
Stats page
listen stats
bind *:7000
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:password
Primary/Write endpoint
listen postgres-primary
bind *:5000
mode tcp
option tcplog
option tcp-check
# Health check via Patroni REST API
tcp-check connect port 8008
tcp-check send GET\ /primary\ HTTP/1.0\r\n\r\n
tcp-check expect string HTTP/1.1\ 200
default-server inter 3s fall 3 rise 2
server node1 10.0.1.11:5432 check port 8008
server node2 10.0.1.12:5432 check port 8008
server node3 10.0.1.13:5432 check port 8008
Replicas/Read-only endpoint
listen postgres-replicas
bind *:5001
mode tcp
option tcplog
option tcp-check
balance roundrobin
# Health check via Patroni REST API
tcp-check connect port 8008
tcp-check send GET\ /replica\ HTTP/1.0\r\n\r\n
tcp-check expect string HTTP/1.1\ 200
default-server inter 3s fall 3 rise 2
server node1 10.0.1.11:5432 check port 8008
server node2 10.0.1.12:5432 check port 8008
server node3 10.0.1.13:5432 check port 8008
Read-write endpoint (primary only)
listen postgres-read-write
bind *:5002
mode tcp
option tcplog
option tcp-check
tcp-check connect port 8008
tcp-check send GET\ /read-write\ HTTP/1.0\r\n\r\n
tcp-check expect string HTTP/1.1\ 200
default-server inter 3s fall 3 rise 2
server node1 10.0.1.11:5432 check port 8008
server node2 10.0.1.12:5432 check port 8008
server node3 10.0.1.13:5432 check port 8008
Read-only endpoint (replicas only)
listen postgres-read-only
bind *:5003
mode tcp
option tcplog
option tcp-check
balance leastconn
tcp-check connect port 8008
tcp-check send GET\ /read-only\ HTTP/1.0\r\n\r\n
tcp-check expect string HTTP/1.1\ 200
default-server inter 3s fall 3 rise 2
server node1 10.0.1.11:5432 check port 8008
server node2 10.0.1.12:5432 check port 8008
server node3 10.0.1.13:5432 check port 8008
# Install
sudo apt install -y haproxy
Configure
sudo nano /etc/haproxy/haproxy.cfg
(paste config above)
Validate config
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
Start
sudo systemctl restart haproxy
sudo systemctl enable haproxy
Check status
sudo systemctl status haproxy
# Connect to primary (port 5000)
psql -h haproxy_host -p 5000 -U app_user -d myapp -c "SELECT pg_is_in_recovery();"
Should return: f (false = primary)
Connect to replica (port 5001)
psql -h haproxy_host -p 5001 -U app_user -d myapp -c "SELECT pg_is_in_recovery();"
Should return: t (true = replica)
View HAProxy stats
curl http://haproxy_host:7000/stats
Or open in browser: http://haproxy_host:7000/stats
stream {
# Upstream for primary
upstream postgres_primary {
least_conn;
server 10.0.1.11:5432 max_fails=3 fail_timeout=10s;
server 10.0.1.12:5432 max_fails=3 fail_timeout=10s backup;
server 10.0.1.13:5432 max_fails=3 fail_timeout=10s backup;
}
# Upstream for replicas
upstream postgres_replicas {
least_conn;
server 10.0.1.11:5432 max_fails=3 fail_timeout=10s;
server 10.0.1.12:5432 max_fails=3 fail_timeout=10s;
server 10.0.1.13:5432 max_fails=3 fail_timeout=10s;
}
# Primary endpoint
server {
listen 5000;
proxy_pass postgres_primary;
proxy_connect_timeout 5s;
proxy_timeout 300s;
}
# Replicas endpoint
server {
listen 5001;
proxy_pass postgres_replicas;
proxy_connect_timeout 5s;
proxy_timeout 300s;
}
}
#!/bin/bash
/usr/local/bin/patroni_health_check.sh
set -e
NODE_IP="$1"
PORT="${2:-8008}"
ENDPOINT="${3:-/primary}" # or /replica
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "http://${NODE_IP}:${PORT}${ENDPOINT}")
if [ "$RESPONSE" = "200" ]; then
echo "Healthy"
exit 0
else
echo "Unhealthy (HTTP $RESPONSE)"
exit 1
fi
# Check if node is primary
./patroni_health_check.sh 10.0.1.11 8008 /primary
Check if node is replica
./patroni_health_check.sh 10.0.1.12 8008 /replica
# Install postgres_exporter
wget https://github.com/prometheus-community/postgres_exporter/releases/download/v0.15.0/postgres_exporter-0.15.0.linux-amd64.tar.gz
tar -xzf postgres_exporter-0.15.0.linux-amd64.tar.gz
sudo mv postgres_exporter-0.15.0.linux-amd64/postgres_exporter /usr/local/bin/
Create systemd service
sudo tee /etc/systemd/system/postgres_exporter.service > /dev/null << EOF
[Unit]
Description=PostgreSQL Exporter
After=network.target
[Service]
Type=simple
User=postgres
Environment="DATA_SOURCE_NAME=postgresql://exporter:password@localhost:5432/postgres?sslmode=disable"
ExecStart=/usr/local/bin/postgres_exporter
Restart=always
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl start postgres_exporter
sudo systemctl enable postgres_exporter
# /etc/postgres_exporter/queries.yaml
patroni_info:
query: |
SELECT
CASE WHEN pg_is_in_recovery() THEN 'replica' ELSE 'primary' END as role,
1 as value
metrics:
- role:
usage: "LABEL"
description: "PostgreSQL role"
- value:
usage: "GAUGE"
description: "Node role indicator"
#!/usr/bin/env python3
/usr/local/bin/patroni_monitor.py
import requests
import json
import sys
NODES = [
"http://10.0.1.11:8008",
"http://10.0.1.12:8008",
"http://10.0.1.13:8008"
]
def check_cluster():
results = []
for node_url in NODES:
try:
response = requests.get(f"{node_url}/patroni", timeout=5)
data = response.json()
results.append({
"node": data["patroni"]["name"],
"role": data["role"],
"state": data["state"],
"timeline": data["timeline"],
"lag": data.get("xlog", {}).get("replayed_location", 0)
})
except Exception as e:
print(f"Error checking {node_url}: {e}", file=sys.stderr)
results.append({
"node": node_url,
"role": "unknown",
"state": "unreachable",
"error": str(e)
})
return results
def main():
cluster_status = check_cluster()
print(json.dumps(cluster_status, indent=2))
# Check if we have a leader
leaders = [n for n in cluster_status if n.get("role") == "master"]
if len(leaders) != 1:
print(f"ERROR: Expected 1 leader, found {len(leaders)}", file=sys.stderr)
sys.exit(1)
# Check all nodes reachable
unreachable = [n for n in cluster_status if n.get("state") == "unreachable"]
if unreachable:
print(f"WARNING: {len(unreachable)} nodes unreachable", file=sys.stderr)
sys.exit(1)
print("Cluster is healthy")
sys.exit(0)
if name == "main":
main()
python3 /usr/local/bin/patroni_monitor.py
Output:
[
{
"node": "node1",
"role": "master",
"state": "running",
"timeline": 1,
"lag": 0
},
{
"node": "node2",
"role": "replica",
"state": "running",
"timeline": 1,
"lag": 0
},
{
"node": "node3",
"role": "replica",
"state": "running",
"timeline": 1,
"lag": 0
}
]
Cluster is healthy
# Node role
patroni_info{role="primary"}
Replication lag
pg_stat_replication_replay_lag_seconds
Timeline
patroni_timeline
Number of replicas
count(patroni_info{role="replica"})
Synchronous replica status
patroni_sync_state{sync_state="sync"}
restapi:
listen: 0.0.0.0:8008
connect_address: 10.0.1.11:8008
Basic authentication
authentication:
username: admin
password: secure_password_here
# Using curl
curl -u admin:secure_password_here http://10.0.1.11:8008/patroni
Or with header
curl -H "Authorization: Basic $(echo -n admin:secure_password_here | base64)"
http://10.0.1.11:8008/patroni
# Create CA
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt
-subj "/CN=Patroni-CA"
Create server certificate
openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr
-subj "/CN=node1.example.com"
Sign with CA
openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key
-set_serial 01 -out server.crt
Set permissions
sudo chown postgres:postgres server.key server.crt ca.crt
sudo chmod 600 server.key
restapi:
listen: 0.0.0.0:8008
connect_address: 10.0.1.11:8008
certfile: /etc/patroni/certs/server.crt
keyfile: /etc/patroni/certs/server.key
cafile: /etc/patroni/certs/ca.crt
Optional: Require client certificates
verify_client: required
authentication:
username: admin
password: secure_password_here
curl -k -u admin:secure_password_here https://10.0.1.11:8008/patroni
Or with CA certificate
curl --cacert /etc/patroni/certs/ca.crt
-u admin:secure_password_here
https://10.0.1.11:8008/patroni
# Allow REST API only from specific IPs
sudo ufw allow from 10.0.1.0/24 to any port 8008
sudo ufw allow from <load_balancer_ip> to any port 8008
sudo ufw allow from <monitoring_server_ip> to any port 8008
Deny from everywhere else
sudo ufw deny 8008
#!/bin/bash
Check if failover is safe
CLUSTER_URL="http://10.0.1.11:8008/cluster"
Get cluster info
CLUSTER_DATA=$(curl -s "$CLUSTER_URL")
Count healthy replicas
HEALTHY_REPLICAS=$(echo "$CLUSTER_DATA" | jq '[.members[] | select(.role != "leader" and .state == "running")] | length')
if [ "$HEALTHY_REPLICAS" -ge 1 ]; then
echo "Safe to failover: $HEALTHY_REPLICAS healthy replicas"
exit 0
else
echo "NOT safe to failover: only $HEALTHY_REPLICAS healthy replicas"
exit 1
fi
#!/bin/bash
Get current primary IP:port
get_primary() {
for NODE in 10.0.1.11 10.0.1.12 10.0.1.13; do
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "http://${NODE}:8008/primary")
if [ "$RESPONSE" = "200" ]; then
echo "${NODE}:5432"
return 0
fi
done
echo "No primary found" >&2
return 1
}
PRIMARY=$(get_primary)
echo "Current primary: $PRIMARY"
Use in connection string
psql "host=$(echo $PRIMARY | cut -d: -f1) port=5432 user=app_user dbname=myapp"
#!/bin/bash
Alert if replication lag > threshold
THRESHOLD_MB=100
for NODE in 10.0.1.11 10.0.1.12 10.0.1.13; do
LAG=$(curl -s "http://${NODE}:8008/patroni" | jq '.replication[]? | select(.sync_state != "sync") | .replay_lag' | wc -l)
if [ "$LAG" -gt "$THRESHOLD_MB" ]; then
echo "ALERT: Node $NODE replication lag > ${THRESHOLD_MB}MB"
# Send notification
fi
done
# Test script
for ENDPOINT in / /primary /replica /read-write /read-only /health /patroni; do
echo "=== $ENDPOINT ==="
for NODE in 10.0.1.11 10.0.1.12 10.0.1.13; do
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "http://${NODE}:8008${ENDPOINT}")
echo " Node $NODE: $HTTP_CODE"
done
done
# 1. Verify Patroni is running
sudo systemctl status patroni
2. Check if port is listening
sudo netstat -tlnp | grep 8008
3. Check firewall
sudo ufw status | grep 8008
4. Test locally
5. Check logs
sudo journalctl -u patroni -n 50 | grep -i rest
# Verify certificate
openssl x509 -in /etc/patroni/certs/server.crt -text -noout
Check certificate matches key
openssl x509 -modulus -noout -in server.crt | md5sum
openssl rsa -modulus -noout -in server.key | md5sum
Should match
Test SSL connection
openssl s_client -connect 10.0.1.11:8008 -CAfile ca.crt
/プライマリ_/レプリカ_/読み取り/書き込み/読み取り専用/health/patroni/クラスター_✅ 3 VMs prepared (Bài 4)
✅ PostgreSQL 18 installed (Bài 5)
✅ etcd cluster running (Bài 6)
✅ Patroni installed (Bài 7)
✅ Patroni configured (Bài 8)
✅ Cluster bootstrapped (Bài 9)
✅ Replication configured (Bài 10)
✅ Callbacks implemented (Bài 11)
✅ REST API integrated (Bài 12)
Next: Failover management