目標____HTMLTAG_66__HTMLTAG_67___學完本課後，您將：HTMLTAG_68__HTMLTAG_69__HTMLTAG_70___了解 Patroni中的錯誤偵測機制____HTMLTAG_71__HTMLTAG_72___了解領導者選舉過程___HTMLTAG_73__HTMLTAG_74___追蹤故障轉移時間表詳細資訊___HTMLTAG_75__HTMLTAG_76___在多種情況下測試自動故障轉移___HTMLTAG_77__HTMLTAG_78___解決故障轉移問題___HTMLTAG_79__HTMLTAG_80___最佳化故障轉移速度___HTMLTAG_81__HTMLTAG_82__HTMLTAG_83___1。自動故障轉移概述___HTMLTAG_84__HTMLTAG_85___1.1。什麼是故障轉移？ HTMLTAG_86__HTMLTAG_87自動故障轉移 =處理自動 當主副本出現時，將副本提升為主副本失敗.HTMLTAG_94__HTMLTAG_95__HTMLTAG_96___特殊點 HTMLTAG_97：HTMLTAG_98__HTMLTAG_99__HTMLTAG_100⚡自動：無須幹預手動___HTMLTAG_103__HTMLTAG_104_____ HTMLTAG_105___未計劃：由於嘗試而發生___HTMLTAG_107__HTMLTAG_108__⏱️ 快速：30-60秒（可設定）__HTMLTAG 111__HTMLTAG_112___🎯 目標：最大限度地減少停機時間____HTMLTAG_115__HTMLTAG_116__HTMLTAG_117__HTMLTAG_118___何時發生故障轉移？ ___HTMLTAG_119__HTMLTAG_120__HTMLTAG_121__HTMLTAG_122___主伺服器崩潰___HTMLTAG_123__HTMLTAG_124___PostgreSQL進程終止____HTMLTAG_125__HTMLTAG_126___網路分區___HTMLTAG_127__HTMLTAG_128___硬體失敗___HTMLTAG_129__HTMLTAG_130___DCS連線遺失___HTMLTAG_131__HTMLTAG_132___磁碟已滿___HTMLTAG_133__HTMLTAG_134__HTMLTAG_135___1.2。故障轉移與複製
WITHOUT Patroni (Manual Failover):
- Primary fails
- DBA gets paged
- DBA investigates (10-30 mins)
- DBA manually promotes replica
- DBA updates application config
- Service restored Total downtime: 30+ minutes ❌
WITH Patroni (Automatic Failover):
- Primary fails
- Patroni detects (10 seconds)
- Patroni promotes best replica (20 seconds)
Service restored automatically Total downtime: 30-60 seconds ✅
2。故障偵測機制___HTMLTAG_138__HTMLTAG_139___2.1。運行狀況檢查循環___HTMLTAG_140__HTMLTAG_141__HTMLTAG_142___Patroni 運行狀況檢查組件：___HTMLTAG_144__CODEBLOCK_1__HTMLTAG_145___2.2。 PostgreSQL 運作狀況檢查___HTMLTAG_146__HTMLTAG_147__HTMLTAG_148___Patroni 執行多項檢查:___HTMLTAG_150__HTMLTAG_151___A。進程檢查___HTMLTAG_152__CODEBLOCK_2__HTMLTAG_153___B。連線檢查___HTMLTAG_154__CODEBLOCK_3__HTMLTAG_155___C。複製檢查（副本上）
-- Check if replication is active SELECT status, received_lsn, replay_lsn FROM pg_stat_wal_receiver;
-- If no data or status != 'streaming' → Problem!
D。時間軸檢查___HTMLTAG_158__CODEBLOCK_5__HTMLTAG_159___2.3。 DCS 連線檢查___HTMLTAG_160__HTMLTAG_161__HTMLTAG_162___為什麼 DCS 連線很重要:___HTMLTAG_164__CODEBLOCK_6__HTMLTAG_165__HTMLTAG_166___DC檢查範例:____HTMLTAG_168__CODEBLOCK_7__HTMLTAG_169___2.4.領導者鎖定TTL___HTMLTAG_170__HTMLTAG_171__HTMLTAG_172___TTL（存活時間）機制：___HTMLTAG_174__CODEBLOC K_8__HTMLTAG_175__HTMLTAG_176___時間軸：____HTMLTAG_178__CODEBLOCK_9__HTMLTAG_179___3。領導者選舉流程___HTMLTAG_180__HTMLTAG_181___3.1。選舉觸發器____HTMLTAG_182__HTMLTAG_183__HTMLTAG_184___領導者選舉開始於:
___CODEBLOCK_10__HTMLTAG_187___3.2。候選選擇標準Patroni 選擇的是基於:__HTMLTAG_192__HTMLTAG_193___優先權 1：複製狀態___HTMLTAG_194__CODEBLOCK_11__HTMLTAG_195___優先權 2：複製滯後CODEBLOCG_196___CODEBLOC 3：時間軸____HTMLTAG_198__CODEBLOCK_13__HTMLTAG_199___優先權4：標籤____HTMLTAG_200__CODEBLOCK_14__HTMLTAG_201__HTMLTAG_202___範例：
___CODEBLOCK_15__HTMLTAG_205___優先權5：同步State-- Synchronous replica preferred over async SELECT sync_state FROM pg_stat_replication;
sync > potential > async
3.3.競爭條件與鎖定取得___HTMLTAG_208__HTMLTAG_209__HTMLTAG_210___多個副本競爭__ HTMLTAG_211__:___HTMLTAG_212__CODEBLOCK_17__HTMLTAG_213__HTMLTAG_214___DCS保證:_HTMLTAG_216__HTMLTAG_217 HTMLTAG_218__HTMLTAG_219___原子性：只有一個節點獲得鎖定___HTMLTAG_221__HTMLTAG_222__HTMLTAG_223___一致性：所有節點看到相同的領導___HTMLTAG_225__HTMLTAG_226__HTMLTAG_227___隔離：不可能出現裂腦___HTMLTAG_229__HTMLTAG_230__HTMLTAG_231___3.4。促銷過程___HTMLTAG_232__HTMLTAG_233__HTMLTAG_234___獲勝者節點執行:___HTMLTAG_236__CODEBLOCK_18__HTMLTAG_237___4。 Failover Timeline Detailed
4.1. Complete Failover Flow
Timeline of Automatic FailoverT+0s: NORMAL OPERATION Primary (node1): Healthy, serving requests Replica (node2): Streaming from node1, lag=0 Replica (node3): Streaming from node1, lag=0
T+1s: PRIMARY FAILS node1: PostgreSQL crashes / server dies node2: Still streaming (buffered data) node3: Still streaming (buffered data)
T+5s: REPLICATION BROKEN node2: WAL receiver error "connection lost" node3: WAL receiver error "connection lost" node1: Still holds leader lock (TTL not expired yet)
T+10s: HEALTH CHECK CYCLE 1 node2: Check replication → FAILED, wait... node3: Check replication → FAILED, wait... node1: Cannot renew lock (crashed)
T+20s: HEALTH CHECK CYCLE 2 node2: Still cannot connect to node1 node3: Still cannot connect to node1
T+30s: LEADER LOCK EXPIRES DCS: /service/postgres/leader TTL expired → key deleted node2: Detects no leader key node3: Detects no leader key
T+31s: CANDIDATE ELECTION BEGINS node2: Check eligibility → YES (lag=0, priority=100) node3: Check eligibility → YES (lag=1MB, priority=100)
T+32s: RACE FOR LOCK node2: PUT /service/postgres/leader "node2" → SUCCESS node3: PUT /service/postgres/leader "node3" → FAILED
T+33s: NODE2 PROMOTES node2: Run pre_promote callback node2: pg_promote() executed node2: Timeline: 1 → 2
T+35s: PROMOTION COMPLETE node2: pg_is_in_recovery() → false node2: Now accepting writes node2: Run post_promote & on_role_change callbacks
T+36s: NODE3 RECONFIGURES node3: Detects new leader = node2 node3: Update primary_conninfo → node2:5432 node3: Restart WAL receiver
T+38s: REPLICATION RESTORED node3: Connected to node2 node3: Streaming at timeline 2
T+40s: CLUSTER OPERATIONAL Primary: node2 (was replica) Replica: node3 (following node2) Failed: node1 (needs manual intervention)
Total Downtime: ~35-40 seconds ✅
4.2.影響故障轉移速度的因素___HTMLTAG_242__HTMLTAG_243__HTMLTAG_244___配置參數:___HTMLTAG_246__CODEBLOCK_20__HTMLTAG_247__HTMLTAG_248___權衡:
HTMLTAG_252__HTMLTAG_253__HTMLTAG_254__HTMLTAG_255___參數____HTMLTAG_256__HTMLTAG_257____較低值___H TMLTAG_258__HTMLTAG_259___較高值值___HTMLTAG_260__HTMLTAG_261__HTMLTAG_262__HTMLTAG_263__HTMLTAG_264 HTMLTAG_265__HTMLTAG_266____TTL____HTMLTAG_267__HTMLTAG_268__HTMLTAG_269___更快的故障轉移___HTMLTAG_270_ HTMLTAG_271___更多穩定___HTMLTAG_272__HTMLTAG_273__HTMLTAG_274__HTMLTAG_275__HTMLTAG_276__HTMLTAG_277_ 更多誤報____HTMLTAG_278__HTMLTAG_279___較慢故障轉移____HTMLTAG_280__HTMLTAG_281__HTMLTAG_282__HTMLTAG_283 HTMLTAG_284____loop_wait___HTMLTAG_285__HTMLTAG_286__HTMLTAG_287____更快的偵測____HTMLTAG_288__HTMLTAG 289___更少的DCS流量___HTMLTAG_290__HTMLTAG_291__HTMLTAG_292__HTMLTAG_293__HTMLTAG_294__HTMLTAG_295___更多CPU/網路____HTMLTAG_296__HTMLTAG_297___反應較慢___HTMLTAG_298__HTMLTAG_299__HTMLTAG_300__HTMLTAG_301
Typical configurations:
# Conservative (stable, slower) ttl: 30 loop_wait: 10 → Failover: ~40-50sBalanced (recommended)
ttl: 20 loop_wait: 10 → Failover: ~30-40s
Aggressive (fast, sensitive)
ttl: 15 loop_wait: 5 → Failover: ~20-30s
5. Testing Automation Failover5. Testing Automation Failover
___PostG_308150.進程終止___HTMLTAG_310__HTMLTAG_311__HTMLTAG_312___模擬 PostgreSQL崩潰:____HTMLTAG_314__CODEBLOCK_22__HTMLTAG_315__HTMLTAG_316____監控故障轉移:___HTMLTAG_318 __CODEBLOCK_23__HTMLTAG_319__HTMLTAG_320___預期時間軸:___CODEBLOCK_24__HTMLTAG_323___5.2. Test Scenario 2: Network Partition
5. Testing Automation Failover
___PostG_308150.進程終止___HTMLTAG_310__HTMLTAG_311__HTMLTAG_312___模擬 PostgreSQL崩潰:____HTMLTAG_314__CODEBLOCK_22__HTMLTAG_315__HTMLTAG_316____監控故障轉移:___HTMLTAG_318 __CODEBLOCK_23__HTMLTAG_319__HTMLTAG_320___預期時間軸:___CODEBLOCK_24__HTMLTAG_323___5.2. Test Scenario 2: Network PartitionSimulate network partition:___HTMLTAG_328__CODEBLOCK_25__HTMLTAG_329__HTMLTAG_330___觀察:_HTMLTAG_332__CODEBLOCK_26 ____HTMLTAG_333__HTMLTAG_334___恢復:___HTMLTAG_336__CODEBLOCK_27__HTMLTAG_337___5.3。測試情境 3：伺服器重新啟動___HTMLTAG_338__HTMLTAG_339__HTMLTAG_340___模擬伺服器崩潰:___HTMLTAG_342__CODEBLOCK_28__HTMLTAG_343__HTMLTAG_342__CODEBLOCK_28__HTMLTAG_343__HTMLTAG_344_______455KML unavailable.
5.4.測試場景4：磁碟已滿___HTMLTAG_348__HTMLTAG_349__HTMLTAG_350___模擬磁碟已滿:___HTMLTAG_352__CODEBLOCK_29__HTMLTAG_353__HTMLTAG_352__CODEBLOCK_29__HTMLTAG_353__HTMLTAG_352__COPatroni 將檢測不健康觸發故障轉移.___HTMLTAG_356__HTMLTAG_357___5.5。測試情境 5：DCS 故障___HTMLTAG_358__HTMLTAG_359__HTMLTAG_360___在所有節點上停止etcd:___HTMLTAG_362__CODEBLOCK_30__HTMLTAG_363__HTMLTAG_364___預期行為：___HTMLTAG_366__CODEBLOCK_31__HTMLTAG_367___6。驗證故障轉移是否成功___HTMLTAG_368__HTMLTAG_369___6.1。 Check cluster status
# List cluster members patronictl list postgresExpected after failover:
+ Cluster: postgres (7001234567890123456) ----+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------+---------------+---------+---------+----+-----------+
| node1 | 10.0.1.11:5432| Replica | stopped | 1 | | ← Old primary
| node2 | 10.0.1.12:5432| Leader | running | 2 | | ← NEW primary
| node3 | 10.0.1.13:5432| Replica | running | 2 | 0 |
+--------+---------------+---------+---------+----+-----------+
Note timeline changed: 1 → 2
6.2.驗證新的主要___HTMLTAG_372__CODEBLOCK_33__HTMLTAG_373___6.3。測試寫入操作___HTMLTAG_374__CODEBLOCK_34__HTMLTAG_375___6.4。 Check failover history
# View history via REST API curl -s http://10.0.1.12:8008/history | jqOutput:
[
[1, 67108864, "no recovery target specified", "2024-11-25T10:00:00+00:00"],
[2, 134217728, "no recovery target specified", "2024-11-25T11:30:15+00:00"]
]
↑ Timeline 2 = Failover event
Check Patroni logs
sudo journalctl -u patroni --since "30 minutes ago" | grep -i "promote|failover|leader"