Background

See #2757

Todo List

Architecture

• Build github actions that running APISIX on k8s, and deploy chaos on APISIX with chaos mesh (feat: chaos test on route could still works when etcd is down #3404)

Chaos Test
overall related:

• The skywalking server is down: heartbeat in skywalking-nginx-lua would throw error at interval of 3s, so nothing to worry about
• The oauth 2.0 server is down
• Kafka server down/broker failure
• cpu/memory stressed out and see how apisix works
• The disk of apisix is full

etcd related:

• kill all etcd, see if route visit still works (feat: chaos test on route could still works when etcd is down #3404)
• kill minority of etcd, see if apisix works (feat: enable etcd health-check #4191)
• give low&high network delay between apisix and etcd, see if it works (test: add chaos test on network delay between apisix and etcd #4315)
• add network partition to etcd cluster, and see how apisix works: would be same to part of etcd failure (ref)

Could not be tested now:
DP CP have not been separated yet

• Part or all of the dp and cp networks are disconnected
• The cp node is down randomly
• The dp node is attacked by ddos

confirmed need hard work but not much benefit

• restart etcd then apisix and see how it works (resend from memory to etcd)

Welcome more ideas!😆