/gke-ai-troubleshooting-tpu-performance-degradation
用 ML Diagnostics Workload Monitoring 與 Cloud Monitoring 系統指標，診斷 GKE Cloud TPU 訓練吞吐量下降與 step time 退化（TPU 使用率掉 15% 以上）。
- 安裝次數
- 1100
- 專案星數
- 2.2 萬 ★
- 授權
- Apache-2.0
- 專案更新
- 2026-11-02
怎麼裝
npx skills add https://github.com/google/skills --skill gke-ai-troubleshooting-tpu-performance-degradation安裝前請先看過原文內容。中文說明是 AI 翻譯，不保證安全。
原文
Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).