monitoring-ingestion-pipeline✓ تایید شده

محتوای SKILL.md

---
name: monitoring-ingestion-pipeline
description: >
  Guide for using the Grafana MCP to monitor and diagnose the Node.js ingestion
  pipeline workers in production. Use when investigating event lag, drops,
  pipeline errors, person/group processing, Kafka consumer health, Redis,
  Postgres, ClickHouse downstream health, or any ingestion worker question.
  Covers prod-us and prod-eu environments.
---

# Monitoring the ingestion pipeline with Grafana MCP

The ingestion pipeline (`nodejs/`) is PostHog's Node.js event processing layer.
It consumes events from Kafka (produced by the capture service), runs them through
processing steps (person resolution, group assignment, property overrides, etc.),
and produces enriched events to ClickHouse-bound Kafka topics.

A single codebase is deployed as **many K8s Deployments** via the `posthog-app`
Helm chart (golden-chart migration). Each deployment sets `PLUGIN_SERVER_MODE`
and is distinguished in metrics by two default Prometheus labels:

- `ingestion_pipeline` — values: `analytics`, `heatmaps`, `clientwarnings`, `errortracking`
- `ingestion_lane` — values: `main`, `overflow`, `historical`, `async`, `turbo`

The `app` label (set by K8s pod labels) matches the deployment name and is the most
universal scope filter across all telemetry domains.

This skill teaches how to **discover live metrics** using the Grafana MCP tools
rather than memorizing metric names that change as the code evolves.

## Environment context

The Grafana MCP is connected to a **single Grafana instance** scoped to one environment.
If the user hasn't specified, **ask which environment** they want to investigate:

- **prod-us** — US production (us-east-1)
- **prod-eu** — EU production (eu-central-1)

Most ingestion app metrics are environment-specific by virtue of which Grafana
you're connected to — they don't carry an `environment` label. CloudWatch metrics
are also scoped to the connected Grafana's AWS account.

Cross-environment comparison requires switching Grafana instances (not possible in one session).

All datasource UIDs, dashboard UIDs, `ingestion_pipeline`/`ingestion_lane` label values,
and ClickHouse `type` label values are **identical** across prod-us and prod-eu.

Key differences:

- `ingestion-analytics-turbo` deployment exists only in **prod-us**.
- Both envs use a **dedicated ingestion MSK** cluster separate from the events cluster —
  consumers point at `msk-ingestion` not `events`
  (prod-us: c21; prod-eu: `posthog-prod-eu-ingestion-2026-05-04`).
- CloudWatch cluster IDs differ by region suffix (see topology sections below).

## Observability landscape

Six telemetry domains, all validated identical across prod-us and prod-eu:

| Domain                             | Datasource UID                  | Discovery tool                 | Scope filter                              |
| ---------------------------------- | ------------------------------- | ------------------------------ | ----------------------------------------- |
| App metrics (VictoriaMetrics)      | `victoriametrics`               | `list_prometheus_metric_names` | See metric prefixes below                 |
| App metrics (realtime)             | `victoriametrics-realtime`      | same                           | same (lower retention, higher resolution) |
| Logs                               | `P44D702D3E93867EC` (Loki-logs) | `list_loki_label_names`        | `app=~"ingestion-.*"`                     |
| Profiling                          | `pyroscope`                     | `list_pyroscope_profile_types` | See Pyroscope services below              |
| CloudWatch (ElastiCache, MSK, RDS) | `P034F075C744B399F`             | `query_prometheus`             | env-specific cluster IDs                  |
| Dashboards                         | n/a                             | `search_dashboards`            | query `"ingestion"` or deployment name    |

**Datasource notes:**

- Do NOT use primary Loki (`P8E80F9AEF21F6940`) — it returns 502 intermittently in both envs. Always use **Loki-logs** (`P44D702D3E93867EC`).
- Do NOT use `CloudWatch Root` (`PAAE47F430CFD1449`) — it exists only in prod-us.

## Stable waypoints

These facts change infrequently and are hard to discover dynamically.

### Deployment roles

All `PLUGIN_SERVER_MODE=ingestion-v2` deployments (the "analytics ingestion" family), plus specialized modes.
Each golden-chart deployment runs in its **own namespace** matching the deployment name.

| Deployment name                         | Mode                           | Pipeline         | Lane         | Consumer group                        | Consume topic (EU example)            |
| --------------------------------------- | ------------------------------ | ---------------- | ------------ | ------------------------------------- | ------------------------------------- |
| `ingestion-analytics-main`              | `ingestion-v2`                 | `analytics`      | `main`       | `ingestion-analytics-main`            | `ingestion-analytics-main-512`        |
| `ingestion-analytics-overflow`          | `ingestion-v2`                 | `analytics`      | `overflow`   | `ingestion-analytics-overflow`        | `ingestion-analytics-overflow-128`    |
| `ingestion-analytics-historical`        | `ingestion-v2`                 | `analytics`      | `historical` | `ingestion-analytics-historical`      | `ingestion-analytics-historical-128`  |
| `ingestion-analytics-async`             | `ingestion-v2`                 | `analytics`      | `async`      | `ingestion-analytics-async`           | `ingestion-analytics-async-8`         |
| `ingestion-analytics-turbo`             | `ingestion-v2`                 | `analytics`      | `turbo`      | `ingestion-analytics-turbo`           | `ingestion-analytics-turbo-1024`      |
| `ingestion-clientwarnings-main`         | `ingestion-v2`                 | `clientwarnings` | —            | `ingestion-clientwarnings-main`       | `ingestion-clientwarnings-main-32`    |
| `ingestion-heatmaps-main`               | `ingestion-v2`                 | `heatmaps`       | —            | `ingestion-heatmaps-main`             | `ingestion-heatmaps-main-128`         |
| `ingestion-errortracking-main`          | `ingestion-errortracking`      | `errortracking`  | —            | `ingestion-errortracking-main`        | `ingestion-errortracking-main-128`    |
| `ingestion-errortracking-overflow`      | `ingestion-errortracking`      | `errortracking`  | —            | `ingestion-errortracking-overflow`    | `ingestion-errortracking-overflow-32` |
| `logs-ingestion`                        | `ingestion-logs`               | —                | —            | `logs-ingestion`                      | `ingestion-logs`                      |
| `traces-ingestion`                      | (traces)                       | —                | —            | `traces-ingestion`                    | `ingestion-traces`                    |
| `recordings-blob-ingestion-v2`          | `recordings-blob-ingestion-v2` | —                | —            | `session-recordings-blob-v2`          | `ingestion-sessionreplay-main-256`    |
| `recordings-blob-ingestion-v2-overflow` | `recordings-blob-ingestion-v2` | —                | —            | `session-recordings-blob-v2-overflow` | `ingestion-sessionreplay-overflow-32` |

**Notes:**

- `ingestion-analytics-turbo` exists only in **prod-us**.
- Topic partition counts differ by env (e.g., main is 1024 in US, 512 in EU).
- Each deployment also has a DLQ topic (`ingestion-analytics-main-dlq`, etc.).
- KEDA autoscaling queries reference **both old and new** consumer group names during migration
  (e.g., `groupId=~"ingestion-events|ingestion-analytics-main"`).

### Metric prefixes

Every prefix here can be discovered live with `list_prometheus_metric_names`
using `datasourceUid: "victoriametrics"` and `regex: "<prefix>.*"`.

| Prefix                                                                           | Domain                                     | Key scope labels                                         |
| -------------------------------------------------------------------------------- | ------------------------------------------ | -------------------------------------------------------- |
| `ingestion_*`                                                                    | Core ingestion app metrics (~80 metrics)   | `app`, `ingestion_pipeline`, `ingestion_lane`            |
| `ingestion_lag_ms*`                                                              | Per-partition lag (primary lag signal)     | `groupId`, `partition`                                   |
| `consumed_batch_*`                                                               | Kafka consumer batch processing            | `topic`, `groupId`                                       |
| `consumer_batch_*` / `consumer_background_*`                                     | Consumer loop health                       | `topic`, `groupId`                                       |
| `kafka_broker_*`                                                                 | librdkafka broker stats                    | `broker_id`, `broker_name`, `consumer_group`             |
| `kafka_consumer_*`                                                               | Consumer rebalance, assignment             | `groupId`, `type`                                        |
| `events_pipeline_*`                                                              | Legacy pipeline step metrics               | `step_name`                                              |
| `person_*`                                                                       | Person processing (~30 metrics)            | `db_write_mode`, `operation`, `method`                   |
| `group_*` (non-AWS)                                                              | Group processing                           | `operation`                                              |
| `personhog_*`                                                                    | PersonHog gRPC client + service            | `method`, `source`, `client`                             |
| `overflow_redirect_*`                                                            | Stateful overflow routing                  | `type`, `result`, `decision`, `operation`                |
| `cookieless_*`                                                                   | Cookieless mode                            | —                                                        |
| `http_request_duration_seconds`                                                  | HTTP health/readiness server               | `method`, `route`, `status_code`                         |
| `recording_blob_ingestion_v2_*`                                                  | Session replay ingestion                   | `app`                                                    |
| `logs_ingestion_*`                                                               | Logs ingestion pipeline                    | `app`                                                    |
| `error_tracking_*` / `cymbal_*`                                                  | Error tracking pipeline                    | `app`                                                    |
| `kminion_kafka_*`                                                                | KMinion consumer group lag & topic offsets | `group_id`, `topic_name`, `partition_id`                 |
| `aws_msk_kafka_*`                                                                | MSK broker-side JMX metrics                | `environment`                                            |
| `warpstream_agent_*`                                                             | WarpStream agent metrics (~10 metrics)     | `virtual_cluster_id`, `agent_group`, `operation`         |
| `kube_*` / `container_*`                                                         | K8s resources                              | `namespace=~"ingestion-.*"`, `container=~"ingestion-.*"` |
| `pg_*` / `pgbouncer_*`                                                           | Postgres exporter                          | varies                                                   |
| `ClickHouseMetrics_*` / `ClickHouseProfileEvents_*` / `ClickHouseAsyncMetrics_*` | ClickHouse cluster health                  | `type` (=cluster role)                                   |
| `kafka_connect_*`                                                                | Kafka Connect bridge to ClickHouse         | `namespace`, `connector`                                 |
| `posthog_celery_clickhouse_*`                                                    | CH health monitors from Django celery      | `scenario`                                               |

### Redis topology

Ingestion workers depend on up to five Redis instances.
Redis health is inferred from ingestion-side metrics and CloudWatch ElastiCache metrics.

| Redis instance        | ElastiCache cluster (prod-us)     | Env var                    | Use                                  |
| --------------------- | --------------------------------- | -------------------------- | ------------------------------------ |
| Ingestion Redis       | `ingestion-prod-redis`            | `INGESTION_REDIS_HOST`     | Overflow state, pub/sub coordination |
| PostHog/Primary Redis | `posthog-solo`                    | `POSTHOG_REDIS_HOST`       | Billing/quota, restrictions, general |
| Cookieless Redis      | `cookieless-prod-redis`           | `COOKIELESS_REDIS_HOST`    | Cookieless server hash mode          |
| CDP Redis             | `cdp-delivery-prod-redis`         | `CDP_REDIS_HOST`           | CDP Hog function delivery            |
| Dedup Redis           | `ingestion-duplicates-prod-redis` | `DEDUPLICATION_REDIS_HOST` | Event deduplication                  |

Ingestion-side Redis metrics: `overflow_redirect_redis_*`, `cookieless_redis_error`.
Infrastructure-side: CloudWatch datasource `P034F075C744B399F`.

prod-eu uses the same logical cluster names but different endpoint suffixes.
The prod-eu primary Redis is `posthog-prod-redis-encripted` (sic — the typo is in the actual cluster name).
The event restrictions Redis (`ingestion-prod-redis`) is a **separate writable cluster** from
the primary — capture-analytics and ingestion workers both use it via `EVENT_RESTRICTIONS_REDIS_URL`.

### Kafka topology

Ingestion workers interact with **three Kafka systems** via separate producer/consumer configs:

1. **MSK ingestion cluster** (consume side) — `KAFKA_CONSUMER_METADATA_BROKER_LIST`.
   Capture produces here; ingestion workers consume. Carries all `ingestion-analytics-*`,
   `ingestion-errortracking-*`, `ingestion-heatmaps-*`, `ingestion-clientwarnings-*` topics.
   Both envs have a dedicated cluster (prod-us: c21, 12 brokers; prod-eu: `posthog-prod-eu-ingestion-2026-05-04`).
   KMinion: `kminion-msk-ingestion`.

2. **WarpStream ingestion VC** (output side) — `KAFKA_WARPSTREAM_PRODUCER_METADATA_BROKER_LIST`.
   Ingestion workers produce ALL ClickHouse-bound output here (`clickhouse_events_json`,
   `clickhouse_person`, `clickhouse_groups`, `clickhouse_heatmap_events`, `clickhouse_ai_events_json`, etc.).
   In-cluster WarpStream agents (`warpstream-ingestion-v2`) with multi-AZ pools; plaintext, no TLS.
   KMinion: `kminion-warpstream-ingestion`.

3. **MSK ingestion cluster (feedback/DLQ producer)** — same physical cluster as item 1,
   different producer config via `KAFKA_INGESTION_PRODUCER_METADATA_BROKER_LIST`.
   Used for overflow/DLQ/async topics that route events BACK to the ingestion system.

- **MSK (events/analytics)** — the original events cluster. Still carries some legacy topics and
  ClickHouse consumer groups during migration. prod-us: `posthog-prod-us-events-2026-03-08`
  (12 brokers, `kafka.m7g.8xlarge`); prod-eu: `posthog-prod-eu-events-2025-10-16` (15 brokers).
  KMinion: `kminion-msk-analytics`.

**WarpStream virtual clusters** — each VC is a separate logical cluster backed by S3, with its own
agent pool, KMinion instance, and topic namespace:

| VC name                          | Topics carried                                                                                                                                                                                                                                                                                                                       | KMinion instance                         |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- |
| `warpstream-ingestion-v2`        | `clickhouse_events_json`, `clickhouse_person`, `clickhouse_person_distinct_id`, `clickhouse_groups`, `clickhouse_ai_events_json`, `clickhouse_heatmap_events`, `clickhouse_app_metrics2`, `clickhouse_tophog`, `clickhouse_ingestion_warnings`, `distinct_id_usage_events_json`, `log_entries`, `team_event_partitioned_events_json` | `kminion-warpstream-ingestion`           |
| `warpstream-replay-v2`           | `ingestion-sessionreplay-main-*`, `clickhouse_session_replay_events`, `clickhouse_session_replay_features`                                                                                                                                                                                                                           | `kminion-warpstream-replay`              |
| `warpstream-logs`                | `ingestion-logs`, `clickhouse_logs`                                                                                                                                                                                                                                                                                                  | `kminion-warpstream-logs`                |
| `warpstream-traces`              | `ingestion-traces`, `clickhouse_traces`                                                                                                                                                                                                                                                                                              | `kminion-warpstream-traces`              |
| `warpstream-shared`              | `clickhouse_document_embeddings`, error tracking fingerprint/issue topics, `document_embeddings_input`                                                                                                                                                                                                                               | `kminion-warpstream-shared`              |
| `warpstream-calculated-events`   | `clickhouse_precalculated_person_properties`, `clickhouse_prefiltered_events`, `cohort_membership_changed` (US only)                                                                                                                                                                                                                 | `kminion-warpstream-calculated-events`   |
| `warpstream-cyclotron`           | CDP topics (`cdp_cyclotron_hog*`, `cdp_internal_events`, etc.)                                                                                                                                                                                                                                                                       | `kminion-warpstream-cyclotron`           |
| `warpstream-warehouse-pipelines` | `data_warehouse_source_webhooks`, `data_warehouse_sources_jobs` (US only)                                                                                                                                                                                                                                                            | `kminion-warpstream-warehouse-pipelines` |

WarpStream agent metrics: `warpstream_agent_*` prefix (~10 metrics).
Key: `control_plane_operation_counter` (by `operation`), `file_cache_client_fetch_local_or_remote_counter` (cache hit/miss).
Labels: `virtual_cluster_id`, `agent_group` (`default`, `general`, `multi-az`).
Dashboards: `warpstream` (Agent Overview), `dbfj5c31spa1ogf` (MSK vs WarpStream — Active Produce Topics).
US-only personal dashboards (not synced to EU): `8e93b023-…` (CH Consumer Lag),
`ws-coarse-lag-explore` (Coarse Lag exploration).

### Postgres topology

| DB          | Aurora cluster (prod-us)                                      | Ingestion PgBouncer                                     |
| ----------- | ------------------------------------------------------------- | ------------------------------------------------------- |
| Main app DB | `posthog-cloud-prod-us-east-1` (2x `db.r8g.16xlarge`)         | `ingestion-default-pgbouncer.posthog.svc.cluster.local` |
| Persons DB  | `posthog-cloud-persons-prod-us-east-1` (3x `db.r8g.24xlarge`) | `ingestion-events-pgbouncer.posthog.svc.cluster.local`  |

Postgres metrics via `prometheus-postgres-exporter` and `prometheus-postgres-persons-exporter`.

prod-eu: `posthog-cloud-prod-eu-central-1` and `posthog-cloud-persons-prod-eu-central-1`.

### ClickHouse topology

ClickHouse is the **ultimate downstream consumer** of events the ingestion pipeline produces.
Ingestion workers never talk to CH directly — they publish to Kafka topics which CH
consumes via its built-in Kafka engine and Kafka Connect (DuckLake).
CH health directly impacts perceived ingestion quality: if CH falls behind on consumption,
users see stale data.

**Cluster roles** (discovered via `type` label on `ClickHouseMetrics_*`):

| `type` label      | Role                          | Notes                                      |
| ----------------- | ----------------------------- | ------------------------------------------ |
| `events`          | Main analytics events cluster | Consumes `clickhouse_events_json`          |
| `online`          | Online/fast queries cluster   | Replicated from events                     |
| `offline`         | Offline/batch queries cluster | Replicated from events                     |
| `medium`          | Medium-sized tables           | Persons, groups                            |
| `small`           | Small/config tables           | Infrequent writes                          |
| `sessions`        | Session replay data           | Consumes session recording topics          |
| `logs`            | Logs cluster                  | Consumes logs topics                       |
| `logs-new-schema` | Logs new schema migration     | Migration target                           |
| `ai-events`       | AI/LLM events                 | Consumes AI events topics                  |
| `endpoints`       | API endpoints cluster         | Lightweight                                |
| `migrations`      | Migration-specific            | Schema changes                             |
| `aux` / `ops`     | Auxiliary/operations          | Maintenance                                |
| `batch-exports`   | Batch exports                 | prod-us has this; may not exist in prod-eu |
| `test`            | Testing cluster               | May not exist in all envs                  |

Most `type` label values are identical across prod-us and prod-eu. Minor differences
like `batch-exports` or `test` may exist only in one env.

**Two consumption paths from Kafka to ClickHouse:**

ClickHouse now reads primarily from the **WarpStream ingestion VC** (where ingestion workers
produce their output), not directly from MSK.

1. **ClickHouse Kafka Engine** — native CH feature. Metrics prefixed `ClickHouseProfileEvents_Kafka*`
   (e.g., `KafkaMessagesPolled`, `KafkaRowsRead`, `KafkaRowsRejected`, `KafkaCommitFailures`).
   Consumer groups: `clickhouse_events_json` (prod-us), `group1` / `group1_recent` (prod-eu).
   These groups exist on BOTH MSK analytics and WarpStream ingestion — use the correct KMinion
   instance to distinguish: `kminion-warpstream-ingestion` for WarpStream, `kminion-msk-analytics` for MSK.
2. **Kafka Connect** — runs in `kafka-connect` namespace, uses DuckLake sink connector.
   Metrics prefixed `kafka_connect_*` and `kafka_connect_ducklake_sink_task_metrics_*`.
   Consumer groups: `connect-events-ducklake*`.

**Key health signals for ingestion operators:**

- `kminion_kafka_consumer_group_topic_lag{app_kubernetes_io_instance="kminion-warpstream-ingestion", group_id=~"clickhouse_events_json|group1|group1_recent", topic_name="clickhouse_events_json"}` — lag between ingestion output and CH consumption on the WarpStream path (the primary path)
- `kminion_kafka_consumer_group_topic_lag_seconds` with same group filter — same in seconds
- `ClickHouseProfileEvents_KafkaRowsRejected` — rows CH couldn't parse/insert
- `ClickHouseProfileEvents_FailedInsertQuery` — insert failures (schema issues, too many parts, etc.)
- `ClickHouseAsyncMetrics_MaxPartCountForPartition` — rising part count = merge pressure
- `ClickHouseMetrics_ReadonlyReplica` — replicas that fell behind and went read-only
- `ClickHouseAsyncMetrics_ReplicasMaxAbsoluteDelay` — max replication delay
- `posthog_celery_clickhouse_table_parts_count` / `_table_row_count` — Django-side CH health monitors

### Pyroscope services

Both old (`ingestion/{name}`) and new (`{namespace}/{name}`) formats coexist in Pyroscope
during the golden-chart migration. Prefer the new `{namespace}/{name}` format.

| Service name (new format)                                           | Deployment                |
| ------------------------------------------------------------------- | ------------------------- |
| `ingestion-analytics-main/ingestion-analytics-main`                 | Main analytics            |
| `ingestion-analytics-overflow/ingestion-analytics-overflow`         | Overflow lane             |
| `ingestion-analytics-historical/ingestion-analytics-historical`     | Historical lane           |
| `ingestion-analytics-async/ingestion-analytics-async`               | Async lane                |
| `ingestion-analytics-turbo/ingestion-analytics-turbo`               | Turbo lane (prod-us only) |
| `ingestion-heatmaps-main/ingestion-heatmaps-main`                   | Heatmaps                  |
| `ingestion-clientwarnings-main/ingestion-clientwarnings-main`       | Client warnings           |
| `ingestion-errortracking-main/ingestion-errortracking-main`         | Error tracking            |
| `ingestion-errortracking-overflow/ingestion-errortracking-overflow` | Error tracking overflow   |
| `logs-ingestion/logs-ingestion`                                     | Logs ingestion            |
| `traces-ingestion/traces-ingestion`                                 | Traces ingestion          |
| `recordings/recordings-blob-ingestion-v2`                           | Session replay            |
| `recordings/recordings-blob-ingestion-v2-overflow`                  | Session replay overflow   |

Profile types: `process_cpu:cpu:nanoseconds:cpu:nanoseconds`,
`wall:wall:nanoseconds:wall:nanoseconds`,
`memory:inuse_space:bytes:inuse_space:bytes`,
`memory:inuse_objects:count:inuse_space:bytes`.

### Grafana dashboards

| UID                                      | Title                                     | Focus                                                 |
| ---------------------------------------- | ----------------------------------------- | ----------------------------------------------------- |
| `ingestion-analytics`                    | Ingestion - Analytics                     | Per-pipeline analytics breakdown                      |
| `ingestion-health`                       | Ingestion - Health                        | Health overview across all ingestion services         |
| `ingestion-pipelines`                    | Ingestion - Pipelines                     | Per-lane pipeline step breakdown                      |
| `ingestion-reliability`                  | Ingestion - Reliability                   | Event trends, processing-lag SLOs, restarts, E2E lag  |
| `ingestion-person-processing`            | Ingestion -- Person Processing            | Person store, merge, cache                            |
| `ingestion-group-processing`             | Ingestion -- Group Processing             | Group store                                           |
| `ingestion-sessionreplay`                | Ingestion - Session Replay                | Replay blob pipeline                                  |
| `capture`                                | Capture                                   | Capture-side ingestion metrics (overview)             |
| `bexr9fnja75z4f`                         | Kafka Deduplicator                        | Event deduplication pipeline                          |
| `pl-ingestion-slas`                      | Ingestion — SLIs / SLOs / SLAs            | SLI/SLO view from `ingestion_sli_*` metrics (US only) |
| `warpstream`                             | Warpstream Agent Overview                 | Agent health, control plane ops, file cache           |
| `dbfj5c31spa1ogf`                        | MSK vs Warpstream — Active Produce Topics | Side-by-side produce volume comparison                |
| `8e93b023-a544-4a3b-8fac-123459d4eb84`   | WarpStream: ClickHouse Consumer Lag       | CH consumer lag on WarpStream topics (US only)        |
| `ws-coarse-lag-explore`                  | WarpStream Coarse Lag — Explore           | Agent-reported lag (US only, personal dashboard)      |
| `personhog-service`                      | Personhog service                         | PersonHog latency decomposition                       |
| `dbfgkwxs3gw8owd`                        | KMinion Consumer Group Lag                | Consumer lag by group (including CH groups)           |
| `logs`                                   | Logs (product)                            | Logs ingestion                                        |
| `vm-clickhouse-cluster-overview`         | ClickHouse (cluster overview)             | QPS, memory, disk, replication, parts, merges         |
| `clickhouse-ingestion-overview-20260615` | ClickHouse Ingestion Layer Overview       | Ingestion-layer CH cluster health and resources       |
| `clickhouse-keeper`                      | ClickHouse Keeper                         | ZooKeeper replacement health                          |
| `fe469e59-e10a-465a-9dac-ac8f6f82dc9a`   | ClickHouse Stuck Merge Loop Investigation | Merges that stop making progress                      |
| `cdzv7o1635n9ca`                         | Kafka Connect                             | Kafka Connect tasks, lag, DuckLake sink               |
| `fdrcd04np3hfkf`                         | ClickHouse - Kafka consumption            | CH Kafka engine consumption stats (EU only)           |

## Discovery workflows

### Prometheus / VictoriaMetrics

1. `list_prometheus_metric_names` — `datasourceUid: "victoriametrics"`, `regex: "ingestion_.*"` to enumerate app metrics
2. Pick a metric, then `list_prometheus_label_names` scoped to it — see available dimensions
3. `list_prometheus_label_values` — discover actual values for a label
   (e.g. `labelName: "cause"` on `ingestion_event_dropped_total`)
4. `query_prometheus` with PromQL — always scope by `app` or `ingestion_pipeline` and set a time range

Repeat with other prefixes: `consumed_batch_*`, `person_*`, `personhog_*`,
`overflow_redirect_*`, `ClickHouseMetrics_*`, `kafka_connect_*`, etc.

### Loki (logs)

1. `list_loki_label_names` — `datasourceUid: "P44D702D3E93867EC"`
2. `list_loki_label_values` for `app` — find ingestion containers (values like `ingestion-analytics-main`, `logs-ingestion`, etc.)
3. `query_loki_logs` — e.g. `{app=~"ingestion-.*"} |= "error"` or `{namespace="clickhouse"} |= "Exception"`

### Pyroscope (profiling)

1. `list_pyroscope_profile_types` — `data_source_uid: "pyroscope"`
2. `fetch_pyroscope_profile` — `matchers: '{service_name="ingestion-analytics-main/ingestion-analytics-main"}'`,
   `profile_type: "process_cpu:cpu:nanoseconds:cpu:nanoseconds"`

### Dashboards

1. `search_dashboards` — query `"ingestion"` or `"clickhouse"` or a specific deployment name
2. `get_dashboard_by_uid` — use a known UID (e.g. `"ingestion-health"`) to get panel details
3. `get_dashboard_panel_queries` — extract PromQL from existing panels

### Redis / ElastiCache

- Ingestion-side: discover `overflow_redirect_redis_*`, `cookieless_redis_*` metrics in VictoriaMetrics
- Infrastructure: CloudWatch datasource `P034F075C744B399F` for ElastiCache
  (CPU, memory, connections, latency).
  Cluster IDs: `ingestion-prod-redis`, `posthog-solo` (prod-us primary),
  `ingestion-duplicates-prod-redis`, `cookieless-prod-redis`

### Postgres / Aurora

- Ingestion-side: discover `postgres_error_total`, `person_*`, `group_*` metrics
- Infrastructure: `prometheus-postgres-exporter` and `prometheus-postgres-persons-exporter` metrics
- CloudWatch RDS: datasource `P034F075C744B399F` with cluster IDs
  `posthog-cloud-prod-us-east-1` / `posthog-cloud-persons-prod-us-east-1`

### ClickHouse

- Cluster health: `list_prometheus_metric_names` with regex `"ClickHouseMetrics_.*"` or `"ClickHouseProfileEvents_.*"`
  Scope with `type` label for cluster role (e.g. `type="events"`)
- Kafka engine health: regex `"ClickHouseProfileEvents_Kafka.*"`
- Kafka Connect: regex `"kafka_connect_.*"` — scope with `namespace="kafka-connect"`
- Consumer lag (the bridge): `kminion_kafka_consumer_group_topic_lag` with
  `group_id=~"clickhouse_events_json|group1|group1_recent|connect-events-ducklake.*"` and `topic_name="clickhouse_events_json"`.
  **Important:** scope with `app_kubernetes_io_instance="kminion-warpstream-ingestion"` for the
  WarpStream path (primary) or `"kminion-msk-analytics"` for the MSK path.
- Logs: `{namespace="clickhouse"} |= "Exception"` or `{namespace="kafka-connect"}`
- Dashboards: `vm-clickhouse-cluster-overview`, `clickhouse-ingestion-overview-20260615`,
  `cdzv7o1635n9ca`, `8e93b023-a544-4a3b-8fac-123459d4eb84` (WarpStream CH consumer lag)

## Key metric domains

Categories of what to look for. Discover specific metrics live using the prefixes above.

**Kafka consumer health** — batch duration, messages consumed per batch, consumer group
assignment/rebalance events, consumer lag (via KMinion). Metrics: `consumed_batch_*`,
`kafka_consumer_*`, `kminion_kafka_consumer_group_topic_lag*` scoped by `group_id`.

**Pipeline processing** — step-level latency and error rates, pipeline result distribution
(ingested, filtered, dropped, DLQ'd). Metrics: `events_pipeline_step_ms`,
`events_pipeline_step_error_total`, `ingestion_pipeline_results` by `result`.

**Person/group stores** — person flush latency, cache hit rates, Postgres write latency,
merge failures, properties size. Metrics: `person_*`, `group_*`, `personhog_*`.

**Outputs** — Kafka production to ClickHouse-bound topics. Message size, latency, errors.
Metrics: `ingestion_outputs_*` by `topic`.

**Overflow routing** — stateful overflow decisions, Redis operations for overflow state.
Metrics: `overflow_redirect_*` by `type`, `result`, `decision`.

**ClickHouse downstream health** — CH cluster QPS, memory, disk, merge pressure,
replication lag, Kafka engine consumption (rows read/rejected/failed), Kafka Connect
task health and consumer lag. This tells you whether events are actually making it
to the query layer. Metrics: `ClickHouseMetrics_*`, `ClickHouseProfileEvents_*`,
`ClickHouseAsyncMetrics_*` scoped by `type`; `kafka_connect_*` scoped by `namespace`.

**K8s resources** — `container_*` and `kube_*` for CPU, memory, restarts, HPA state.
Scope: `namespace=~"ingestion-.*"`, `pod=~"ingestion-.*"` (or `namespace="clickhouse"`,
`pod=~"chi-ingestion-.*"` for CH ingestion pods).

## Investigation playbooks

See [references/investigation-playbooks.md](./references/investigation-playbooks.md)
for step-by-step workflows covering: health checks, event drops, latency, consumer lag,
person processing (including the pganalyze query-level view of the persons DB),
Kafka/MSK issues, Redis, Postgres, session replay, ClickHouse downstream health,
single-partition lag (keyed-cost diagnosis, handing off to the `querying-tophog`
skill for actor identification), and cross-environment comparison.

مجوز

در منبع این مهارت هیچ مجوزی اعلام نشده است.

متن کامل مجوز در مخزن منبع در دسترس است.

مشاهدهٔ مجوز در مخزن منبع — نسخهٔ منتشرشده در آن‌جا مرجع است.