Log Reading & Analysis

Logs are written in English — log messages, error descriptions, status codes, and structured fields use English vocabulary and patterns. Non-native speakers can read code but struggle with: understanding the exact meaning of log levels (WARN vs. ERROR vs. CRITICAL), interpreting natural-language exception messages, understanding stack traces with framework-specific terminology, and translating log findings into clear incident descriptions for stakeholders. Log reading combines English comprehension with technical debugging skills.

Standard log levels (most → least verbose): TRACE/VERBOSE (detailed diagnostic, usually disabled in production), DEBUG (developer information for troubleshooting), INFO (normal operations: "User logged in", "Request processed"), WARN/WARNING (unexpected but not failing: "Retry attempt 2/3", "Deprecated method called"), ERROR (failure that should be investigated), CRITICAL/FATAL (system-level failure, often triggers alerts and requires immediate response).

Structured log format (JSON): {"timestamp":"2024-07-15T14:32:01Z","level":"ERROR","service":"payment-api","message":"Transaction failed","error":"timeout after 5000ms","traceId":"abc123","userId":"u-789"}. Reading strategy: timestamp (when), level (severity), service (where), message (what happened), error (why), traceId (correlate across services), userId (who was affected). Correlate entries using traceId to reconstruct the full request journey.

Key log patterns: Timeout patterns ("connection timed out after 30000ms"), Retry patterns ("attempt 3 of 5"), Authentication failures ("invalid token", "signature verification failed"), Resource exhaustion ("connection pool exhausted", "OOMKilled"), Dependency failures ("upstream service returned 503"), Deployment markers ("app version 2.4.1 starting"), Graceful shutdown ("received SIGTERM, shutting down in 30s"). Recognising these reduces diagnosis time from hours to minutes.

Log-to-stakeholder translation: "The logs show a database connection failure at 14:32 UTC — the payment service couldn't reach the database for 47 seconds" (not "we got ECONNREFUSED in the PostgreSQL connection pool"). "We can see from the logs that roughly 2,000 users were affected during the window of 14:32 to 15:19 UTC." Always translate: error code → plain description, timestamp → duration, technical cause → business impact.

Distributed tracing tracks a single request across multiple microservices using a shared trace ID. Reading traces: find the initial request in the gateway log, follow the traceId across service logs, identify where the request slows or fails. Vocabulary: span (one operation within a trace), parent span (calling service), child span (called service), latency (duration per span), error span (span where failure occurs). Tools: Jaeger, Zipkin, AWS X-Ray, OpenTelemetry.

OOMKilled means "Out Of Memory Killed" — Kubernetes terminated a pod because it exceeded its memory limit. In logs: `reason: OOMKilled`, `exit code 137`. Response: check the pod's memory usage with `kubectl top pods`, review memory limits in the pod spec, analyse heap dumps to find memory leaks. In incident communication: "The payment service pod was killed at 14:32 due to memory exhaustion — it consumed 512MB against a 256MB limit."

Log search vocabulary (grep, Splunk, Elasticsearch): `grep "ERROR" app.log`, filtering by time range, filtering by service or hostname, searching for specific error messages or trace IDs. In conversation: "Let me grep the production logs for the traceId", "I'm filtering the Datadog logs to show only ERROR and CRITICAL entries from the payment service in the last hour", "Can you share the Kibana query that shows the spike?" Knowing the English vocabulary for log queries speeds up investigations.

CI/CD log vocabulary: pipeline (automated build/test/deploy sequence), stage (phase: build, test, deploy), job (unit of work within a stage), artifact (build output), cache hit/miss (dependency caching), lint (code style check), flaky test (test that intermittently fails), deployment gate (approval required before deployment), rollback trigger (condition that reverts deployment). Failed CI logs often show these terms in error context.

Good log message principles: be specific ("User authentication failed for userId u-789: invalid password hash" not "auth error"), include relevant IDs (userId, orderId, requestId) for correlation, use consistent verb tenses (past: "Request completed", "Connection failed"), avoid abbreviations in messages, and include context values ("Retry 2/3 after 1000ms delay"). Bad: "Error!"; Good: "Failed to connect to Redis at redis-cluster:6379 after 3 retries — falling back to in-memory cache".