DevOps Engineer Interview Questions & Answers

Sample Answer

Our primary database cluster failed over during peak hours due to a disk space issue that our monitoring didn't catch. The failover itself worked, but the replica was 30 seconds behind, causing some user-facing data inconsistency. I led the incident response: isolated the problem in 10 minutes, coordinated with the engineering team to identify affected users in 20 minutes, and had a communication out to customers within 45 minutes. Total downtime was 8 minutes. In the post-mortem, I implemented three changes: added disk space monitoring with 80% threshold alerts, created a runbook for database failover scenarios, and set up weekly chaos engineering tests for our failover mechanisms. We also moved to synchronous replication for critical tables. Over the next 12 months, we had zero unplanned database failovers.

Sample Answer

The development team was deploying manually via SSH and resisted automated deployments because 'we need control.' Instead of mandating the change, I started with their pain. I tracked that manual deployments averaged 45 minutes, had a 15% failure rate, and always happened on Fridays at 5 PM. I built a CI/CD pipeline that replicated their exact manual steps, ran it in parallel for 2 weeks, and showed that automated deploys took 3 minutes with a 1% failure rate. I let the most skeptical developer run the first automated production deploy. When it worked flawlessly, he became the pipeline's biggest advocate. Within a month, the team was deploying 3x daily instead of once weekly. The key was making the change feel safe, not forcing compliance.

Sample Answer

I audited our AWS spend and found we were paying $28K monthly for resources that ran 24/7 but were only needed during business hours (development environments, staging clusters, batch processing). I implemented an automated scheduling system using AWS Lambda and CloudWatch Events that scaled non-production environments down to zero overnight and on weekends. I also right-sized production instances using 3 months of CloudWatch metrics — we were running m5.2xlarge instances that never exceeded 30% CPU. Switching to m5.large with autoscaling saved another $8K monthly. I moved infrequently accessed S3 data to Glacier, saving $3K monthly. Total savings: $18K monthly ($216K annually) with zero performance impact. I set up a monthly cost review dashboard that became a standard practice.

Sample Answer

Developers were spending an average of 90 minutes setting up their local development environment, and configuration drift caused 'works on my machine' bugs weekly. I containerized the entire development stack with Docker Compose — database, message queue, cache, and all microservices. I wrote a single 'make dev' command that pulled the latest images, seeded the database, and started all services. Setup time dropped from 90 minutes to 5 minutes. I also built a pre-commit hook system that ran linting, type checking, and unit tests locally, catching 80% of CI failures before push. Developer satisfaction survey scores for 'tooling and infrastructure' went from 2.8 to 4.3 out of 5. The key insight: developer productivity is an infrastructure problem.

Sample Answer

VMs virtualize hardware — each VM runs its own OS kernel, managed by a hypervisor. Containers virtualize the OS — they share the host kernel and isolate at the process level using namespaces and cgroups. Containers are lighter (MB vs GB), start faster (seconds vs minutes), and pack more densely on a host. I choose containers for microservices, stateless applications, and anything that needs rapid scaling. I choose VMs when I need full OS isolation (multi-tenant environments with strict security requirements), need to run different operating systems on the same host, or am running software that requires kernel modifications. In practice, most modern workloads are containerized. But legacy applications, certain compliance requirements (PCI-DSS environments that mandate VM-level isolation), and Windows workloads may still be better suited for VMs.

Sample Answer

I'd build it in layers. The inner loop (per-service): each microservice has its own pipeline triggered by commits to its directory. It runs linting, unit tests, builds a container image with a content-addressable tag (git SHA), pushes to a registry, and deploys to a dev environment. The integration layer: after a service passes its own tests, integration tests run against a shared staging environment with all services. I'd use contract testing (Pact) to catch inter-service compatibility issues early without needing all services running. The deployment layer: I'd use GitOps with ArgoCD — merging to the deploy branch updates a manifest repo, and ArgoCD syncs the cluster state. Rollbacks are a git revert. For safety: canary deployments with automated rollback based on error rate thresholds, deployment windows with automatic holdbacks during high-traffic periods, and a manual approval gate for production in the first few months until confidence is established.

Sample Answer

Infrastructure as code means defining your infrastructure — servers, networks, databases, load balancers — in declarative configuration files rather than clicking through consoles or running ad-hoc scripts. Tools like Terraform, CloudFormation, and Pulumi enable this. It matters for three reasons. First, reproducibility: I can spin up an identical environment in any region by running the same code. No 'we think staging matches production.' Second, version control: infrastructure changes go through the same code review, approval, and audit trail as application code. I can see who changed what, when, and why — and roll back if needed. Third, automation: infrastructure changes can be tested, validated, and applied by CI/CD pipelines, eliminating manual errors. In practice, I use Terraform with remote state in S3, state locking with DynamoDB, and modules for reusable components. Every infrastructure change goes through a pull request with a terraform plan output for review.

Sample Answer

I follow the three pillars: metrics, logs, and traces. Metrics (Prometheus + Grafana) give me aggregate system health — CPU, memory, request rate, error rate, latency percentiles. I set alerts on symptoms (high error rate, elevated latency) not causes (high CPU), because symptoms tell you users are affected. Logs (ELK or Loki) provide detailed event context for debugging. I enforce structured logging (JSON) with correlation IDs so I can trace a request across services. Traces (Jaeger or Tempo) show me the full request path through the system, revealing which service is slow and why. Beyond the three pillars, I build business-level dashboards: orders per minute, sign-ups per hour, payment success rate. These are the first things I check during an incident because they answer 'are users affected?' before I dive into infrastructure metrics. I also implement SLOs with error budgets — this turns monitoring from 'is anything broken?' into 'are we meeting our reliability commitments?'

Sample Answer

Short-term: I'd enable the Kubernetes Cluster Autoscaler to add nodes automatically when pods can't be scheduled. I'd verify our node pool has room to scale and the autoscaler's scaling policies are sensible (scale up fast, scale down slowly). I'd also check if existing resources are over-allocated — it's common for pods to request 2x the resources they actually use. Right-sizing resource requests based on actual usage metrics could free 30-40% capacity without adding nodes. Medium-term: I'd implement Horizontal Pod Autoscaler (HPA) on all workloads with proper CPU/memory targets, and consider Vertical Pod Autoscaler (VPA) for workloads with unpredictable resource patterns. I'd also move batch workloads to spot/preemptible instances. Long-term: I'd analyze traffic patterns and implement predictive scaling — if we know peak hours are 9-11 AM, pre-scale 15 minutes before rather than reacting to demand. I'd also evaluate whether some workloads should move to serverless (Lambda/Cloud Run) to remove the capacity planning burden entirely.

Sample Answer

First, I'd fix the immediate issue: revert the configuration change and restore service. Then I'd run a blameless post-mortem — the focus is on the system that allowed this, not the person who made the change. My preventive measures: first, all configuration changes go through the same CI/CD pipeline as code — pull request, review, automated validation, staged rollout. No direct pushes to production config. Second, implement configuration validation in the pipeline — schema checking, dry-run application, and integration tests against a staging environment. Third, progressive rollout for config changes: apply to 5% of instances, monitor error rates for 10 minutes, then proceed to 25%, 50%, 100%. Fourth, automated rollback triggers: if error rate exceeds baseline by 2x within 5 minutes of a config change, automatically revert. The goal is making the safe path the easy path — it should be harder to push config directly than to go through the pipeline.

Sample Answer

I'd never do a big-bang rewrite — that fails consistently. Instead, I'd use the strangler fig pattern: incrementally extract services while the monolith continues to serve traffic. Step 1: identify service boundaries using domain-driven design. Map the monolith's domains, data ownership, and dependencies. Start with the least coupled, highest-value service to extract. Step 2: build the new service alongside the monolith, routing traffic to it through an API gateway. The monolith and new service initially share the database. Step 3: once the service is stable, migrate its data to its own database and remove the shared database dependency. Step 4: repeat for the next service. I'd expect the full migration to take 12-18 months for a medium-sized monolith. Key principles: each extracted service must be independently deployable and testable, the system must work in the hybrid state (some services extracted, some still in the monolith), and I'd add comprehensive monitoring at the boundary between old and new to catch integration issues early.

Sample Answer

I'd evaluate based on four factors rather than personal preference. First, existing expertise: what does the team already know? Cloud learning curves are steep, and the productivity cost of switching is real. If the team has 3 years of AWS experience, the new platform needs to offer a compelling advantage to justify the ramp-up time. Second, specific service requirements: does the project need a service that one cloud does significantly better? GCP for ML (Vertex AI), Azure for enterprise integration (Active Directory), AWS for the broadest service catalog. Third, pricing model: I'd estimate costs using each cloud's calculator for our specific workload pattern. Reserved instances, spot pricing, and egress costs vary significantly. Fourth, organizational context: vendor agreements, compliance requirements, and existing infrastructure. If the company already has an AWS Enterprise Agreement, using GCP adds procurement complexity. I'd present a recommendation with a clear decision matrix, not just my gut feeling.