Hard lessons from real-world deployments, and how to avoid the most expensive mistakes. Hard lessons from real-world deployments, and how to avoid the most expensive mistakes. Docker is amazing — until it’s not. You write a Dockerfile, run it locally, and it works like magic. But the moment you try to scale it, secure it, or run it reliably in production, reality hits hard. From bloated images to silent security risks, there’s a lot that can go wrong. Dockerfile This article collects the hard-won lessons I’ve learned from building and running Dockerized platforms in production: what broke, what slowed us down, and how we fixed it. hard-won lessons 1. Your Docker Image is Probably Too Big The problem: Most production Docker images are bloated. I’ve seen images over 1GB that do nothing more than serve a simple Node.js or Python app. These slow down CI/CD, eat up bandwidth, and kill cold start performance. 1GB The fix: Use multi-stage builds and start from minimal base images. multi-stage builds minimal base images Example: # dev stage FROM node:18 as builder WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build # production stage FROM node:18-alpine WORKDIR /app COPY --from=builder /app/dist ./dist COPY --from=builder /app/node_modules ./node_modules CMD ["node", "dist/server.js"] # dev stage FROM node:18 as builder WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build # production stage FROM node:18-alpine WORKDIR /app COPY --from=builder /app/dist ./dist COPY --from=builder /app/node_modules ./node_modules CMD ["node", "dist/server.js"] With this change, we reduced image size from 900MB to 160MB. Faster builds, faster deploys, happier ops. 900MB to 160MB Also: Use .dockerignore Don’t copy entire project blindly Use .dockerignore Use .dockerignore .dockerignore Don’t copy entire project blindly Don’t copy entire project blindly 2. Stop Running Containers as Root The problem: By default, Docker containers run as root. If someone breaks into the container, they have root privileges in the container, which can be escalated depending on your host setup. root The fix: Add a non-root user inside your image: RUN addgroup app && adduser -S -G app app USER app RUN addgroup app && adduser -S -G app app USER app And don’t forget to use --user flag in docker run if needed. --user docker run Also: Don’t mount the Docker socket unless absolutely required Use tools like dockle, trivy, hadolint for security checks Don’t mount the Docker socket unless absolutely required Don’t mount the Docker socket unless absolutely required Use tools like dockle, trivy, hadolint for security checks Use tools like dockle, trivy, hadolint for security checks dockle trivy hadolint 3. Your Build Context is Slowing Everything Down The problem: If your .dockerignore is missing or incomplete, Docker might be copying your entire repo, including .git, node_modules, .env, etc. into the image. .dockerignore .git node_modules .env The fix: Use .dockerignore like your .gitignore: .dockerignore .gitignore .git node_modules *.log .env .DS_Store tests/ .git node_modules *.log .env .DS_Store tests/ This cuts down build context size dramatically, making builds faster and less prone to cache busting. 4. Volume Mounts Work Differently in Production The problem: In development, you rely on volume mounts for hot-reloading and persistence. In production, improper volume usage can: Break statelessness Create permission issues Introduce state inconsistency between replicas Break statelessness Create permission issues Introduce state inconsistency between replicas The fix: Only mount named volumes with a clear purpose Make volumes read-only where possible Never write to container’s internal FS expecting it to persist Only mount named volumes with a clear purpose Only mount named volumes with a clear purpose named volumes Make volumes read-only where possible Make volumes read-only where possible read-only Never write to container’s internal FS expecting it to persist Never write to container’s internal FS expecting it to persist 5. Logging Can Wreck Your Node The problem: Docker’s default logging driver (json-file) stores logs on disk. If you don’t set limits, logs can consume all space. json-file on disk The fix: Set logging limits in docker-compose.yml: docker-compose.yml logging: driver: json-file options: max-size: "10m" max-file: "3" logging: driver: json-file options: max-size: "10m" max-file: "3" Or use centralized logging: Fluentd, Logstash, or a logging SaaS. Golden rule: log to stdout/stderr, not to files. log to stdout/stderr 6. Use Docker's Built-in Security Features The problem: Containers share the host kernel. They’re isolated by namespaces, but not foolproof. The fix: Drop unnecessary Linux capabilities: docker run --cap-drop ALL --cap-add NET_BIND_SERVICE ... Run with --read-only FS where possible Apply a seccomp profile or AppArmor Don’t run privileged containers unless absolutely necessary Drop unnecessary Linux capabilities: docker run --cap-drop ALL --cap-add NET_BIND_SERVICE ... Drop unnecessary Linux capabilities: docker run --cap-drop ALL --cap-add NET_BIND_SERVICE ... docker run --cap-drop ALL --cap-add NET_BIND_SERVICE ... Run with --read-only FS where possible Run with --read-only FS where possible --read-only Apply a seccomp profile or AppArmor Apply a seccomp profile or AppArmor Don’t run privileged containers unless absolutely necessary Don’t run privileged containers unless absolutely necessary 7. CI/CD Pipelines Need Container-Aware Design The problem: Many CI pipelines rebuild full images unnecessarily or fail to cache layers properly. The fix: Use layer caching: GitHub Actions cache, GitLab’s built-in Docker cache Split COPY in Dockerfile to minimize cache invalidation: COPY package.json . RUN npm ci COPY . . Avoid docker:dind unless necessary Use layer caching: GitHub Actions cache, GitLab’s built-in Docker cache Use layer caching: GitHub Actions cache, GitLab’s built-in Docker cache cache Split COPY in Dockerfile to minimize cache invalidation: COPY package.json . RUN npm ci COPY . . Split COPY in Dockerfile to minimize cache invalidation: COPY Dockerfile COPY package.json . RUN npm ci COPY . . COPY package.json . RUN npm ci COPY . . Avoid docker:dind unless necessary Avoid docker:dind unless necessary docker:dind Conclusion: Docker is Not Magic Docker is a brilliant abstraction, but it doesn’t mean "write once, scale forever." Treat containers as production infrastructure, not dev toys. Optimize them. Secure them. Monitor them. And most importantly: test everything in conditions as close to production as possible. Bonus: Tooling I Recommend Dive — Inspect Docker layers and image size Trivy — Vulnerability scanner Dockle — Best practices and security analyzer Hadolint — Linter for Dockerfiles Docker Scout — Docker's built-in SBOM and vuln check Dive — Inspect Docker layers and image size Dive — Inspect Docker layers and image size Dive Trivy — Vulnerability scanner Trivy — Vulnerability scanner Trivy Dockle — Best practices and security analyzer Dockle — Best practices and security analyzer Dockle Hadolint — Linter for Dockerfiles Hadolint — Linter for Dockerfiles Hadolint Docker Scout — Docker's built-in SBOM and vuln check Docker Scout — Docker's built-in SBOM and vuln check Docker Scout Stay safe and ship fast. Questions or feedback? I’d love to hear what lessons you learned from Docker in production. you [Author] - Oleksii Bondar