Sindika - Building Tomorrow's Digital Infrastructure

“It works on my machine” is the original sin of deployment. GitOps eliminates it by making Git the single source of truth for everything: application code, infrastructure config, environment variables, and deployment state. If it's not in Git, it doesn't exist.

GitLab CI/CD is uniquely suited for GitOps because the CI/CD system and the Git repository live in the same platform — no webhook gymnastics, no third-party integrations, no sync issues between your code host and your CI system. One platform, one workflow, one audit trail.

“We run all our CI/CD through GitLab — from building Docker images to deploying across staging and production. Every deployment is a merge request. Every rollback is a git revert. The Git log is the deployment history.”

— Sindika DevOps

Chapter 1: GitOps Principles — Not Just Buzzwords

GitOps has four non-negotiable principles. Every decision we make in our CI/CD pipeline flows from these. If your pipeline violates any of them, you're doing CI/CD — but not GitOps.

📋 1. Declarative

The entire system is described declaratively — docker-compose.yml, Helm charts, Terraform. No imperative scripts that mutate state.

📚 2. Versioned

All desired state is stored in Git. Every change is a commit. Every deployment is traceable to a specific SHA. git log is the audit trail.

⚡ 3. Automated

Approved changes are automatically applied. No SSH. No manual docker pull. The pipeline deploys. Humans review and approve.

🔄 4. Self-Healing

The system continuously reconciles actual state with desired state. If someone manually changes something, the system reverts it.

The practical implication: nobody deploys by SSH-ing into a server. Deployments happen through merge requests. Rollbacks happen through reverts. The server's state is always a reflection of what's in the main branch.

Chapter 2: Branch Strategy → Environment Mapping

The foundation of GitOps is mapping Git branches to deployment environments. Every branch has a clear purpose, and merging into a branch triggers deployment to the corresponding environment. No ambiguity, no surprises.

Feature branches merge into develop (→ staging). Develop promotes to main (→ production) via merge request with manual approval.

# Branch → Environment mapping in .gitlab-ci.yml

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      variables:
        DEPLOY_ENV: production
        DEPLOY_URL: https://app.example.com
    - if: '$CI_COMMIT_BRANCH == "develop"'
      variables:
        DEPLOY_ENV: staging
        DEPLOY_URL: https://staging.example.com
    - if: '$CI_MERGE_REQUEST_IID'
      variables:
        DEPLOY_ENV: review
    - when: always   # Feature branches: run tests only

🤔 Branch Strategy Pitfalls

▸Long-lived feature branches — branches open for weeks accumulate merge conflicts and integration bugs. Merge to develop daily. Small, frequent merges beat big-bang integrations.
▸Deploying from feature branches — never deploy feature branches to shared environments. They're untested, unreviewed code. That's what staging is for.
▸Skipping staging — “It's a small change, I'll push straight to main.” Famous last words. Every change goes through staging, no exceptions.

Chapter 3: The Production Pipeline

Here's the complete .gitlab-ci.yml we use for production deployments. It's been refined through hundreds of deployments and handles building, testing, publishing, and deploying — with manual gates for production.

CI stages (build, test, publish) run automatically. CD stages deploy to staging automatically, with a manual gate before production.

# .gitlab-ci.yml — complete production pipeline

stages:
  - build
  - test
  - publish
  - deploy-staging
  - deploy-production

variables:
  IMAGE: $CI_REGISTRY_IMAGE
  TAG: $CI_COMMIT_SHORT_SHA

# ─── Build Stage ───
build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build
        --cache-from $IMAGE:latest
        --build-arg BUILDKIT_INLINE_CACHE=1
        -t $IMAGE:$TAG
        -t $IMAGE:latest
        .
    - docker push $IMAGE:$TAG
    - docker push $IMAGE:latest
  rules:
    - changes:
        - "src/**/*"
        - Dockerfile
        - package.json
        - package-lock.json

# ─── Test Stage ───
test:unit:
  stage: test
  image: node:20-alpine
  script:
    - npm ci --prefer-offline
    - npm run test:unit -- --coverage
  coverage: '/All files.*\|\s*([\d.]+)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

test:lint:
  stage: test
  image: node:20-alpine
  script:
    - npm ci --prefer-offline
    - npm run lint
    - npm run type-check

# ─── Publish Stage ───
publish:
  stage: publish
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $IMAGE:$TAG
    - docker tag $IMAGE:$TAG $IMAGE:staging
    - docker push $IMAGE:staging
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
    - if: '$CI_COMMIT_BRANCH == "main"'

# ─── Deploy Staging (automatic) ───
deploy:staging:
  stage: deploy-staging
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - ssh deploy@staging.example.com "
        cd /opt/app &&
        TAG=$TAG docker compose -f docker-compose.prod.yml pull app &&
        TAG=$TAG docker compose -f docker-compose.prod.yml up -d --no-deps app &&
        sleep 5 &&
        curl -sf https://staging.example.com/health || exit 1
      "
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'

# ─── Deploy Production (manual gate) ───
deploy:production:
  stage: deploy-production
  environment:
    name: production
    url: https://app.example.com
  script:
    - ssh deploy@prod.example.com "
        cd /opt/app &&
        TAG=$TAG docker compose -f docker-compose.prod.yml pull app &&
        TAG=$TAG docker compose -f docker-compose.prod.yml up -d --no-deps app &&
        sleep 10 &&
        curl -sf https://app.example.com/health || exit 1
      "
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual     # 🔒 Manual approval required
  allow_failure: false  # Block pipeline until approved

✅ Pipeline Design Decisions

✓Image tagged with commit SHA — every build produces an image tagged with $CI_COMMIT_SHORT_SHA. You can always trace a running container back to the exact commit.
✓Build cache with --cache-from — reuses layers from the previous build, reducing build time from 5+ minutes to under 60 seconds.
✓rules: + changes: — the build stage only runs when relevant files change. README edits don't trigger a Docker build.
✓Health check after deploy — the deploy script curls the health endpoint. If it fails, the pipeline fails, and you know immediately.
✓Manual gate for production — staging deploys automatically. Production requires a human to click “Play” in the GitLab UI. This is the last line of defense.

Chapter 4: Environment Promotion Strategy

A critical GitOps principle: promote images, don't rebuild them. The exact same Docker image that passed tests in CI is the one deployed to staging, then promoted to production. Never rebuild for production — that introduces the risk of a different binary.

# Environment-specific configuration — NOT in the image

# staging.env (on staging server)
ASPNETCORE_ENVIRONMENT=Staging
ConnectionStrings__Default=Host=db;Database=app_staging;...
Redis__Connection=redis:6379
SMTP__Host=mailhog     # Catch-all email in staging
LOG_LEVEL=Debug

# production.env (on production server)
ASPNETCORE_ENVIRONMENT=Production
ConnectionStrings__Default=Host=db;Database=app_prod;...
Redis__Connection=redis:6379
SMTP__Host=smtp.sendgrid.net
LOG_LEVEL=Warning

# The Docker image is IDENTICAL in both environments
# Only the .env file (and DNS) differs

This means your Docker image must be environment-agnostic. No hardcoded URLs, no baked-in API keys, no build-time environment flags. Everything configurable flows through environment variables at runtime.

Chapter 5: Secret Management

GitOps says “everything in Git” — but secrets are the exception. Database passwords, API keys, and TLS certificates must never touch a Git commit. GitLab CI/CD Variables provide a secure, auditable way to inject secrets into your pipeline without exposing them.

# GitLab CI/CD Variables configuration
# Settings → CI/CD → Variables

# Per-environment secrets (environment-scoped):
DB_PASSWORD        → masked, protected, env: production
DB_PASSWORD        → masked, protected, env: staging
DEPLOY_SSH_KEY     → file type, protected
REGISTRY_PASSWORD  → masked

# In .gitlab-ci.yml, reference like normal ENV vars:
deploy:production:
  script:
    - echo "$DEPLOY_SSH_KEY" > /tmp/key && chmod 600 /tmp/key
    - ssh -i /tmp/key deploy@prod.example.com "
        cd /opt/app &&
        echo 'DB_PASSWORD=$DB_PASSWORD' >> .env &&  
        TAG=$TAG docker compose up -d --no-deps app
      "

# Protected = only available on protected branches (main, develop)
# Masked = hidden in pipeline logs (shows [MASKED])

✅ Secret Management Rules

✓Mark as Protected — secrets only available on protected branches. A random feature branch cannot access production database credentials.
✓Mark as Masked — GitLab replaces the value with [MASKED] in pipeline logs. Even accidental echo $SECRET won't leak.
✓Environment-scope — staging and production use different credentials. A staging pipeline can't accidentally write to the production database.
✓Rotate regularly — treat credentials like milk, not wine. Rotate every 90 days. Automate the rotation if possible.

Chapter 6: Rollback Strategies

The real test of your deployment pipeline isn't how fast you can ship — it's how fast you can un-ship. A broken deployment at 3 AM with no rollback plan is the nightmare scenario. GitOps makes rollbacks trivial because every deployment is traceable to a Git commit.

# Strategy 1: Re-run previous deploy job (fastest)
# In GitLab UI: CI/CD → Pipelines → Find last green pipeline → 
# Click "Retry" on deploy:production job
# Time: ~30 seconds (image is already cached)

# Strategy 2: Revert the merge request
# In GitLab UI: Merge Request → "Revert" button → New MR created
# Merge it → Pipeline runs automatically → Deploys reverted code
# Time: ~5 minutes (full pipeline runs)

# Strategy 3: Emergency manual rollback
# When everything is on fire and you need instant recovery
ssh deploy@prod.example.com
cd /opt/app
# Find the previous working tag
docker images registry.example.com/app --format '{{.Tag}}' | head -5
# Deploy the previous version
TAG=abc1234 docker compose -f docker-compose.prod.yml up -d --no-deps app
# Fix in Git later — never leave the Git state out of sync for long

Strategy 1 is preferred — it's the fastest and maintains GitOps principles. Strategy 3 breaks GitOps temporarily (server state diverges from Git), but sometimes minutes matter. The rule: fix the fire first, fix the process second. Always follow up an emergency rollback with a proper revert MR to keep Git and production in sync.

Chapter 7: Pipeline Optimization

A slow pipeline kills developer productivity. If CI takes 15 minutes, developers context-switch, lose flow, and batch changes into bigger (riskier) deploys. Our target: merge to staging deploy in under 5 minutes. Here's how we get there.

Pipeline Optimization Techniques

Technique	Before	After	Impact
Docker layer caching	5-8 min build	30-60s build	~10x faster
Parallel test stages	12 min serial	4 min parallel	3x faster
rules: + changes:	Run everything	Skip unchanged	50% fewer jobs
Image pre-pulling	90s image pull	0s (cached)	Deploy speedup
Kaniko (rootless build)	Docker-in-Docker	No privileged mode	Security ↑

# Optimized Dockerfile — maximize layer caching

# Stage 1: Dependencies (cached when lock file unchanged)
FROM node:20-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --prefer-offline

# Stage 2: Build (only busts cache when src/ changes)
FROM deps AS build
COPY . .
RUN npm run build

# Stage 3: Production (minimal final image)
FROM node:20-alpine AS production
WORKDIR /app
RUN addgroup -S app && adduser -S app -G app
COPY --from=build --chown=app:app /app/.next ./.next
COPY --from=build --chown=app:app /app/public ./public
COPY --from=build --chown=app:app /app/node_modules ./node_modules
COPY --from=build --chown=app:app /app/package.json ./
USER app
EXPOSE 3000
CMD ["npm", "start"]

# Result: dependency install is cached 95% of the time
# Only src/ changes trigger a full rebuild

✅ Pipeline Speed Wins

✓Run tests in parallel — unit tests and lint run simultaneously, not sequentially. Two 4-minute jobs in parallel = 4 minutes total, not 8.
✓Use rules: changes: — don't rebuild Docker when only docs changed. Only trigger builds for relevant file paths.
✓Cache node_modules — use npm ci --prefer-offline with GitLab's cache directive. Save 60-90 seconds per pipeline.
✓Multi-stage Dockerfile — separate dependency install from build. Dependencies change infrequently; source code changes on every commit.

Chapter 8: Monitoring Your Deployments

A deployment isn't done when the pipeline turns green. It's done when you've confirmed the new version is healthy in production. Here's what we watch after every deploy.

# Post-deployment verification checklist (automated)

deploy:production:
  script:
    # ... deploy commands ...
    
    # 1. Health check  
    - curl -sf https://app.example.com/health || exit 1
    
    # 2. Version verification
    - |
      DEPLOYED=$(curl -s https://app.example.com/health | jq -r .version)
      if [ "$DEPLOYED" != "$TAG" ]; then
        echo "Version mismatch! Expected $TAG, got $DEPLOYED"
        exit 1
      fi
    
    # 3. Smoke test (critical user flow)
    - curl -sf https://app.example.com/api/v1/status || exit 1
    
    # 4. Notify team
    - |
      curl -X POST "$SLACK_WEBHOOK" -H 'Content-type: application/json' \
        -d "{
          \"text\": \"✅ Deployed v$TAG to production\",
          \"attachments\": [{
            \"color\": \"good\",
            \"fields\": [
              {\"title\": \"Commit\", \"value\": \"$CI_COMMIT_SHORT_SHA\"},
              {\"title\": \"Author\", \"value\": \"$CI_COMMIT_AUTHOR\"},
              {\"title\": \"Pipeline\", \"value\": \"$CI_PIPELINE_URL\"}
            ]
          }]
        }"

The notification step is critical: every deployment posts to Slack/Telegram with the commit SHA, author, and pipeline link. When something breaks, the team immediately knows what changed, who changed it, and where to find the logs. No detective work needed.

“The goal of GitOps isn't to make deployments possible — it's to make them boring. When deploying to production is so routine that it doesn't cause anxiety, you've won.”