Software Deployment SOP Template

Action:

• Verify all code changes for this release are merged and peer-reviewed:
  • Confirm every pull request has at least one approved review
  • Check that no open PRs targeted for this release are still pending
  • Verify merge conflicts are resolved and the main branch is clean
• Run the full automated test suite:
  • Unit tests must pass with 100% success rate
  • Integration tests must pass against the current test environment
  • Review code coverage reports — flag any significant drops
• Create a release branch or tag:
  • Follow your team's branching strategy (e.g., release/v2.4.1 or tag v2.4.1)
  • Lock the release branch — no new commits without explicit approval
• Update version numbers and changelog:
  • Bump the version number in package manifests, config files, and API docs
  • Add a changelog entry summarizing features, fixes, and breaking changes
  • Commit version bump and changelog to the release branch

Expected Outcome: A locked release branch or tag with all code merged, tests passing, version bumped, and changelog updated

Action:

• Verify the CI pipeline passes on the release branch:
  • Build artifacts are generated successfully
  • All pipeline stages (lint, test, build, package) complete without errors
  • Docker images or deployment artifacts are pushed to the artifact registry
• Check dependency compatibility:
  • Review dependency updates for known vulnerabilities (use Snyk, Dependabot, or similar)
  • Confirm no breaking changes in upstream libraries or APIs your service depends on
  • Validate that pinned dependency versions match what was tested
• Review security scan results:
  • Static Application Security Testing (SAST) reports show no critical or high-severity findings
  • Container image scans pass with no unpatched CVEs above your threshold
  • Secrets scanning confirms no credentials or API keys are committed in the codebase
• Confirm database migrations are ready:
  • Migration scripts have been reviewed and tested against a copy of production data
  • Rollback scripts exist for every migration
  • Estimated migration run time is documented and falls within the maintenance window

Expected Outcome: CI pipeline green, dependencies validated, security scans clear, and database migrations reviewed and rollback-ready

Action:

• Deploy the release package to your staging environment:
  • Use the same deployment mechanism you will use in production (identical pipeline, scripts, and config)
  • Apply database migrations to the staging database
  • Verify environment variables and feature flags are set correctly for staging
• Run integration and smoke tests:
  • Execute automated end-to-end tests against the staging deployment
  • Run smoke tests covering critical user flows (login, core features, payment if applicable)
  • Test API endpoints with expected request/response payloads
• Verify functionality against acceptance criteria:
  • Walk through each feature or fix included in the release against its ticket acceptance criteria
  • Test edge cases and error handling paths
  • Verify backward compatibility with existing clients and integrations
• Get QA sign-off:
  • QA team confirms all test cases pass
  • Product owner or stakeholder reviews and approves the staging deployment
  • Document QA approval in your release tracking tool (Jira, Linear, or similar)

⚠️ Tip: Never treat staging as optional. If you skip staging, you are testing in production — and your users become your QA team. Staging should mirror production as closely as possible, including data volume, network topology, and third-party integrations.

Expected Outcome: Release is deployed to staging, all tests pass, acceptance criteria verified, and QA sign-off documented

Action:

• Document the deployment steps and timeline:
  • Write a step-by-step deployment runbook specific to this release
  • Include estimated time for each step (migration, deploy, validation, etc.)
  • Specify who is responsible for each step and who is on-call for support
• Identify rollback triggers and procedure:
  • Define clear rollback criteria (e.g., error rate exceeds 1%, P95 latency doubles, health checks fail)
  • Document the exact rollback steps — revert to previous container image, roll back migrations, restore config
  • Assign a rollback decision-maker and ensure they are available during the deployment window
• Notify the team via Slack, email, or your communication channel:
  • Announce the deployment window, expected duration, and potential user impact
  • Share the deployment runbook link with all stakeholders
  • Tag on-call engineers, support teams, and product managers
• Schedule a maintenance window if needed:
  • Coordinate with customer support for any expected downtime
  • Update status pages or send customer notifications if user-facing impact is expected
  • Avoid deploying during peak traffic hours unless the deployment strategy supports zero-downtime

Expected Outcome: Deployment runbook completed, rollback plan documented, stakeholders notified, and maintenance window scheduled if required

Action:

• Follow the deployment runbook step by step:
  • Do not deviate from the documented plan — if something unexpected comes up, pause and assess before proceeding
  • Check off each step as you complete it in the runbook
  • Communicate progress in the deployment Slack channel or war room in real time
• Deploy incrementally if possible (canary or blue-green strategy):
  • Canary: Route a small percentage of traffic (e.g., 5%) to the new version, observe metrics, then gradually increase
  • Blue-green: Deploy the new version alongside the current one, validate, then switch traffic
  • Rolling: Replace instances one at a time, waiting for each to pass health checks before proceeding
• Monitor error rates and performance metrics in real time:
  • Watch application error rate, HTTP 5xx responses, and exception counts on your monitoring dashboard
  • Track P50, P95, and P99 latency — compare against your baseline
  • Monitor CPU, memory, and pod/instance health across the deployment
• Verify health checks pass:
  • All instances or pods report healthy status
  • Load balancer confirms new instances are receiving and serving traffic
  • Database connection pools are healthy and queries are executing within expected thresholds

⚠️ Tip: If any rollback trigger is hit during deployment, roll back immediately. Do not debug in production under pressure. Roll back, stabilize, investigate in a calm environment, fix, and redeploy. A fast rollback protects your users and your team's sanity.

Expected Outcome: New version deployed to production, traffic shifted incrementally, all health checks passing, and error rates within acceptable thresholds

Action:

• Run production smoke tests:
  • Execute a lightweight test suite against production to confirm core functionality works
  • Test critical paths: user authentication, primary workflows, payment processing (if applicable)
  • Verify API responses return expected status codes and payloads
• Monitor application logs and error rates for 30–60 minutes:
  • Watch for new error patterns, stack traces, or warning spikes that did not exist before deployment
  • Compare error rates and log volume against the pre-deployment baseline
  • Check for slow queries, connection timeouts, or resource exhaustion
• Verify key user flows work correctly:
  • Manually walk through 3–5 critical user journeys in production
  • Confirm data integrity — records created, updated, or migrated as expected
  • Validate third-party integrations (payment gateways, email services, webhooks) are functioning
• Check alerting and monitoring dashboards:
  • Confirm no new alerts have fired since deployment
  • Verify SLO dashboards show metrics within acceptable ranges
  • Review infrastructure metrics — no unexpected scaling events or resource spikes

Expected Outcome: Production smoke tests pass, error rates are stable, key user flows verified, and monitoring dashboards show healthy metrics for at least 30 minutes post-deployment

Action:

• Update the deployment log with results:
  • Record the deployment date, time, duration, and version number
  • Document who performed the deployment and who was on-call
  • Note the deployment strategy used (canary, blue-green, rolling, or direct)
• Note any issues encountered and how they were resolved:
  • Document unexpected errors, delays, or deviations from the runbook
  • Record root cause and resolution for each issue
  • Flag items that should be addressed before the next deployment (process gaps, tooling improvements)
• Archive release artifacts:
  • Tag the release in version control with the final version number
  • Store build artifacts (Docker images, binaries, config snapshots) in your artifact repository
  • Archive the deployment runbook and any incident notes alongside the release
• Notify stakeholders of successful deployment:
  • Send a release summary to the team via Slack, email, or your communication channel
  • Update release notes in your documentation or customer-facing changelog
  • Close the release ticket and move all associated work items to "Done"

Expected Outcome: Deployment log updated, issues documented with resolutions, artifacts archived, stakeholders notified, and the release is officially closed