[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-faqs-production-change-release-management-en":3,"blog-post-production-change-release-management-en":28},[4,8,12,16,20,24],{"answer":5,"id":6,"question":7},"In this article, change management means deciding what will change, who approves it and on what evidence. Release management means putting an approved change into the live system, including choosing the window, the release method and the way back. The two can be separated on paper, but in practice they should share one release request, so the approver sees the way back before signing.","8xb7l91t84roiac","What is the difference between change management and release management?",{"answer":9,"id":10,"question":11},"No. We recommend scaling approval to risk. Changes that repeat an established pattern and have gone through many times can be pre-approved. Normal changes go to an approver who did not write the change. Changes that touch the database schema or affect how users work deserve more reviewers. If every change waits for a weekly meeting, the team bundles work into large releases, which are harder to diagnose when they fail.","9q86pl6fpqrycll","Does every change need to go to a change approval meeting (CAB)?",{"answer":13,"id":14,"question":15},"Choose per release, based on what is changing. Blue-green needs two sets of infrastructure that can each handle the full production load, so you pay for two full-size environments, but switching back is fast if the database works with both versions. Canary gives the new version to a small group first and then expands step by step, so you must be able to watch each group's signals separately. Both roll out progressively, and both need bake time at each step.","euu24fd9n5v42qu","Should we choose blue-green or canary?",{"answer":17,"id":18,"question":19},"Once the feature is on for every user and has passed its bake time without problems. Every flag is a branch in the code, and if flags are left in place, the number of possible flag combinations grows beyond what can be tested. We recommend setting a removal date when the flag is created, and treating the removal as a task in the next release rather than something that waits for spare time.","goy5nyhxc9u8rtj","When should a feature flag be removed from the code?",{"answer":21,"id":22,"question":23},"Use the emergency path defined before the freeze began. The Azure Well-Architected guidance recommends defining in advance who can approve an accelerated process and what criteria must be met. Testing can be accelerated, but it should still be completed as quickly as possible. Not releasing the patch at all is a risk of its own, so a freeze calendar should never exist without this path.","gr28yqash9c3kp5","What should we do if a security patch is needed during a freeze?",{"answer":25,"id":26,"question":27},"The real question is whether someone will be watching during the observation period after release. Azure recommends leaving bake time of hours or days before expanding to the next group of users, and we think the observation period after a release should be counted in the same units. If you ship a large release on Friday evening, the period that needs watching falls on the weekend. A small change that can be rolled back immediately carries less risk. So we look at the way back and who is watching the system, rather than at the day of the week.","hos76ba1g8t8e7v","Should we release to production on a Friday?",{"id":29,"parentId":30,"title":31,"excerpt":32,"content":33,"image":34,"date":35,"isoDate":36,"views":37,"tag":38,"categorySlug":39,"slug":40,"to":41,"readTime":42,"isHighlight":43,"sortOrder":44,"video":45,"video_id":45,"keywords":46},"aykouh8pi6d6j6d","tt80d2d0lbz5adr","Deploying to Production Safely: Blue-Green, Canary and Rollback","Deploy and release decisions: who approves on what evidence, when to freeze, choosing blue-green, canary or feature flags, and why schema rollback is hard.","\u003Cp>If the production release request on your desk says little more than \"tested, passed\", you are being asked to sign for a risk you cannot see. The signer cannot see what changed, and cannot see how the team would back out if the system misbehaves after release. A signature like that does not reduce risk at all. It only moves the responsibility to the person in the room with the least information.\u003C\u002Fp>\u003Cp>This article is about routine releases on a system people use every day, such as a bug fix or a configuration change. It is not about replacing a whole system. If you are moving data from an old system to a new one, that has its own procedure, which we cover separately in \u003Ca target=\"_blank\" rel=\"noopener noreferrer\" href=\"https:\u002F\u002Fwww.superdev.tech\u002Fen\u002Fblogs\u002Fdata-migration-validate-cutover\">our article on validation and cutover when replacing a system\u003C\u002Fa>.\u003C\u002Fp>\u003Ch2>There is no such thing as a low-risk production deployment\u003C\u002Fh2>\u003Cp>Microsoft's \u003Ca target=\"_blank\" rel=\"noopener noreferrer\" href=\"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Fwell-architected\u002Foperational-excellence\u002Fsafe-deployments\">safe deployment practices guide in the Azure Well-Architected Framework\u003C\u002Fa> says it plainly: there are no low-risk deployments to production. Whether the change is application code, infrastructure as code, a feature flag or a configuration update, it introduces risk. The same guide recommends that every deployment follow a standard pattern and receive the same level of risk management.\u003C\u002Fp>\u003Cp>Google's Site Reliability Engineering book, in its chapter on \u003Ca target=\"_blank\" rel=\"noopener noreferrer\" href=\"https:\u002F\u002Fsre.google\u002Fsre-book\u002Frelease-engineering\u002F\">Release Engineering\u003C\u002Fa>, also notes that configuration changes are a potential source of instability. We think this matters most for organizations that handle configuration edits outside the normal process because \"no code changed\". Changing a database connection setting, or switching a feature on through an admin screen, can take a system down just as new code can.\u003C\u002Fp>\u003Cp>Microsoft also recommends frequent small deployments over infrequent large ones, because small changes are easier to resolve when something goes wrong. That has a direct bearing on how you design approvals, which we come back to in the section on objections.\u003C\u002Fp>\u003Ch2>How close to production does staging need to be before you trust the test results?\u003C\u002Fh2>\u003Cp>The Azure guidance says to deploy changes to staging environments that mirror production before they reach the live environment. In practice, a staging environment that matches production in every respect is expensive, and an organization may choose not to pay for all of it. That is an acceptable decision, as long as everyone knows where staging differs from the real thing.\u003C\u002Fp>\u003Cp>Tests only prove what staging shares with production.\u003C\u002Fp>\u003Cp>We check four kinds of difference first. The first is the versions of the runtime, the operating system and the database. The second is configuration, including which feature flags are actually switched on. The third is connections to external systems, since staging may talk to a provider's sandbox rather than the real service. The last is data volume: a statement that runs quickly against ten thousand rows can lock a table for minutes against several million.\u003C\u002Fp>\u003Cp>We recommend writing these differences down as a list in the system documentation and attaching it to every release request. The approver can then see what this round of testing proved and what it did not. The list also doubles as an investment plan. Whichever difference keeps causing failed releases is where it is worth paying to bring staging closer to production.\u003C\u002Fp>\u003Ch2>Who approves, and on what evidence\u003C\u002Fh2>\u003Cp>The Release Engineering chapter of the Google SRE book describes several layers of security and access control that determine who can perform which release operations. The gated operations include approving source code changes and deploying a new release. Google's automated release system also produces a report of every change contained in a release and archives it with the build artifacts, so the team can troubleshoot faster when a release causes problems.\u003C\u002Fp>\u003Cp>An organization does not need automation on that scale, but the same principle applies. The release request we want to see answers these six questions before anyone signs.\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cp>What does this release change, and which request or ticket does each item come from?\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>What were the staging test results, and what could staging not prove?\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>Does it change the database structure? If so, how far back can it be rolled?\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>If something goes wrong after release, how will the team back out, and who decides to do it?\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>Which users are affected, and have they been told?\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>Who is watching the system after release, and which signals will show it is healthy?\u003C\u002Fp>\u003C\u002Fli>\u003C\u002Ful>\u003Cp>The approver should be someone other than the person who wrote the change, and the business owner of the system should share the decision when a change affects how users work. We sort changes into three levels. Changes that repeat an established pattern and have gone through many times can be pre-approved. Normal changes go through a regular approver. Emergency changes, such as a security patch, need a fast path that was defined before the emergency.\u003C\u002Fp>\u003Cp>The Azure Well-Architected guidance makes the same point about that last case: define in advance who can approve an accelerated process in an emergency, and what criteria must be met. If that is not defined, then on the day of a real emergency, whoever happens to be at the keyboard ends up making the call.\u003C\u002Fp>\u003Ch2>Release windows, and the days to leave the system alone\u003C\u002Fh2>\u003Cp>Releasing at 2 a.m. reduces the number of users affected, but it also means a tired team, and external providers may be unreachable if you need their help. So we choose a window when few people are using the system and the people who can fix it are all awake. For internal systems used during office hours, we think early morning before the working day, on a day that is not a peak day, meets that test better than the middle of the night.\u003C\u002Fp>\u003Cp>A freeze is a period in which the organization agrees not to release anything new, apart from emergencies. Examples worth considering are the close of the fiscal year, month-end closing, payroll day and registration days announced in advance. Build this calendar with the business side at the start of the year, because the IT team does not know every date that matters to other departments.\u003C\u002Fp>\u003Cp>A freeze has its own cost. The Azure guidance warns that failing to deploy regular updates introduces other risks, such as security vulnerabilities that must be fixed through a deployment. A freeze calendar therefore needs an emergency path with the approver named. Another risk to plan for is the week before a freeze, which can turn into a rush of teams pushing changes at the same time, and that is riskier than an ordinary day.\u003C\u002Fp>\u003Cp>The observation period after a release belongs in the plan as well. Azure uses the term bake time for the time left between rollout groups, and says it should be measured in hours and days rather than minutes. We think the observation period after a release should be counted in the same units, which is why we would not ship a large release right before a long holiday: the period that needs watching falls on days when nobody is watching.\u003C\u002Fp>\u003Ch2>A way back for each release: feature flags, blue-green or canary\u003C\u002Fh2>\u003Cp>Choose the way back release by release, rather than picking one method for every change. The Azure guidance describes three ways to recover when a deployment causes a problem: roll back to the last known working configuration, roll forward by fixing the issue mid-rollout with something like a hotfix, or deploy new infrastructure from the last known working configuration. Each technique below makes one of those paths faster, and each has a different price.\u003C\u002Fp>\u003Cp>\u003Cstrong>Feature flags\u003C\u002Fstrong> switch features on or off selectively in production. Microsoft says they help control the exposure of new code and let you roll back quickly if issues arise. The price, as we see it, is that every flag is a branch in the code that has to be removed later. If flags are not removed, the number of possible flag combinations grows beyond what anyone can test. And a flag does nothing for you when the change is to the database structure or the infrastructure.\u003C\u002Fp>\u003Cp>\u003Cstrong>Blue-green\u003C\u002Fstrong>, as Azure describes it, means two identical sets of infrastructure, each able to handle the full production load. One runs the current version, the other runs the new version, and traffic is shifted across gradually. The direct price is paying for two full-size environments. The hidden price is that if both share one database, switching back is only fast when that database still works with both versions.\u003C\u002Fp>\u003Cp>\u003Cstrong>Canary\u003C\u002Fstrong> releases give the new version to a small group of users first, then expand to larger groups step by step until everyone has it. Azure notes that feature flags are typically used to decide who gets the new version. This limits the damage well, but every step needs its own bake time, and you have to be able to watch the health signals of the new-version group separately from everyone else. For that period two versions run side by side, so data written by the new version must still be readable by the old one.\u003C\u002Fp>\u003Cp>All of these depend on knowing when to stop. Azure recommends halting a rollout immediately when an issue is detected. Which signals should stop a release and wake someone up is a separate question, covered in \u003Ca target=\"_blank\" rel=\"noopener noreferrer\" href=\"https:\u002F\u002Fwww.superdev.tech\u002Fen\u002Fblogs\u002Fmonitoring-alerting-what-to-page-on\">our article on monitoring and alerting\u003C\u002Fa>.\u003C\u002Fp>\u003Ch2>Database schema changes: the part you cannot easily undo\u003C\u002Fh2>\u003Cp>The Azure guidance says that rolling back changes, especially database, schema or other stateful component changes, can be complex, and that your guidelines should give clear instructions on how to handle data changes. An example shows why.\u003C\u002Fp>\u003Cp>Picture a release that splits a single address field in the database into several fields, along with new code that writes data in the new format. Users save records for two hours, then the team finds a bug that forces a rollback. If only the code is rolled back, the old version does not know about the new fields and cannot see the records just saved. If the database is restored to its state before the release, those two hours of data are gone.\u003C\u002Fp>\u003Cp>Many Azure database services offer point-in-time restore, which Microsoft lists as something that can help you roll back. We treat it as the last resort, because a restore takes the whole database back to a single point in time, and anything written after that point has to be recovered by hand.\u003C\u002Fp>\u003Cp>Our approach is to split a schema change into several steps. First, add the new structure alongside the old one. Next, have the code write to both and read from both. Once that has passed its bake time, switch to reading from the new structure only. Removing the old structure is a separate release that comes later. Every step before the removal can still be rolled back. The cost is more releases, and temporary code that someone has to clean up.\u003C\u002Fp>\u003Cp>Deletion is the step you cannot reverse. In its guidance on decommissioning, Azure recommends treating every deletion as irreversible and taking a snapshot or backup before removal. We apply the same rule to dropping columns and tables, and we require the release request for that kind of change to state clearly that there is no way back.\u003C\u002Fp>\u003Ch2>Tell users before the change, not after something breaks\u003C\u002Fh2>\u003Cp>In our view, users who know in advance that a screen is about to change report problems more precisely than users who run into the change on their own. A good notice tells them what changes for them, when, which period the system may be unavailable, and who to contact if something looks wrong. Send it through the channel users actually read, not the one that is convenient for IT.\u003C\u002Fp>\u003Cp>We recommend telling the helpdesk or call center before general users, because they will field the first questions. For changes that affect how people work, such as a changed field on a form, give enough notice for supervisors to adjust their team's procedures. Communicating during an outage is a different matter, with its own roles and timing, which is covered in \u003Ca target=\"_blank\" rel=\"noopener noreferrer\" href=\"https:\u002F\u002Fwww.superdev.tech\u002Fen\u002Fblogs\u002Fdr-plan-roles-and-decisions-during-an-outage\">our article on DR plans and decisions during an outage\u003C\u002Fa>.\u003C\u002Fp>\u003Ch2>When this is more than you need\u003C\u002Fh2>\u003Cp>Everything above has a cost, and not every system justifies it. If yours is an internal system with a few dozen users, can stop briefly with advance notice, and has no time-critical work running on it, blue-green is not worth paying for two sets of infrastructure. Canary hardly means anything either, because the user groups are too small to show any signal. For a system like that, we recommend only a short checklist, a pre-release backup whose restore has been tested, and a release window users have been told about. The one thing we do not relax is splitting schema changes into steps, because data loss does not get smaller with the size of the system.\u003C\u002Fp>\u003Ch2>Objections we hear often\u003C\u002Fh2>\u003Cp>The first objection is that more approval steps make the team release more slowly. The concern is reasonable. If every approval has to wait for a weekly meeting, the team will bundle several changes into one release, which runs against Microsoft's recommendation of small, frequent deployments. The bigger the release, the harder it is to find what broke. The answer is not to drop approval, but to scale it to the risk of the change. Changes that repeat an established pattern can be pre-approved, and changes that touch the schema get more reviewers.\u003C\u002Fp>\u003Cp>The way to tell whether a process helps or only slows things down is to measure both sides. \u003Ca target=\"_blank\" rel=\"noopener noreferrer\" href=\"https:\u002F\u002Fdora.dev\u002Fguides\u002Fdora-metrics\u002F\">DORA\u003C\u002Fa> defines deployment frequency as \"The number of deployments over a given period or the time between deployments.\" It defines change fail rate as \"The ratio of deployments that require immediate intervention following a deployment. Likely resulting in a rollback of the changes or a 'hotfix' to quickly remediate any issues.\" And it defines failed deployment recovery time as \"The time it takes to recover from a deployment that fails and requires immediate intervention.\" If a new step lowers deployment frequency without improving change fail rate, all it has added is waiting time.\u003C\u002Fp>\u003Cp>The second objection is that there is a nightly backup, so a failed release can simply be restored. The answer is in the schema section above. A backup keeps you from losing everything, but it does not bring back what users saved after the backup was taken. A backup is not a release's way back. It is the last safety net when every other way back has failed.\u003C\u002Fp>\u003Ch2>Where to start\u003C\u002Fh2>\u003Col>\u003Cli>\u003Cp>List the differences between staging and production as they stand today, and attach the list to every release request.\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>Agree on a release request form that answers the six questions in the approval section, and require that the approver is not the person who wrote the change.\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>Build a freeze calendar with the business side, with an emergency path that names the approver.\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>Require every release to state its way back, and who decides to use it, before it goes for approval.\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>Split schema-changing releases into steps, and always make deleting data or structures a separate release.\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>Start tracking deployment frequency, change fail rate and failed deployment recovery time from your next release.\u003C\u002Fp>\u003C\u002Fli>\u003C\u002Fol>\u003Cp>If an external developer maintains your system, write these steps into the TOR or contract from the start, rather than negotiating them after the first failed release.\u003C\u002Fp>\u003Cdiv data-type=\"horizontalRule\">\u003Chr>\u003C\u002Fdiv>\u003Ch2>Summary\u003C\u002Fh2>\u003Cp>If you take one thing from this article, make every release answer a single question before anyone signs: if it breaks, how do we back out, and who decides? Releases that touch the database schema are where that answer is weakest, so they deserve more thought before release than an ordinary change.\u003C\u002Fp>\u003Cp>If you would like to talk through the release process for a system you already run, you can reach us at 088-983-9386 or contact@superdev.tech. Our office is in Bangkapi, Bangkok. If the conversation shows that the process your organization already has is enough for that system, we will say so plainly and will not propose work you do not need. More about our team and the platforms we work with is on the \u003Ca target=\"_blank\" rel=\"noopener noreferrer\" href=\"https:\u002F\u002Fwww.superdev.tech\u002Fen\">Superdev homepage\u003C\u002Fa>.\u003C\u002Fp>","https:\u002F\u002Ftwsme-r2.tumwebsme.com\u002Fpbc_2128795511\u002Faykouh8pi6d6j6d\u002Fcover_cover_xyiyq3yeke.webp","September 30, 2026","2026-09-30 06:18:49.628Z",104,"Enterprise","enterprise","production-change-release-management","\u002Fblogs\u002Fproduction-change-release-management",14,false,0,"",[47,48,49,50,51,52,53],"production deployment","blue-green deployment","canary deployment","rollback plan","release management","feature flag","change management"]