name: center layout: true class: center, middle, title --- name: title template: center class: inverse background-image: url(images/loop.gif) # Strategies for Frequent, Zero Down Time, Deployments .footnote3[Nick Humrich] .footnote2[Canopy - Tools and Infrastructure] .footnote[Big Mountain Data - 11/2/2019] ??? welcome --- template: center # Two types of downtime ### Deployment ### Systems --- # Why Downtime? ??? * The default tutorial for web apps is a simple script. Doesnt go over deployment. * We like downtime because it makes our jobs easier * Less work * zero downtime is a lot of work, ergo scheduled downtime is "faster" --- class: normal ## Common Misconception: # Scheduled Downtime is "faster" in regards to development time ??? * Tell story of "scheduled maintainence" rollback that happened over and over, because every time there was an unforseen issue. (prod data changes) --- layout: false class: normal # The Downtime Death Trap * Downtime -> Deploy Less Often * Deploying Less -> Deploy Big * Deploying Big -> Higher Risk * Higher Risk -> Fear * Fear -> Suffering * Dark Side ??? * In reality, fear leads to Deploying less often, which begins this giant cycle of death, up until everyone hates deploys, and your doing it far far too little, potenially actually killing the company --- layout: false class: normal # Continous Delivery * Deploying Often -> Shorter LT * Shorter LT -> Smaller commits * Small commits -> Small risk * Small impacts -> Easier to test * Testing -> Less stress * Better, Faster, Stronger ??? LT = Lead Time --- template: center # "If it's painful, do it more often." ??? - we DONT deploy often, because its painful/scary - you never fix your deployment woes, because its too infrequent, so things get worse - Doing thing more often, makes you fix it, out of self preservation --- template: center # No Downtime! ??? Commit to no downtime deploys. * Wouldn't you rather deploy in the middle of the day, during work hours? * Wouldn't you rather alarms happened during work hours, when everyone is there. Not in the middle of the night? * Wouldn't you rather deploy that one stupid change, right away, without having to wait for the release train? --- template: center # How? --- background-image: url(images/blue_green_deployments.png) ### Blue/Green Deployment ??? * Deploy everything you need to pre-prod, then swap * Two environments, you swap suddenly * DNS can be troublesome * Load balancer swaps (careful!!: sometimes causes milliseconds of downtime) * Con: At large scale, your paying for two environments * Could use the other as "staging" though * Have to keep two environments completly the same (including scale) * There are gradiants to blue/green, where instead of a hard flip, you gradually move green * Some do it with two databases, some dont * You can do web/app peices independently --- background-image: url(images/rolling-deploy.png) ### Rolling Deployment ??? * Swap an instance/container out at a time * Uses some sort of LB (careful: could cause milliseconds of downtime) * Have to be careful of connection draining (but its easier than blue-green) * Connection timeouts to down containers/instances * healthchecks * --- class: normal # Non-Patterns * process swap * git push * directory symlink --- class: normal # Tools * Kubernetes * Elastic Beanstalk * Heroku --- template: center # Backwards compatibility ??? * Blue green you have the issue of already loaded js. The js isnt going to change when you deploy, so you still have to be backwards compatible * both are live at a time if you use dns ^ * rolling deploy you have muliple versions live at a time. So even css/etc needs to be compatible. * databases need to also be this way, not just api's --- background-image: url(images/blue_green_deployments.png) ### Blue/Green Deployment ??? * Even though you have a hard swap, you have two running. DB needs to be backwards compatible, or else you have a race condition. * If you use dns. Both env's will still be in use at a single time * Front-end assets are typically loaded already in the browser, so your api has to be backwards compat --- background-image: url(images/rolling-deploy.png) ### Rolling Deployment ??? * You have two versions of your code running side by side. Everything has to be completely backwards compatible. --- name: n layout: true class: normal --- # Ways you need to change your application * Stateless * Use cdn for front-end assets * Fully backward compatible API's * Fully backwards compatible database schema changes * Healthchecks --- template: n .left-column[ ## Stateless ] .right-column[ * Shy away from sticky sessions/connection draining * Use databases for *all* state management (i.e. sessions) ] --- .left-column[ ## Stateless ## CDN ] .right-column[ * Use a CDN for front-end assets * This may be self-hosted * Everything should be versioned * Store every version, forever ] --- .left-column[ ## Stateless ## CDN ## BC APIs ] .right-column[ * No renaming/removing fields in json * Adding new fields is ok * Maintain basic object structure * Use api versions, and maintain old ones forever ] --- layout:false .left-column[ ## Stateless ## CDN ## BC APIs ## BC Schema Changes ] .right-column[ ```python CREATE TABLE genres ( id SERIAL PRIMARY KEY, name varchar(200), ) ‏‏‎ ``` ] --- .left-column[ ## Stateless ## CDN ## BC APIs ## BC Schema Changes ] .right-column[ ```python ALTER TABLE genres ALTER COLUMN name varchar(300); ‏‏‎ ``` ] --- template: n .left-column[ ## Stateless ## CDN ## BC APIs ## BC Schema Changes ] .right-column[ * No "data" migrations or serious schema changes * If needed, use 3-step deploy process * Run migration __before__ deploy, not during ] --- template: n .left-column[ ## Stateless ## CDN ## BC APIs ## BC Schema Changes ] .right-column[ # 3-step deploy process 1. Support both ways 2. Move data over 3. Remove the old ] --- .left-column[ ## Stateless ## CDN ## BC APIs ## BC Schema Changes ## Healthchecks ] .right-column[ ## - Continously check instance/container health ## - Auto cancel/rollback deploys on bad healthchecks ] --- layout: true template: center --- # Do it often ### Increase Confidence --- # Small Batch Sizes --- # Artifacts (i.e. containers) --- # Rollbacks ??? You need to have a very good, easy rollback strategy so that its always an easy plan b --- # Shadow Deploys --- # Feature Toggles ## Seperate releases from deployments ??? - separating releases from deploys - allows deployment schedule to be decoupled from business release schedule - smaller commits - usually required wen doing MBD --- # Pipelines/Environments --- # Worker Tier --- # Testing ??? - You can never test enough - Contract testing is a must - Keep in mind the pyramid - Load tests/performance tests/etc. --- # No manual steps in pipeline ??? - manual steps lead to delayed/larger deploys. - Pipelines can take longer, if they are fully automated - Leads to higher quality over time - getting rid of manual can feel scary, but leads to better code reviews and quality over time --- # Master (Trunk) Based Development ??? - why MBD (or trunk-based) is required for CD - How this effects environments - why not feature branches --- template: section # Thanks .pull-right[ Big Mountain Data & Dev 2019 @nhumrich nick@humrich.us ]