Skip to content

Instantly share code, notes, and snippets.

@ShonM
Created March 30, 2017 17:38
Show Gist options
  • Save ShonM/29a3d61867e394f2550cbf90738b9991 to your computer and use it in GitHub Desktop.
Save ShonM/29a3d61867e394f2550cbf90738b9991 to your computer and use it in GitHub Desktop.
todo
Before next release:
- Brain needs an error handler. Consider #vanguard
- Automated tests for everything
- Lock/unlock/enable/disable
- Claim
- GET /claim?token=123 (Jira)
- POST /api/v1/stacks/claim (Hubot)
- Deploy/rollback
- Webhooks from GH, Jira, and Brain
Backlog:
- Brain should handshake package hosts before starting
- There should be a quick check step to verify application state across all Neurons
- Top level directory structure should match
- State file should match
- Judo add a "what's this?" link to a staff-only page explaining how to use Judo
- Devchess: "reactive" properties on entities to mark which kind of updates are eligible for socket push?
- Devchess "Go to" button on Stacks, using the environment switch lander?
- That environment switch lander like /switch/production should allow GET but render a page that shows "Preview -> Production" with a confirmation button or something
- Wiki graphic showing how many of who go to which environment. Infographic style. Maybe right on Devchess as well?
- Put socket on location socket/ in nginx. Avoids separate subdomain.
- Brain: Heartbeats could be published and real-time on daemons page
- Static pinned stack locations (Preview -> Staging -> Production)
- Preview environment for preview
- Disabled reason, like note. When disabling automatically, set a reason. When reporting that a Stack is disabled, include the reason and person.
- Log views should be a hard navigation
- Health check cron for Devchess, alert when stacks are unhealthy
- Devchess regular logging where audit logs would have been, stream events for other important things
- Use runSilently for more commands, but use a variable for the original command and include it in any halt/error calls
- Devchess: How does AutoQA work multi-tenant? Have we assumed ChessCom/chess as the repository? Have we assumed "CV-"? Can we make this configurable?
- Brain: Deploy does not go directly to the included commit
- Bower+NPM -> Yarn
- Page titles / breadcrumbs (history, forms)
- More help text on forms / controls
- After changing a Stack, user loses authorization - can we retain?
This is actually due to events coming across the wire
The solution is likely to only use data from socket events that isn't related to auth
So only replace the inner annotated item? How?
The answer is: We need to implement authorization annotation on the receiver?
window.context.user can help here (user on deploy team or not)
- DD metrics for all Devchess actions (include if via Hubot - need Hubot UA for API hits?)
- Gravatars for System and AutoQA
- System activity for crons auto unlock etc.
- Hubot: Check if Stack should notify Slack before blindly doing so
- Should we replace Hubot entirely with slash commands?
App: https://api.slack.com/apps/A1X4MDQRY/general
Still ChatOps-y by showing the commands http://stackoverflow.com/questions/35682927/slack-slash-commands-show-response-only-to-channel
Would be less typing, "@hubot deploy prod" -> "/deploy prod"
Removes Hubot entirely
- Stack purpose/explanation? "Auto deploys develop" ex https://s3.amazonaws.com/heroku-devcenter-files/article-images/1456226930-Multiple-production-apps.png
- Deploy: Explode logging by newline
- Judo: Save open state in session, load faster?
- User info in tooltips to avoid "I don't know this username"
- Can we use reactphp for threading stop signals etc? Listen for signals starting at boot?
- Brain comms entirely with queues?
- Write synapse as a Deployer plugin? Something like push and wait on Redis instead of SSH or whatever
- Rename synapse and neuron? Neuron is runner... Synapse is controller? Scheduler.
- "deploy production from staging" to deploy production from stagings current commit
- When receiving a deploy event, it should be dispatched as deploy.success, deploy.failure, etc.
Instead of checking deploy state in every single listener :(
This probably requires sending more specific events from Brain
deploy.success instead of deploy.finish
deploy.step.success and deploy.step.failure instead of deploy.step_start and deploy.step_finish
- Pretend mode deploys should wait 1 minute for each step (in 1s intervals)
- Write out all uses of Twig extensions etc. and rely on just JS/API
- Similarly, write out all uses of flash messages (including an all-AJAX rewrite?)
- Can we show who is currently viewing each page?
http://stackoverflow.com/questions/18649705/node-js-socket-io-adding-username
http://stackoverflow.com/questions/18093638/socket-io-rooms-get-list-of-clients-in-specific-room
- Hubot: Roll more info into the status command (jira, pr, etc.)
- Option to notify when deploys head to certain environments
- Per-deploy notifications:
Notify channels/people about this deploy (without needing it enabled on the Stack)
- "Notify me" button on deploy start notification. Slack button API
- Tasks
- Tasks on the Devchess side need more payload data added (branch, environment)
- Tasks queued need to add an activity event to the Stack's feed
- Additionally, there should be a way to queue a task across certain roles of a Stack
- For example, queueing a Resque restart across all Neurons with [worker]
- Could this be more automatic with the task roles already in place?
- Logs
- One log for Synapse
- One log for each Neuron for the entire deploy (how it is now)
- One log for each step of a sequence
- Display just one hosts logs for each step?
- Every single action/change needs some kind of timeline event
Reading
https://github.com/octref/vetur/blob/master/README.md
https://github.com/codecasts/spa-starter-kit/blob/develop/README.md
https://m.signalvnoise.com/feature-flags-e09fd463183f
https://launchdarkly.com/featureflags.html
https://signalfx.com/
http://www.wercker.com/
http://stackoverflow.com/questions/42324454/uwebsockets-push-events-to-server
https://github.com/feross/standard/blob/master/README.md
https://developers.google.com/web/tools/chrome-devtools/device-mode/emulate-mobile-viewports#media-queries
https://github.com/ChessCom/chess/pull/18010#issuecomment-288500336
https://www.emailthis.me/open-source/extension-boilerplate
Inspiration
DeployBot
https://deploybot.com/images/ss-dashboard.png
https://deploybot.com/images/ss-release.png
http://sanasol.ws/wp-content/uploads/2016/03/ea5e8eba38b1f022d4cc898bac48.png
Azure
http://tattoocoder.com/content/images/2016/05/deploy_log_azure.png
Features
- Abstract the keys needed for feature voters? Stop using attributes?
- Features should use the compile flag to determine static vs dynamic features. Automatically determine? Hot features should still use an APCu microcache, like 10 seconds, configurable
- Feature preloading. Fetch all features that are marked preload at the start, saving individual requests
- Feature metrics. DD every check and result. This should effectively show adoption rate
- Feature last access time with SETEX?
- In FM itself we should detect NR and custom annotate any checked features and their state. Query in nrql
- What if we used a feature toggle saas but our own strong caching and invalidation? APCu micro, Redis regular reads. Daemon populates Redis from saas API
- Feature flags are useful to avoid rollback, which is not just your code
- If we move the feature storage to Redis only, other clients can interact. Do we need a feature API? Can that be fast enough?
Notes
Exporting deploys and re-importing them
Export
- Remote: `redis-cli --raw dump "deploy:deploy:20170327080750" > outfile && truncate -s -1 outfile`
- Local: `scp deploy1:~/outfile . && cat outfile | redis-cli -x restore "deploy:deploy:20170327080750" 0`
- Set the deploy ID in MySQL
Import
- redis-cli --raw dump "deploy:deploy:20170327080750:deploy3.chess.com-preview:log" > outfile && truncate -s -1 outfile
- scp deploy1:~/outfile . && cat outfile | redis-cli -x restore "deploy:deploy:20170327080750:deploy3.chess.com-preview:log" 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment