Sprint Progress

sprint.md

New Services/Projects:

• Configuration REST API is up and online within cluster. You can access it from any node in the cluster at api.internal.o18m.com
• Ruby configuration API client: https://github.com/methodmill/ConfigurationApiClientRuby
• Scala configuration API client: https://github.com/methodmill/
• New Jobqueue server https://dashboard.methodmill.com/client-jobqueue/

Bugfixes

Client Dashboard:

• Fix broken link in retrieve password email password recovery process
• Fix 500 error which would occur when user updated account in dashboard
• Fix issue where client credential encryption was applying Base64 encoding twice
• Fix some foreign key constraint issues in PG database for custome dashboard

DevOps/Networking

• Fix missing entry in routing table for Client gateway

Product/Service Updates

Client Dashboard

• New queueing service with background job server to process tasks and jobs from Client dashboard (dashboard.methodmill.com/client-jobqueue)
• Client-facing Redshift connections can now be created and managed from the Admin dashboard
• Connection details for SSH jumpbox can be created and updated within the admin interface, and associated with relevant data sources needing SSH
• Integrate with Chronos API to automatically create a new job when a data source is validated
• Add logging capability to view history of created/validated client connections
• Show chronos settings in dashboard
• Send internal email when a customer creates a new data source
• Client organizations can now set a timezone from settings
• Improve theme & look and feel

Client Connection Validator

• Add script to enable dynamic SSH tunnelling across multiple servers for clients who are using jumpbox hosts (e.g. Draft)
• Update connection validation service to better interface with Redis Queue

Redshift Loader

• Allow EscaleSEO's database to be loaded, minus the rows with invalid dates
• Allow SSH tunnelling

Dev-ops

• Propagate SSH keys within cluster for easier management
• Add network level logging in Cloudwatch

Upcoming

• Phase out Zookeeper in favor of configuration REST API
• Integration testing of "turnkey" automation: Dashboard, configuration validation/propagation, job and tasks allocation on cluster
• Sanitization step for Redshift loader
• SSH tunneling
• Allow clients to access Redshift credentials from Dashboard
• Staging cluster and staging Redshift
• Site improvements

Client updates

• Beam: Loader is working great need to give them credentials
• EscaleSEO: Have access to S3, not databsae, likely they haven't added our IP to their security group
• Am I missing anything?