Skip to content

Instantly share code, notes, and snippets.

@benhorne44
Created November 8, 2016 15:40
Show Gist options
  • Save benhorne44/e458b51d4b0d295687690e9a27ffa0e9 to your computer and use it in GitHub Desktop.
Save benhorne44/e458b51d4b0d295687690e9a27ffa0e9 to your computer and use it in GitHub Desktop.
Refactor the Custom View Process
- TL;DR; Solutions will be overseeing ALL custom view generation, exclusively or through pairing with an implementation team member with more defined releases
- Problem: There is a very large technical knowledge gap that is resulting in increasing demands from views without the appropriate knowledge of how to approach them with performance and stability in mind
- Performance: For example, I run the queries manually before posting them and query times for some recent data intensive views have been 80+ seconds.
- Biggest problem is that even though the datatable will time out, the query won't be killed in the DB. Often this leads to many page refreshes, and many queries. This has taken prod down, and we can't set timeout limits in the DB without affecting advanced reporting
- Customer Demands
- There are some things that exist in GoodData that people desire in the Reporting tab, but are not ideal on our postgres database
- ex: SWS all possible combinations of user-to-place (through missions to place groups, potentially with UAPs) and which missions they have completed a response at
- Bug Fixing
- Blame typically goes on Account Management and Solutions
- We usually have to fix them quickly, and it takes a LOT of effort to jump into something without context and without best practives
- Materialized Views
- MVs are being viewed as an alternative to writing performant queries, and should be a last resort after significant work on performance
- Urgency / Set release times
- Occasionally we are given a view on a Tues morning (release is Tues afternoon) and told that it is needed for a demo in 30 min
- Solutions:
- Set a limit on query times: if its more than 'x' (5 seconds? 10?) then it does not get posted. Not as a limit in the database itself, but would be part of the release process
- Some views are slower on prod with the DB bloat, so running the query manually in the console before connecting it to the Reporting tab is a great 1st step
- Solutions assigns 't-shirt' size and timelines to views
- What may seem small for a request may actually be a large resulting data set, or require historical tables that we currently don't keep track of in postgres
- Custom view 'code freezes' the afternoon before release
- Views to be released must be delivered on Mon/Wed afternoon to be considered for release the next day
- We have eyes on all views start to finish
- We are going to have 'Office Hours' where 1-2 of us will plan to be available (tentatively Mon/Wed mornings from 9-12) to build views and have them ready for Tues/Thurs releases
- Goal is to improve custom view quality while establishing best-practices and learning opportunities
- Long term goal:
- Set up a DB and/or separate app that does all of the queries and delivers content to the reporting tab (with whatever implementation makes the most sense)
- Read only access to WHITE through importing a foreign schema?
- Can set hard query time limits on the new DB (or let them run without affecting the webapp performance)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment