- WA-4929 / NF-18593 was merged and Michael verified it.
- Made good progress on NF-14938:
- Changed the designation/requirements job introduced in WA-4929 to work by applicant to not double process requirements owned by multiple programs and to take advantage of more eager loading and caching.
- Found out that the join table between programs and requirements allows duplicates, and actually there are 428k join records but only 64k of them are unique. Without doing anything, we would be processing records based on the extra joins so I made sure we were only processing unique jobs.
- Made a "performance" / integration type test that performs the job and counts the number of SQL queries executed since it's a good indicator of most of the performance issues we see. This is going to be very helpful identifying what else should be fixed.
- Today is national scrum meeting day: Standup, grooming, pre-planning and retro meetings today.
- Since today is broken up a lot by meetings, rather than get deep into development I want to:
- Do a couple tricks I learned about on Friday from some blog posts to identify exactly which lines of code are causing which SQL queries. When I combine this with the performance tests done on Friday I should be able to identify some pain points.
- Run a general Ruby-prof test that takes into account all code not just SQL in case there are other issues that need to be solved.
- I should have a pretty good plan on the rest of the steps I need to take after today which will put me in a good spot tomorrow.
- Srikanth is also trying to test WA-4929. Not sure if that is necessary / on purpose or not. Need to either let him know it's already been tested or I need to provide better QA instructions.
- Paired with Michael on the list errors that Alok had seen. They appear not to be code-related but rather environmental. I'm going to continue monitoring production through Friday.
- Setup Borat / Regular server Resque workers to handle new queue introduced in WA-4929. Verified that monster/k8 envs will work by default.
- WA-4929 was re-reviewed by Jason and he had some great feedback that I ended up implementing and ultimately got his +1.
- WA-4929: Also wrote up detailed QA instructions and spun up a K8 environment for testing. Realized the branch had not been rebased from master in a while and was running into ETL errors in my environment so I rebased to bring it current.
- Started some brainstorming on NF-14938. It's running a job for each of the applicant's designations. But multiple designations could require the same requirement/prereqs to be run so there may be some duplicity there. I want the job to run at the applicant level instead. This will also provide more opportunities for eager-loading and caching data in the worker.
- I will try spinning up a K8 for WA-4929 again and hopefully the errors will be resolved after having rebased. I'll verify functionality and then set as Ready for QA.
- WA-4929 is my first priority, and then I will work on implementing my ideas for NF-14938.
- Dan Q has some really nice looking changes for GPA jobs, but since I'm behind on planned sprint work I don't want to focus too much on these nice to have things for the sprint. I figure when I'm done my prioritized work I'll look at this next. Even if Dan gets things partially done, it's going to give us a really nice head start on the GPA jobs.
- Finished changes I wanted to make to WA-4929 and approved both WA and ETL PR's.
- Met up with Ken to discuss how scalability things are going.
- Paired with Mehar for a couple hours in the afternoon and we discussed how my recent changes to eliminate accessibility scoping for some fields could be used in exports. We did a couple of quick tests and had some really promising results. We also discussed a slight change to export testing by focusing more on the actual CSU data which has less applicants but more realistic data. Mehar is finding that exports scale linearly with the number of applicants so this probably makes sense.
- The last hour of the day, I paired up with Ben K to fix an ETL issue that QA was having with the AAMC-QA environment.
- The last step dev step for WA-4929 (assuming 2nd code review doesn't turn up more) is to adjust the Resque workers in all environments to handle the new worker type. I plan on doing this today.
- Finally start on NF-14938.
- We have one hour of preplanning/retro meetings in the afternoon.
- None