Subject: automagically logging student errors
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Tue, Jul 22, 2014 at 2:19 PM
To: Teachers <teachers@devbootcamp.com>
it occurred to me today that automatically capturing errors to a log file while students are working in phase 1 might help us build a list of the most common errors that students encounter, how they approach fixing them and how long they spend fixing them (ie, before the error changes).
it reminds me of a project that Keith Tom was working on last year that never saw the light of day called "snoopy"
with a little playing around i was able to get this to work but it's pretty janky.
anyone have any better ideas for how something like this might be implemented?
see here: https://gist.github.com/amgando/41770aefc7b1dc11592f
Steven Harms suggested that doing this in rails would be much easier but how would this work with the straight Ruby interpreter?
thoughts?
From: Matthew Weitzel <matthew@devbootcamp.com>
Date: Tue, Jul 22, 2014 at 3:05 PM
To: Sherif Abushadi <sherif@devbootcamp.com>
Cc: Teachers <teachers@devbootcamp.com>
I think it'd be more valuable earlier on.
Its fairly simple to wrap a ruby script, especially with ruby. Here's a thing: https://gist.github.com/mweitzel/48a0d33b966c3dacaab9
From: Myles Byrne <myles@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 11:50 AM
To: Matthew Weitzel <matthew@devbootcamp.com>
Cc: Sherif Abushadi <sherif@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Architecturally, here's how I'd go about it:
-
Rename ~/.rbenv/shims/ruby to ~/.rbenv/shims/__ruby (with the underscore)
-
Then write a shim "wrapper" at ~/.rbenv/shims/ruby that captures sterr (only) via tee and writes it /tmp/ruby-errors/{timestamp}.txt
-
Have some background process running on the machine that parses the files /tmp/ruby-err-output/{timestamp}.txt and posts them to keen.io
That way the worst that can happen is that /tmp/ruby-err-output/ fills up with files (which get deleted on reboot anyway)
I don't think it's wise to inject any ruby code (no matter how clever) into the ruby interpreter itself, there's going to be a ton of corner cases to consider.
From: Mike Busch <mike@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 12:03 PM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Matthew Weitzel <matthew@devbootcamp.com>, Sherif Abushadi <sherif@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Rather mucking with rbenv's shims, create our own ruby shim that uses rbenv's shim and comes before rbenv's bin in the $PATH. This decouples our "Snoopy" shim from the underlying ruby environment manager.
-- Mike
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 12:07 PM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
thanks Myles, that's really helpful.
i'll work with Topher in the next week or two to get this up in a way that aligns with our current machine config
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 12:07 PM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Thanks Mike!
From: Myles Byrne <myles@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 12:10 PM
To: Mike Busch <mike@devbootcamp.com>
Cc: Matthew Weitzel <matthew@devbootcamp.com>, Sherif Abushadi <sherif@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Yes. This is better. Something like /usr/local/dbc/bin
And if it's having problems you just rm/move the shim and everything goes back to normal
From: Mike Busch <mike@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 1:37 PM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Matthew Weitzel <matthew@devbootcamp.com>, Sherif Abushadi <sherif@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
And then I made bash things...
https://gist.github.com/mikelikesbikes/8f8756f558fd4f25d566
Sadly, this only works with "ruby" not with irb, rails, rspec, rake, etc. We'd need to build shims the same way that rbenv does for each potential use. I'd think this would also be useful for node output (for the js challenges), and other tools we use (postgres/sqlite3 logs). Although now that I'm enumerating these things, I'm essentially rebuilding the wishlist for data collection from Snoopy. :D
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 2:09 PM
To: Mike Busch <mike@devbootcamp.com>
Cc: Myles Byrne <myles@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Sweet, thanks Mike.
I assume I can remove the line that "tee's" to stdout since I only want to collect errors.
I'll play w this shortly
From: Mike Busch <mike@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 2:21 PM
To: Sherif Abushadi <sherif@devbootcamp.com>
Cc: Myles Byrne <myles@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Yep. Feel free to modify that however you'd like to log more/less stuff. :D
From: Topher Lubaway <topher@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 2:55 PM
To: Mike Busch <mike@devbootcamp.com>
Cc: Myles <myles@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Sherif Abushadi <sherif@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
+1 to mike’s solution.
Topher Lubaway
From: Myles Byrne <myles@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 6:03 PM
To: Topher Lubaway <topher@devbootcamp.com>
Cc: Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Sherif Abushadi <sherif@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Fucking excellent code Mr Bush, I removed the stdout capturing and added some commentary:
https://gist.github.com/quackingduck/898b805aa090e38f1e48
Two hard problems remain though:
- Collecting this data - it's going to be sitting in the tmp folder of 30ish workstations, plus it's going to be tricky to figure out when one error ends and another begins
- Analyzing this data and creating some kind of useful report
My suggestion here is to create a new log file in the tmp folder for every ruby run (instead of appending to the same file) and then write another program that runs continuously in the background and posts the contents of these files (if they're not empty) to keen.io and then deletes them. You'd also want to include the hostname and github usernames in the payload. Keen solves both collecting and reporting and we don't have to setup any schemas or run any server-side infrastructure.
From: Myles Byrne <myles@devbootcamp.com>
Date: Thu, Jul 24, 2014 at 6:04 PM
To: Topher Lubaway <topher@devbootcamp.com>
Cc: Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Sherif Abushadi <sherif@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
On Thu, Jul 24, 2014 at 6:03 PM, Myles Byrne myles@devbootcamp.com wrote:
Fucking excellent code Mr Bush
Name intentionally misspelled for comedic effect ~ Miles
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Fri, Jul 25, 2014 at 12:07 AM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Topher Lubaway <topher@devbootcamp.com>, Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
tried parsing this madness before i read Myles' version:
https://gist.github.com/amgando/97691939f273573aa64a
still seem to be missing a few things but the headache is subsiding :)
will try a few incantations tomorrow to adjust for "new log file in the tmp folder for every ruby run (instead of appending to the same file)"
also signed up for Keen IO and read a bit of their docs. couple of ideas come to mind:
-
tracking student progress through phase funnels defined by challenge repo names https://keen.io/docs/data-analysis/funnels/ https://keen.io/docs/data-analysis/timeframe/
-
suggested pairing sessions for students solving errors at either extreme: fast or slow, based on logging timestamps https://keen.io/docs/data-analysis/group-by/
-
comparing tracked info across cohorts to help us baseline and run vigorous experiments fearlessly https://keen.io/docs/data-analysis/cohorts/
of all of this is a hot mess of nothing til something works so i'll keep reading and breathing.
would be nice to see a bash masterclass come out for us noobs soon.
me like.
From: Mike Busch <mike@devbootcamp.com>
Date: Fri, Jul 25, 2014 at 8:19 AM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Topher Lubaway <topher@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Sherif Abushadi <sherif@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Thanks, Mr. Burn.
From: Myles Byrne <myles@devbootcamp.com>
Date: Fri, Jul 25, 2014 at 10:44 AM
To: Sherif Abushadi <sherif@devbootcamp.com>
Cc: Topher Lubaway <topher@devbootcamp.com>, Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
On Fri, Jul 25, 2014 at 12:07 AM, Sherif Abushadi sherif@devbootcamp.com wrote: tried parsing this madness before i read Myles' version:
https://gist.github.com/amgando/97691939f273573aa64a
added some commentary
also signed up for Keen IO and read a bit of their docs. couple of ideas come to mind:
keen is pretty great, we already have a DBC account on bitium, added you to that
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Fri, Jul 25, 2014 at 11:04 AM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Topher Lubaway <topher@devbootcamp.com>, Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
Thanks for the comments, Myles, that was helpful!
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Tue, Jul 29, 2014 at 5:24 PM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Topher Lubaway <topher@devbootcamp.com>, Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
I've added a line to also capture git user.email and timestamp to each entry.
https://gist.github.com/amgando/2ada43c7b56e1dc002fb#file-ruby-L36-L37
any concerns with this? to be sure, i'm no less baffled by this madness.
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Tue, Jul 29, 2014 at 5:25 PM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Topher Lubaway <topher@devbootcamp.com>, Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>
well, actually Topher and Steven added the 2 lines. I was only taking the credit.
From: Myles Byrne <myles@devbootcamp.com>
Date: Wed, Jul 30, 2014 at 1:00 AM
To: Sherif Abushadi <sherif@devbootcamp.com>
Cc: Topher Lubaway <topher@devbootcamp.com>, Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>, Steven Harms <steven@devbootcamp.com>
Left feedback
From: Sherif Abushadi <sherif@devbootcamp.com>
Date: Wed, Jul 30, 2014 at 7:28 AM
To: Myles Byrne <myles@devbootcamp.com>
Cc: Topher Lubaway <topher@devbootcamp.com>, Mike Busch <mike@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>, Steven Harms <steven@devbootcamp.com>
That's great Myles, thanks for the detail!
Of course it all makes sense, from keeping conventions to minimizing downstream work.
What's even better is I've now found out how to get your undivided attention :)
Do the students know about this?!
From: Mike Busch <mike@devbootcamp.com>
Date: Wed, Jul 30, 2014 at 11:09 AM
To: Sherif Abushadi <sherif@devbootcamp.com>
Cc: Myles Byrne <myles@devbootcamp.com>, Topher Lubaway <topher@devbootcamp.com>, Matthew Weitzel <matthew@devbootcamp.com>, Teachers <teachers@devbootcamp.com>, Steven Harms <steven@devbootcamp.com>
At this point, can we get this the hell out of gists and into some meaningful place (kitout), so that we can contribute via Pull Requests?
-- Mike