gistfile1.txt

Problem:

Drastically fewer segment rows since 22nd May.

Some facts:
1. The raw Appnexus Segment files we download from AppNexus data siphon are still about the same size as before.
2. The extracted Segments files are now much smaller (1/1000th of former size).
3. Running the extract on older files reduces the extracted file size.
=> The extract is causing the problem.
(https://github.com/infectious/etl/blob/master/etl/apn/data_siphon/segments/extract.rb)

There is a commit at exactly the time the problem started:
https://github.com/infectious/etl/commit/b8460a79f1fa44909f5ddb3d48e443a2560af42c
The commit changes the parse format.

We note that removing one of the transform :select conditions removes the problem:
ie. change L34 to the following:    
 row[1] > 0 #&& segments.key?(row[3])
and the extracted file size returns to normal. But this is now too permissive, as inactive or obsolete segments are being allowed through.

The timing points to the new parse format not working for row[3] such that the equivalence 
segments.key?(row[3])
no longer functions. But testing has not been able to isolate this (it works in console, which is confusing). 

What might be going on?