hnykda

$ pyxecutor exec_py data/input/q4_2017/reduced/core.hdf ../lagoon/core/q4_2017/desktop_1/ data/output/q4_2017/desktop_1.hdf                                                                                     <<<
20180105 22:24:58:git.cmd:DEBUG - Popen(['git', 'version'], cwd=/home/dan/prac/gwi/pyxecutor, universal_newlines=False, shell=None)
20180105 22:24:58:git.cmd:DEBUG - Popen(['git', 'version'], cwd=/home/dan/prac/gwi/pyxecutor, universal_newlines=False, shell=None)
20180105 22:24:58:s3fs.core:DEBUG - Open S3 connection.  Anonymous: False
20180105 22:24:58:pyxecutor.io:DEBUG - Getting data/input/q4_2017/reduced/core.hdf
20180105 22:24:58:pyxecutor.io:DEBUG - Loading file from local file system
20180105 22:25:01:pyxecutor.dsl.ops:DEBUG - Assigning only to 85284 rows
20180105 22:25:01:pyxecutor.dsl.pipeline:DEBUG - 1: Assign(condition=(q5d_1==1 | q5d_2==1 | q5d_3==1 | q5d_4==1 | q5d_5==1), target=q999_99, value=1)
20180105 22:25:01:pyxecutor.dsl.ops:INFO - 85284 rows were selected
20180105 22:25:01:pyxecut

hnykda

dan at think460s in ~/prac/gwi/pyxecutor (feature/add-logging-and-skip-exceptions●) 
$ SKIP_ERRORS=True INIT_DEBUG=True SAVE_INTERMEDIATE_DATASETS=True pyxecutor exec_py ../pyxecutor/data/input/q4_2017/reduced/core.hdf  ../lagoon/core/q4_2017/desktop_1/ data/output/q4_2017/desktop_1.hdf 
20180108 22:48:54:git.cmd:DEBUG - Popen(['git', 'version'], cwd=/home/dan/prac/gwi/pyxecutor, universal_newlines=False, shell=None)
20180108 22:48:54:git.cmd:DEBUG - Popen(['git', 'version'], cwd=/home/dan/prac/gwi/pyxecutor, universal_newlines=False, shell=None)
20180108 22:48:54:s3fs.core:DEBUG - Open S3 connection.  Anonymous: False
20180108 22:48:54:pyxecutor.io:INFO - Loading '...lagoon.core.q4_2017.desktop_1.main' module ...
20180108 22:48:54:pyxecutor.main:ERROR - the 'package' argument is required to perform a relative import for '...lagoon.core.q4_2017.desktop_1.main'
Traceback (most recent call last):
  File "/home/dan/prac/gwi/pyxecutor/pyxecutor/main.py", line 72, in respond
    res = actions.env_exec(action, *arg

hnykda

20180118 14:03:39:pyxecutor.actions:INFO - Processing desktop_mobile_ext_4///spss/2.5 - Straightlining.sps -> desktop_mobile_ext_4//straightlining.py ...
20180118 14:03:39:spss2py.preprocessing:INFO - Preprocessed 784 lines
20180118 14:03:39:spss2py.preprocessing:INFO - Total number of lines in the output file: 705
20180118 14:03:39:spss2py.preprocessing:INFO - Total number of lines skipped: 79
20180118 14:03:39:spss2py.parser:WARNING - `q1108a_1_5,
q1108j_1_5,
q1108t_1_5,
q1108u_1_5,
q1108a2_1_5,
q1108v_1_5,

hnykda

20180204 02:07:02:pyxecutor.dsl.utils:DEBUG  [country_specific_screener:677] Assign(condition=((q3<18 | (s2_1==1 & q3<21) | s2_966==1 | s2_971==1 | s2_20==1) & q1017_1_1==1), target=q1017_1_1, value=0)
20180204 02:07:02:pyxecutor.dsl.ops:DEBUG  Ouch. Could not use simple mask. Falling back to pandas one.
20180204 02:07:03:pyxecutor.dsl.utils:DEBUG  [country_specific_screener:678] Assign(condition=((q3<18 | (s2_1==1 & q3<21) | s2_966==1 | s2_971==1 | s2_20==1) & q1017_1_2==1), target=q1017_1_2, value=0)
20180204 02:07:03:pyxecutor.dsl.ops:DEBUG  Ouch. Could not use simple mask. Falling back to pandas one.
20180204 02:07:03:pyxecutor.dsl.utils:DEBUG  [country_specific_screener:679] Assign(condition=((q3<18 | (s2_1==1 & q3<21) | s2_966==1 | s2_971==1 | s2_20==1) & q1017_1_3==1), target=q1017_1_3, value=0)
20180204 02:07:03:pyxecutor.dsl.ops:DEBUG  Ouch. Could not use simple mask. Falling back to pandas one.
20180204 02:07:04:pyxecutor.dsl.utils:DEBUG  [country_specific_screener:680] Assign(condition=((q3<18 | (s

hnykda

gwi-123

# raw data from qualtrics:

```
respondent_id,q2,q3,s2,panelprovider
respid-1,1,0,44,ondevice
respid-2,2,,1,usamp
respid-3,2,,3,usamp
```

hnykda

In [24]: %time df.loc[:, 'cc'] = np.full(df.shape[0], pd.np.nan, dtype='float16')
CPU times: user 286 µs, sys: 5.47 ms, total: 5.75 ms
Wall time: 4.41 ms

In [25]: %time df.loc[:, 'ccc'] = np.full(df.shape[0], pd.np.nan, dtype='float16')
CPU times: user 24.5 ms, sys: 19.2 ms, total: 43.7 ms
Wall time: 39.5 ms

In [26]: %time df.loc[:, 'cccc'] = np.full(df.shape[0], pd.np.nan, dtype='float16')
CPU times: user 492 ms, sys: 1.2 s, total: 1.69 s

hnykda

worker_1     | 20180625 15:52:42:urllib3.connectionpool:DEBUG http://es:9200 "POST /q4_2016/_doc/_update_by_query HTTP/1.1" 404 351
worker_1     | 20180625 15:52:42:elasticsearch:WARNING POST http://es:9200/q4_2016/_doc/_update_by_query [status:404 request:0.038s]
worker_1     | 20180625 15:52:42:elasticsearch:DEBUG > {"query":{"bool":{"should":[{"exists":{"field":"asked.test-project"}},{"exists":{"field":"test-project"}}],"minimum_should_match":1}},"script":{"source":"\n                        def fields = new ArrayList(ctx._source.keySet());\n                        if (params.fresh_sample_fields.containsAll(fields)){\n                            ctx.op = 'delete';\n                        } else {\n                            ctx._source.remove(params.project_id);\n                            ctx._source.asked.remove(params.project_id);\n                        }\n                    ","lang":"painless","params":{"not_exist_project_id":"invalid_project","project_id":"test-project","fresh_sample_fields":["a

hnykda

def migrate_data(apps, schema_editor):
    Tracker = apps.get_model('campaign', 'Tracker')
    Syncer = apps.get_model('campaign', 'Syncer')
    Key = apps.get_model('attribute', 'Key')
    for k in Key.objects.all():
        c = k.parent_campaign
        if c.campaign_type == 'Panel':
            new_camp = Syncer.objects.get(cid=c.cid)
        else:
            new_camp = Tracker.objects.get(cid=c.cid)

hnykda

Dec 02 11:20:45 think460s audit[2167]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=1 pid=2167 comm="chromium" exe="/usr/lib/chromium/chromium" sig=4 res=1
Dec 02 11:20:45 think460s kernel: BUG: unable to handle kernel paging request at 0000000000020008
Dec 02 11:20:45 think460s kernel: PGD 0 P4D 0 
Dec 02 11:20:45 think460s kernel: Oops: 0000 [#1] PREEMPT SMP PTI
Dec 02 11:20:45 think460s kernel: CPU: 3 PID: 19433 Comm: rescuetime Tainted: G           OE     4.19.4-arch1-1-ARCH #1
Dec 02 11:20:45 think460s kernel: Hardware name: LENOVO 20FAS0FD07/20FAS0FD07, BIOS N1CET59W (1.27 ) 08/31/2017
Dec 02 11:20:45 think460s kernel: RIP: 0010:rb_next+0x27/0x50
Dec 02 11:20:45 think460s kernel: Code: 1f 40 00 48 8b 0f 48 39 cf 74 38 48 8b 47 08 48 85 c0 74 22 48 8b 50 10 48 85 d2 74 0c 48 89 d0 48 8b 50 10 48 85 d2 75 f4 c3 <48> 3b 79 08 75 12 48 8b 09 48 89 d7 48 83 e1 fc 48 89 ca 48 89 c8
Dec 02 11:20:45 think460s kernel: RSP: 0018:ffffbae98d4dbd40 EFLAGS: 00010206
Dec 02 11:20:45 think460s kernel: RAX: 00000000000200

hnykda

-- Logs begin at Sat 2017-10-07 14:26:40 CEST, end at Sun 2019-01-13 09:46:32 CET. --
Jan 13 09:45:51 think460s kernel: Linux version 4.20.0-arch1-1-ARCH (builduser@heftig-29859) (gcc version 8.2.1 20181127 (GCC)) #1 SMP PREEMPT Mon Dec 24 03:00:40 UTC 2018
Jan 13 09:45:51 think460s kernel: Command line: initrd=\initramfs-linux.img initrd=\intel-ucode.img root=PARTUUID=8d411a1d-fd6b-4916-b21f-f745a70a39d3 rw intel_iommu=off
Jan 13 09:45:51 think460s kernel: KERNEL supported cpus:
Jan 13 09:45:51 think460s kernel:   Intel GenuineIntel
Jan 13 09:45:51 think460s kernel:   AMD AuthenticAMD
Jan 13 09:45:51 think460s kernel:   Hygon HygonGenuine
Jan 13 09:45:51 think460s kernel:   Centaur CentaurHauls
Jan 13 09:45:51 think460s kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Jan 13 09:45:51 think460s kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'