Skip to content

Instantly share code, notes, and snippets.

@tomsing1
tomsing1 / luigi_first_steps.md
Last active October 7, 2023 13:23
First steps with the Luigi workflow manager

First steps with the Luigi workflow manager

As an introduction into Luigi, I am following this tutorial with some modifications, e.g. installation using conda.

The problems and solutions described in the examples below have led to the development of sciluigi,

@mneedham
mneedham / blog.py
Last active July 11, 2022 03:04
Meetup API -> JSON -> CSV using Python's Luigi library
import json
import os
import luigi
import requests
from collections import Counter
from luigi.contrib.external_program import ExternalProgramTask
class Meetup(luigi.WrapperTask):
def run(self):
@ziadoz
ziadoz / install.sh
Last active April 20, 2024 10:18
Install Chrome, ChromeDriver and Selenium on Ubuntu 16.04
#!/usr/bin/env bash
# https://developers.supportbee.com/blog/setting-up-cucumber-to-run-with-Chrome-on-Linux/
# https://gist.github.com/curtismcmullan/7be1a8c1c841a9d8db2c
# https://stackoverflow.com/questions/10792403/how-do-i-get-chrome-working-with-selenium-using-php-webdriver
# https://stackoverflow.com/questions/26133486/how-to-specify-binary-path-for-remote-chromedriver-in-codeception
# https://stackoverflow.com/questions/40262682/how-to-run-selenium-3-x-with-chrome-driver-through-terminal
# https://askubuntu.com/questions/760085/how-do-you-install-google-chrome-on-ubuntu-16-04
# Versions
CHROME_DRIVER_VERSION=`curl -sS https://chromedriver.storage.googleapis.com/LATEST_RELEASE`
@hkage
hkage / pytest-cheatsheet.rst
Last active September 2, 2018 15:42
Cheatsheet for pytest features

pytest Cheatsheet

@pytest.fixture

  • scope: the scope for which this fixture is shared, one of “function” (default), “class”, “module”, “session”.
  • params: an optional list of parameters which will cause multiple invocations of the fixture function and all of the tests using it.
  • autouse: if True, the fixture func is activated for all tests that can see it. If False (the default) then an explicit reference is needed to activate the fixture.
  • ids: list of string ids each corresponding to the params so that they are part of the test id. If no ids are provided they will be generated automatically from the params.
@ewenchou
ewenchou / README.md
Last active July 8, 2023 04:36
Run Python script as systemd service
  1. Create a service file like dash_sniffer.service
  2. Put it in /lib/systemd/system/
  3. Reload systemd using command: systemctl daemon-reload
  4. Enable auto start using command: systemctl enable dash_sniffer.service
@gboeing
gboeing / pypi.md
Last active June 17, 2022 16:11
How to organize and distribution a package on pypi

To distribute a package on pypi

Directory structure

/project/
    /package/
        __init__.py
        module.py
 setup.py
@peterhurford
peterhurford / pytest-fixture-modularization.md
Created July 28, 2016 15:48
How to modularize your py.test fixtures

Using py.test is great and the support for test fixtures is pretty awesome. However, in order to share your fixtures across your entire module, py.test suggests you define all your fixtures within one single conftest.py file. This is impractical if you have a large quantity of fixtures -- for better organization and readibility, you would much rather define your fixtures across multiple, well-named files. But how do you do that? ...No one on the internet seemed to know.

Turns out, however, you can define fixtures in individual files like this:

tests/fixtures/add.py

import pytest

@pytest.fixture
@DamnedScholar
DamnedScholar / language-sampleGrammar.cson
Last active January 25, 2023 20:14
Grammar boilerplate with annotations.
# TextMate tutorial: http://manual.macromates.com/en/language_grammars
# Regex to convert keys to unquoted: '(include|match|captures|begin|end|beginCaptures|endCaptures|name|patterns|0|1|2|3|4|5|6|7|8|9|comment|fileTypes|scopeName|repository|contentName|firstLineMatch|foldingStartMarker|foldingStopMarker)':
scopeName: 'source.<scope>' # <scope> should be a short, unique indicator for the language ("js", "php", "c", etc.)
name: '<name>' # The title that will show up in grammar selection and on your status bar.
fileTypes: [ # An array of file extensions.
'txt'
'exif'
]
@chenjianjx
chenjianjx / start-celery-for-dev.py
Created March 10, 2016 10:45
A python script which starts celery worker and auto reload it when any code change happens.
'''
A python script which starts celery worker and auto reload it when any code change happens.
I did this because Celery worker's "--autoreload" option seems not working for a lot of people.
'''
import time
from watchdog.observers import Observer ##pip install watchdog
from watchdog.events import PatternMatchingEventHandler
import psutil ##pip install psutil
import os