mamonu

@pytest.fixture(scope="session")
def spark_context(request):
    """ fixture for creating a spark context
    Args:
        request: pytest.FixtureRequest object
    """
    conf = (SparkConf().setMaster("local[2]").setAppName("pytest-pyspark-local-testing"))
    sc = SparkContext(conf=conf)
    request.addfinalizer(lambda: sc.stop())

mamonu

In a classic hadoop job, you've got mappers and reducers. The "thing" being mapped and reduced are key-value pairs for some arbitrary pair of types. Most of your parallelism comes from the mappers, since they can (ideally) split the data and transform it without any coordination with other processes.

By contrast, the amount of parallelism in the reduction phase has an important limitation: although you may have many reducers, any given reducer is guaranteed to receive all the values for some particular key.

So if there are a HUGE number of values for some particular key, you're going to have a bottleneck because they're all going to be processed by a single reducer.

However, there is another way! Certain types of data fit into a pattern:

• they can be combined with other values of the same type to form new values.
• the combining operation is associative. For example, integer addition: ((1 + 2) + 3) == (1 + (2 + 3)) - they have an identity value. (f

mamonu

mamonu

Keybase proof

I hereby claim:

• I am mamonu on github.
• I am mamonu (https://keybase.io/mamonu) on keybase.
• I have a public key ASAaMDZv4UsSo-R8axywMsNGasd119RopVvi6x-Ci_rDago

To claim this, I am signing this object:

mamonu

mamonu

Use the original Nord Modular Editor with OSX

Required

• Homebrew

Compatibility

Confirmed working with

mamonu

http://python-guide-pt-br.readthedocs.io/en/latest/scenarios/xml/

http://eli.thegreenplace.net/2012/03/15/processing-xml-in-python-with-elementtree

https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-ridge-lasso-regression-python/

http://www.nehalemlabs.net/prototype/blog/2014/04/03/quick-introduction-to-gaussian-mixture-models-with-python/

mamonu

zvex lofi:

1 Line select
2 Rack Comp THRSH 50 Ratio 5 Level 30 ATTCK 10
3 COMP Sense 10 Tone 5 Level 40 ATTCK Fast
4 Graphic EQ 160 -6 400 2 800 -2 3.2 -4 6.4 -8 12 -12 Level 100
5 Vibrato Depth 100 Rate 10 Bal 90 Tone 3 Level 120

-----------------------------------------

mamonu

http://localhost:8080/solr/update?stream.body=<delete><query>*:*</query></delete>
http://localhost:8080/solr/update?stream.body=<commit/>


#how to add linenumbers in bash
awk '{ print FNR "," $0 }' train_set.csv > clean.csv
#edit clean.csv and change 1 from header to line...did it by hand ...could automate this :)

mamonu

#!/bin/bash

mkdir my_project
cd my_project

echo " . . . Downloading file stanford-ner-2014-08-27.zip"
# NOTE: need to update link for further versions
wget http://nlp.stanford.edu/software/stanford-ner-2014-08-27.zip

echo " . . . Unpacking stanford-ner-2014-08-27.zip"