tuxdna

Examples from this Youtube Playlist

https://www.youtube.com/watch?v=6rkyHJouYAQ&list=PLIO3UV9ODwNCJOH_EzE_T_fK4-g-G-xKf&index=2

tuxdna

Data Engineering problem from a LinkedIn post

Original LinkedIn discussion: here

I am creating this document for brainstorming and future reference.

Problem Description

As a data engineer how do you solve this problem:

tuxdna

# run tests uisng: pytest -vv combine_lists.py 

from typing import List, Any
import itertools


def combine_lists_of_lists(*args: List[List[Any]]) -> List[List[Any]]:
    results = [list(itertools.chain(*tup))
               for tup in itertools.zip_longest(*args, fillvalue=[])]
    return results

tuxdna

Question:

Prompt: Hello, I study to become ML Engineer. As someone with your experience, what would be your advise to someone like me who wants to be in that field but doesn't have the background in it?

My Advice

Machine Learning is a very broad and deep field to study, which can take many years to learn. However I would advice you to start small. Given some data, can you find some insights from this data that are interesting, actionable, or simply support a story that you want to narrate? You can just use pen and paper, or Excel sheets to start with. I would suggest to use tools readily available online, and build working prototype -- a prototype that is maybe ugly but solves the problem quickly. Then keep digging deeper into how it works underneath.

Further, can you create a system, that periodically finds such insights, from such data that is updated over time? Interestingly you don't necessarily have to always build complex ML models to help end-users.

tuxdna

# find the number of pages in pdf file
qpdf --show-npages file_name.pdf

# split into multiple pages using first-to-last range, where first page is 1 and last page is 4
pdftoppm -jpeg -f 1 -l 4 some-file.pdf p

# combine into single pdf
convert p-1.jpg p-2.jpg p-3.jpg p-4.jpg some-file-downsized.pdf

tuxdna

import io.reactivex.rxjava3.core.Flowable;
import io.reactivex.rxjava3.core.Maybe;
import io.reactivex.rxjava3.core.Observable;
import io.reactivex.rxjava3.core.Scheduler;
import io.reactivex.rxjava3.parallel.ParallelFlowable;
import io.reactivex.rxjava3.schedulers.Schedulers;
import org.apache.commons.lang3.tuple.ImmutablePair;

import java.time.LocalTime;
import java.util.concurrent.ExecutorService;

tuxdna

How to play audio on multiple devices on your Ubuntu machine?

List sources

$ pacmd list-sources |grep name:
	name: <bluez_sink.00_00_00_00_D2_89.a2dp_sink.monitor>
	name: <alsa_output.pci-0000_00_1f.3.analog-stereo.monitor>

tuxdna

# -*- coding: utf-8 -*-

u"""
Beta regression for modeling rates and proportions.

References
----------
Grün, Bettina, Ioannis Kosmidis, and Achim Zeileis. Extended beta regression
in R: Shaken, stirred, mixed, and partitioned. No. 2011-22. Working Papers in
Economics and Statistics, 2011.

tuxdna

https://twitter.com/goodfellow_ian/status/1127994332416364549?s=19

tuxdna

Python Data:

Dimension: 2 x 768

[array([-7.3584e-01, -1.9531e-03, -6.8970e-03, -6.0303e-01, -2.1008e-01,
        -4.2114e-03,  2.0935e-02,  3.6157e-01, -1.7712e-01, -3.5449e-01,
        -4.4629e-01, -1.0791e-01,  2.7563e-01,  1.0791e-01, -3.0664e-01,
        -6.1676e-02, -2.9468e-01,  2.1594e-01, -1.3232e-01,  1.3171e-01,
        -4.1772e-01,  9.2346e-02, -4.5239e-01, -1.3147e-01,  2.9175e-01,