Skip to content

Instantly share code, notes, and snippets.

@ilovefood2
ilovefood2 / data_loading_utils.py
Created September 14, 2019 18:09 — forked from iyvinjose/data_loading_utils.py
Read large files line by line without loading entire file to memory. Supports files of GB size
def read_lines_from_file_as_data_chunks(file_name, chunk_size, callback, return_whole_chunk=False):
"""
read file line by line regardless of its size
:param file_name: absolute path of file to read
:param chunk_size: size of data to be read at at time
:param callback: callback method, prototype ----> def callback(data, eof, file_name)
:return:
"""
def read_in_chunks(file_obj, chunk_size=5000):
@tombigel
tombigel / README.md
Last active April 24, 2024 12:32 — forked from a2ikm/limit.maxfiles.plist
How to Change Open Files Limit on OS X and macOS Sierra (10.8 - 10.12)

How to Change Open Files Limit on OS X and macOS

This text is the section about OS X Yosemite (which also works for macOS Sierra) from https://docs.basho.com/riak/kv/2.1.4/using/performance/open-files-limit/#mac-os-x

The last time i visited this link it was dead (403), so I cloned it here from the latest snapshot in Archive.org's Wayback Machine https://web.archive.org/web/20170523131633/https://docs.basho.com/riak/kv/2.1.4/using/performance/open-files-limit/

Mac OS X

To check the current limits on your Mac OS X system, run:

@stackia
stackia / homebrew.mxcl.aria2.plist
Last active October 13, 2023 04:50
aria2c daemon start at boot. Put this file into /Library/LaunchDaemons, them 'chmod 600' and 'chown root'. Remember to fill placeholders below.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>KeepAlive</key>
<true/>
<key>RunAtLoad</key>
<true/>
<key>Label</key>
<string>homebrew.mxcl.aria2</string>