Skip to content

Instantly share code, notes, and snippets.

@vanga
vanga / nfhs5-descriptions.json
Created September 25, 2023 16:32
NFHS5 variable descriptions
{
"RECH0": {
"HHID": "Case identification uniquely identifies each household. In most surveys, this is constructed by concatenating the cluster or sample point number and the household number, but in some surveys this may be the questionnaire number taken from the front page of the questionnaire.",
"HV000": "Alphabetic country code to identify the survey from which the data were collected. The code is based on an international standard code. This variable is 3 characters in length, with the third character indicating the format of the recode file used for this survey. For all surveys in DHS VII following this standard, this code will be 7. For example: DR7 is the Dominican Republic, HT7 is Haiti and KH7 is Cambodia.",
"HV001": "Cluster number is the number identifying the sample point as used during the fieldwork. This variable may be a composite of several variables in the questionnaire. If so, the non- standard variables are included in RECH3 as country-specific variables.",
@vanga
vanga / nfhs4-descriptions.json
Last active September 25, 2023 16:33
NFHS4 variable descriptions
{
"RECH0": {
"HHID": "Case identification uniquely identifies each household. In most surveys this is constructed by concatenating the cluster or sample point number and the household number, but in some surveys this may be the questionnaire number taken from the front page of the questionnaire.",
"HV000": "Alphabetic country code to identify the survey from which the data were collected. The code is based on an international standard code. This variable is 3 characters in length, with the third character indicating the format of the recode file used for this survey. For all surveys in DHS VI following this standard, this code will be 6. For example: DR6 is the Dominican Republic, HT6 is Haiti and KH6 is Cambodia.",
"HV001": "Cluster number is the number identifying the sample point as used during the fieldwork. This variable may be a composite of several variables in the questionnaire. If so, the non- standard variables are included in RECH3 as country-specific variables.",
@vanga
vanga / nco-occupation-codes.csv
Last active May 4, 2023 14:05
2 digit NCO 2004 Occupation codes CSV/Excel
nco_code label
11 LEGISLATORS AND SENIOR OFFICIALS
12 CORPORATE MANAGERS
13 GENERAL MANAGERS
21 PHYSICAL, MATHEMATICAL AND ENGINEERING SCIENCE PROFESSIONALS
22 LIFE SCIENCE AND HEALTH PROFESSIONALS
23 TEACHING PROFESSIONALS
24 OTHER PROFESSIONALS
31 PHYSICAL AND ENGINEERING SCIENCE ASSOCIATE PROFESSIONALS.
32 LIFE SCIENCE AND HEALTH ASSOCIATE PROFESSIONALS
@vanga
vanga / nic-industry-codes.csv
Created May 4, 2023 10:00
NIC 2008 Industry codes
nic_2008_code label
01 Crop and animal production, hunting and related service activities
02 Forestry and logging
03 Fishing and aquaculture
05 Mining of coal and lignite
06 Extraction of crude petroleum and natural gas
07 Mining of metal ores
08 Other mining and quarrying
09 Mining support service activities
10 Manufacture of food products
@vanga
vanga / se-data-processor.py
Last active September 23, 2022 13:08
For processing SE data dumps
# Convert xml to json/parquet and stores them in TEMP_DIR
# Column names are modified to remove "@"" at the beginning for convenience
# Transforms tags column of posts into a list
# Uses pandas joins and groupby features do one-one and one-many joins
import xmltodict
import json
import simplejson
import os
{
"Id": 1,
"PostTypeId": 1,
"AcceptedAnswerId": 52,
"CreationDate": "2017-02-28T16:47:57.887",
"Score": 19,
"ViewCount": 407,
"Body": "<p><a href=\"https://jenkins.io/\" rel=\"nofollow noreferrer\">Jenkins</a> is a fork of <a href=\"http://hudson-ci.org/\" rel=\"nofollow noreferrer\">Hudson</a>.</p>\n\n<p>What are the benefits/advantages of using Jenkins over Hudson for a small private business?</p>\n",
"OwnerUserId": 3,
"LastEditorUserId": 210,
@vanga
vanga / flask-server.py
Created September 8, 2022 15:01
Sample flask application with Elastic APM integration
from flask import Flask
from elasticapm.contrib.flask import ElasticAPM
from flask import request
app = Flask(__name__)
app.config['ELASTIC_APM'] = {
'SERVICE_NAME': 'flask',
'SERVER_URL': 'https://<apm-server-ip/dns>.com:8200',
'SERVER_CERT': 'path to elk.example.com/elk.example.com.crt',
'SERVER_TIMEOUT': '5s',
@vanga
vanga / apm-server.yml
Created September 8, 2022 14:56
APM server configuration with auth and SSL certs
apm-server:
host: "0.0.0.0:8200"
auth:
secret_token: 'secret token generated using openssl or any other string'
ssl:
enabled: true
certificate: '/usr/share/apm-server/certs/elk.example.com/elk.example.com.crt'
key: '/usr/share/apm-server/certs/elk.example.com/elk.example.com.key'
output.elasticsearch:
@vanga
vanga / kibana-ssl-config.yml
Created September 8, 2022 14:47
SSL configuration for Kibana
server.ssl.enabled: true
server.ssl.certificate: '/home/ubuntu/certs/elk.example.com/elk.example.com.crt'
server.ssl.key: '/home/ubuntu/certs/elk.example.com/elk.example.com.key'
@vanga
vanga / create-self-signed-certs.sh
Created September 8, 2022 14:41
Create self signed certificates using Elasticsearch util scripts
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.4.1-linux-x86_64.tar.gz
tar -zxf elasticsearch-8.4.1-linux-x86_64.tar.gz
cd elasticsearch-8.4.1/
./bin/elasticsearch-certutil ca --pem
unzip elastic-stack-ca.zip
./bin/elasticsearch-certutil cert --ca-cert ca/ca.crt --ca-key ca/ca.key --pem --name elk.example.com
unzip certificate-bundle.zip
cd ../