Skip to content

Instantly share code, notes, and snippets.

@CharlesNepote
CharlesNepote / csv2datasette
Last active March 17, 2024 17:02
csv2datasette
#!/usr/bin/env bash
# sudo ln -s "$(pwd)/csv2datasette" /usr/bin/csv2datasette
# csv2datasette is meant to explore CSV data. It is not meant to create a sustainable DB.
# csv2datasette is a bash script which open CSV files directly in Datasette. It offers
# a number of options for reading and exploring CSV files, such as --stats, inspired by WTFCsv.
#
# `--stats` option includes, for each column: the column name, the number of unique values,
# the number of filled rows, the number of missing values, the mininmum value, the maximum value,
# the average, the sum, the shortest string, the longest string, the number of numeric values,
(function() {
'use strict';
console.log("DS: start");
// Part I. -- Add "CSV without limit" link
// document.getElementsByClassName("export-links")?
if (document.getElementsByClassName('show-hide-sql')[0] && document.getElementById('sql-editor') !== null) {
displayLinkCSVWithoutLimit();
}
@CharlesNepote
CharlesNepote / count.js
Last active December 21, 2022 11:35
Datasette: button to count rows
// Add "count" button aside the "Custom SQL query returning XX rows" title
// Clicking on the button counts and displays total number of rows
// Probably does not play well with complex queries (eg. sub-queries)
// Test if we're located in a query's page
const hide = document.getElementsByClassName('show-hide-sql')[0] || false;
if (hide) {
var sql_request = document.getElementById('sql-editor').value;
// Remove request's comments
# get total requests by status code
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
# get top requesters by IP
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head | awk -v OFS='\t' '{"host " $2 | getline ip; print $0, ip}'
# get top requesters by user agent
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
# get top requests by URL
CREATE TABLE IF NOT EXISTS "all" (
[code] TEXT,
[url] TEXT,
[creator] TEXT,
[created_t] INTEGER,
[created_datetime] TEXT,
[last_modified_t] INTEGER,
[last_modified_datetime] TEXT,
[product_name] TEXT,
[abbreviated_product_name] TEXT,
@CharlesNepote
CharlesNepote / export.txt
Last active April 14, 2022 17:54
Open Food Facts import
code
creator
created_t
last_modified_t
product_name
abbreviated_product_name
generic_name
quantity
packaging
packaging_text
@CharlesNepote
CharlesNepote / standard.sh
Last active February 16, 2022 17:57 — forked from hfossli/standard.sh
Standard bash script format
#!/bin/bash
CLEAR='\033[0m'
RED='\033[0;31m'
function usage() {
if [ -n "$1" ]; then
echo -e "${RED}👉 $1${CLEAR}\n";
fi
echo "Usage: $0 [-n number-of-people] [-s section-id] [-c cache-file]"
#!/bin/bash
# Install useful tools
# TODO: compléter
sudo apt install curl git htop iftop lynx mc recode -y
# TODO: csvkit
# TODO: docker
# TODO: VSCODE
@CharlesNepote
CharlesNepote / checkbot.py
Last active October 11, 2021 14:01
Checkbot
import sys # to allow inputs from command line
import simplejson as json # to read json data; simplejson is easier to read nested data
import urllib.request # to make http request
# What's the issue?
# 1. A lot of data quality issues are not solved or owned.
# 2. The current lists are not efficient because they are mostly unknown by the community and we have to look at it to see what have to be done.
# The goal of this tool is to push quality errors to some volonteers.
@CharlesNepote
CharlesNepote / feus.user.js
Last active May 17, 2021 07:16
FEUS - Folksonomy Engine user script
// ==UserScript==
// @name Folksonomy Engine user script
// @description Add Folksonomy Engine UI to Open Food Facts web pages.
// @namespace openfoodfacts.org
// @version 2021-05-10T13:18
// @include https://*.openfoodfacts.org/*
// @include https://*.openproductsfacts.org/*
// @include https://*.openbeautyfacts.org/*
// @include https://*.openpetfoodfacts.org/*
// @include https://*.pro.openfoodfacts.org/*