This small subclass of the Pandas sqlalchemy-based SQL support for reading/storing tables uses the Postgres-specific "COPY FROM" method to insert large amounts of data to the database. It is much faster that using INSERT. To acheive this, the table is created in the normal way using sqlalchemy but no data is inserted. Instead the data is saved to a temporary CSV file (using Pandas' mature CSV support) then read back to Postgres using Psychopg2 support for COPY FROM STDIN.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# One of the fastest ways to insert bulk data into Postgres (at least, aside from COPY) is using the psycopg2 extras function execute_values. | |
# However, this doesn't return an accurate row count value - instead, it just returns the row count for the last page inserted. | |
# This wraps the execute_values function with its own pagination to return an accurate count of rows inserted. | |
# Performance is approximately equivalent to underlying execute_values function - within 5-10% or so in my brief tests. | |
import psycopg2 | |
import psycopg2.extras | |
import math | |
db_connection_string = "dbname=EDITME host=EDITME" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## FOR UBUNTU | |
Dependencies install | |
1. apt-get install nginx-extras | |
2. apt-get install lua-zlib | |
lua file decompress request body | |
-------------------------------- | |
see https://gist.github.com/iammehrabalam/30f5402bbcdad139c9eafd3a6f47ce6c |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Please set the ROOT to the folder your nxlog was installed into, | |
## otherwise it will not start. | |
#define ROOT C:\Program Files\nxlog | |
define ROOT C:\Program Files (x86)\nxlog | |
define CERTDIR %ROOT%\cert | |
Moduledir %ROOT%\modules | |
CacheDir %ROOT%\data | |
Pidfile %ROOT%\data\nxlog.pid |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# ------Instructions--------- | |
# Install (and configure) subfinder, assetfinder, and httprobe | |
# go get -v github.com/projectdiscovery/subfinder/cmd/subfinder && go get -v github.com/tomnomnom/httprobe && go get -v github.com/tomnomnom/assetfinder | |
# cat firefox.sh >> ~/.bashrc | |
# source ~/.bashrc | |
# Usage - subf_ff target.tld | |
# asset_ff target.tld | |
subf_ff () { | |
subfinder -d $1 -silent -t 100 | httprobe -c 50 | sort -u | while read line; do firefox $line; sleep 10; done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/2 | |
/graphql-proxy/admin | |
/3.0/ | |
/3ds_callback | |
/3ds_update_payment_callback | |
/accounts | |
/active | |
/activity | |
/actuator | |
/actuator/auditevents |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Moved to https://github.com/lanmaster53/pyscripter-er/tree/master/snippets |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#make directories | |
mkdir /git | |
mkdir /db | |
mkdir /db/modules | |
mkdir /downloads | |
#update packages, install gcc and redis | |
add-apt-repository ppa:chris-lea/redis-server -y && \ | |
apt-get update && \ | |
apt-get upgrade -y && \ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from java.awt import Font | |
from javax.swing import JScrollPane, JTextPane | |
from javax.swing.text import SimpleAttributeSet | |
from burp import IBurpExtender, IExtensionStateListener, IHttpListener, ITab | |
import base64 | |
import traceback | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from lxml import html | |
import unicodecsv as csv | |
import requests | |
from time import sleep | |
import re | |
import argparse | |
import json | |
def parse(url): |
OlderNewer