Skip to content

Instantly share code, notes, and snippets.

View imfht's full-sized avatar
🎯
Focusing

imfht imfht

🎯
Focusing
View GitHub Profile
package main
import (
"bytes"
"fmt"
"net"
"runtime"
"sync"
"sync/atomic"
"time"
@imfht
imfht / httpdns.go
Created September 4, 2018 05:10 — forked from fangdingjun/httpdns.go
use dnspod's httpdns service to query dns
package main
import (
"bytes"
"fmt"
"github.com/miekg/dns"
"io/ioutil"
"log"
"net/http"
)
@imfht
imfht / gist:af5970a3799b6064964fd5294499ec06
Created August 19, 2018 12:27 — forked from evilpacket/gist:3628941
Top 1000 from Alexa Top 1million
wget -q http://s3.amazonaws.com/alexa-static/top-1m.csv.zip;unzip top-1m.csv.zip; awk -F ',' '{print $2}' top-1m.csv|head -1000 > top-1000.txt; rm top-1m.csv*
@imfht
imfht / setup.md
Created June 30, 2018 04:07 — forked from xrstf/setup.md
Nutch 2.3 + ElasticSearch 1.4 + HBase 0.94 Setup

Info

This guide sets up a non-clustered Nutch crawler, which stores its data via HBase. We will not learn how to setup Hadoop et al., but just the bare minimum to crawl and index websites on a single machine.

Terms

  • Nutch - the crawler (fetches and parses websites)
  • HBase - filesystem storage for Nutch (Hadoop component, basically)
@imfht
imfht / ssurt.py
Created November 5, 2017 05:38 — forked from ato/ssurt.py
#!/usr/bin/python3
# coding=utf-8
import re
SSURT_RE = r"""
\A
(?P<scheme> [a-zA-Z] [a-zA-Z0-9+.-]* : )?
(?P<authority>
(?P<slashes> /* )
@imfht
imfht / dnsmasq.conf
Created August 29, 2017 06:34 — forked from aa65535/dnsmasq.conf
dnsmasq config
# 并发查询所有上游DNS服务器
all-servers
# 本地DNS缓存最小有效期
min-cache-ttl=3600
# 本地DNS缓存数目
cache-size=65535
# 上游DNS服务器, 可设置多个
server=8.8.4.4
#server=8.8.8.8
server=114.114.114.114
@imfht
imfht / gist:341468d410155b5be35c66acbb103d7f
Created July 27, 2017 09:08 — forked from hest/gist:8798884
Fast SQLAlchemy counting (avoid query.count() subquery)
def get_count(q):
count_q = q.statement.with_only_columns([func.count()]).order_by(None)
count = q.session.execute(count_q).scalar()
return count
q = session.query(TestModel).filter(...).order_by(...)
# Slow: SELECT COUNT(*) FROM (SELECT ... FROM TestModel WHERE ...) ...
print q.count()
@imfht
imfht / captcha.py
Created July 3, 2017 11:30 — forked from liberize/captcha.py
简单验证码识别
#!/usr/bin/env python
# coding: utf-8
import os
import sys
from PIL import Image, ImageChops
templates = []
@imfht
imfht / fofa_rule.sql
Created May 30, 2017 09:11 — forked from Tr3jer/fofa_rule.sql
fofa_rule.sql
/*
Navicat Premium Data Transfer
Source Server : localhost
Source Server Type : MySQL
Source Server Version : 50542
Source Host : localhost
Source Database : rule
Target Server Type : MySQL