Skip to content

Instantly share code, notes, and snippets.

@harisec
Created October 15, 2019 06:03
Show Gist options
  • Save harisec/bff0ac7ad2e90da7e82297d945c63ef1 to your computer and use it in GitHub Desktop.
Save harisec/bff0ac7ad2e90da7e82297d945c63ef1 to your computer and use it in GitHub Desktop.
filter links
An idea for a tool.
So, going through Wayback archive we get thousands of links like the ones below.
It's the same path and different values for GET parameters.
A very useful tool would filter these using this algo:
- for each link, parse the link with an URL parser:
- create a hash from: hostname + path + (parameter names (without values) sorted).
- save the hash in an array
- for the next links, compute the hash
- if the hash is in the array, ignore the link
- if not pass it through
The links:
http://answers.yahoo.com/question/?qid=20110416092517AAbELYv
http://answers.yahoo.com/question/?qid=20110420134930AAu4umN
http://answers.yahoo.com/question/?qid=20110421192556AAhgq4D
http://answers.yahoo.com/question/?qid=20110423171909AAg40TX
http://answers.yahoo.com/question/?qid=20110427120319AAIOx5N
http://answers.yahoo.com/question/?qid=20110427151942AAtZbxo
http://answers.yahoo.com/question/?qid=20110429151912AAlrIHT
http://answers.yahoo.com/question/?qid=20110510150840AAf6A4r
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420805&b=0&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420809&b=1020&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420809&b=1330&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420812&b=1330&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420815&b=1541&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420817&b=1751&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420820&b=1550&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420825&b=1291&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420828&b=40&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420831&b=540&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420833&b=430&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420834&b=1700&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420836&b=1211&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420841&b=1180&keyword=&byto=title
From these links only 2 should pass:
http://answers.yahoo.com/question/?qid=20110416092517AAbELYv
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420805&b=0&keyword=&byto=title
@smackerdodi
Copy link

i made it with easier way by renaming every parameter value to "FUZZ" with sed then sort unique the output gives me non repeatable URLs which i could inject any payload on it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment