Skip to content

Instantly share code, notes, and snippets.

@zhaokuohaha
zhaokuohaha / Robots协议和robots.txt解读.md
Last active June 15, 2023 05:38
robots.txt基本语法, 以及robots协议的各家搜索引擎实现。

文件简介

简单的说 Robot 协议是用于告诉网络爬虫(主要是搜索引擎爬虫)本域名下的网页中, 那些部分是不应该爬取的,具体来说就是在网站的域名根目录下添加一份robots.txt文件, 并在文件中声明对应的规则. 由于 Url 对大小写敏感, 所以 robots.txt 文件名要求全部小写.

搜索引擎爬虫在爬取一个网站时, 首先检查是否存在 robots.txt 文件, 如果存在, 则在爬取该网站下其他网页时, 应遵循文件所定义的范围进行爬取.

需要注意的是 Robots 协议是一个共同遵循的"道德约定", 但是不具有法律效应, 同时也不保证所有搜索引擎都会遵守这个约定

内容和语法

@sketchpunk
sketchpunk / b64_to_float32Array.js
Created September 1, 2017 03:08
Base64 String to Float32Array in Javascript, good for WebGL
var blob = window.atob("AAAAAAAAAAAAAAAAAACAPwAAAAAAAAAAAAAAAAAAgD8AAAAA"), // Base64 string converted to a char array
fLen = blob.length / Float32Array.BYTES_PER_ELEMENT, // How many floats can be made, but be even
dView = new DataView( new ArrayBuffer(Float32Array.BYTES_PER_ELEMENT) ), // ArrayBuffer/DataView to convert 4 bytes into 1 float.
fAry = new Float32Array(fLen), // Final Output at the correct size
p = 0; // Position
for(var j=0; j < fLen; j++){
p = j * 4;
dView.setUint8(0,blob.charCodeAt(p));
dView.setUint8(1,blob.charCodeAt(p+1));
@jwilson8767
jwilson8767 / es6-element-ready.js
Last active April 22, 2024 20:28
Wait for an element to exist. ES6, Promise, MutationObserver
// MIT Licensed
// Author: jwilson8767
/**
* Waits for an element satisfying selector to exist, then resolves promise with the element.
* Useful for resolving race conditions.
*
* @param selector
* @returns {Promise}
*/
@FranklinYu
FranklinYu / README.markdown
Last active April 27, 2024 05:46
links for old versions of Docker for Mac (inspired by docker/for-mac#1120)

links for old versions of Docker for Mac

Deprecated

Docker provides download links in release note. They promised that

(we) will also include download links in release notes for future releases.

Note:

@ntuaha
ntuaha / getDateString.js
Last active September 20, 2021 21:36
Javascript 漂亮日期顯示 YYYYMMDDhhmmss or YYYY-MM-DD hh:mm:ss
'use strict'
function pad(v){
return (v<10)?'0'+v:v
}
function getDateString(d){
var year = d.getFullYear();
var month = pad(d.getMonth()+1);
var day = pad(d.getDate());
@kottenator
kottenator / simple-pagination.js
Created July 13, 2015 20:44
Simple pagination algorithm
// Implementation in ES6
function pagination(c, m) {
var current = c,
last = m,
delta = 2,
left = current - delta,
right = current + delta + 1,
range = [],
rangeWithDots = [],
l;
@huqi
huqi / trial.key
Created April 11, 2015 09:01
Beyond Compare 4 license for Mac
Beyond Compare 4
Licensed to: ASIO Allsoftinone
Quantity: 1 user
Serial number: 1822-9597
License type: Pro Edition for Windows
--- BEGIN LICENSE KEY ---
H1bJTd2SauPv5Garuaq0Ig43uqq5NJOEw94wxdZTpU-pFB9GmyPk677gJ
vC1Ro6sbAvKR4pVwtxdCfuoZDb6hJ5bVQKqlfihJfSYZt-xVrVU27+0Ja
hFbqTmYskatMTgPyjvv99CF2Te8ec+Ys2SPxyZAF0YwOCNOWmsyqN5y9t
@Ivlyth
Ivlyth / js-date-format.js
Last active March 28, 2024 10:54
format javascript date to format "YYYY-mm-dd HH:MM:SS"
var d = new Date();
d = new Date(d.getTime() - 3000000);
var date_format_str = d.getFullYear().toString()+"-"+((d.getMonth()+1).toString().length==2?(d.getMonth()+1).toString():"0"+(d.getMonth()+1).toString())+"-"+(d.getDate().toString().length==2?d.getDate().toString():"0"+d.getDate().toString())+" "+(d.getHours().toString().length==2?d.getHours().toString():"0"+d.getHours().toString())+":"+((parseInt(d.getMinutes()/5)*5).toString().length==2?(parseInt(d.getMinutes()/5)*5).toString():"0"+(parseInt(d.getMinutes()/5)*5).toString())+":00";
console.log(date_format_str);
//2015-03-31 13:35:00
@kiinlam
kiinlam / regex_example.txt
Last active July 26, 2022 13:04
正则匹配
正则匹配
匹配中文字符的正则表达式: [\u4e00-\u9fa5]
评注:匹配中文还真是个头疼的事,有了这个表达式就好办了
匹配双字节字符(包括汉字在内):[^\x00-\xff]
评注:可以用来计算字符串的长度(一个双字节字符长度计2,ASCII字符计1)
匹配空白行的正则表达式:\n\s*\r
评注:可以用来删除空白行
@Swaagie
Swaagie / visibly.js
Last active December 21, 2015 00:39 — forked from addyosmani/visibly.js
Updated visibly against firefox native API
/*!
* isVis - v0.5.5 Aug 2011 - Page Visibility API Polyfill
* Copyright (c) 2011 Addy Osmani
* Dual licensed under the MIT and GPL licenses.
*/
(function () {
window.visibly = {
b: null,
q: document,