Skip to content

Instantly share code, notes, and snippets.

@jussker
jussker / copilot-cralwer-coding.md
Last active September 18, 2024 12:44
github copilot python coding instructions.md

For Python 3.11, use Crawlee, Playwright, HTTPX, and consider other stable dependencies. \nUse Google style, Ruff, Black. Short names for complex vars, clear names for simple. \nComprehensive types: \n- Params, returns \n- Vars in funcs \n- Generics (e.g., List, Dict[str, Any]) \nAsync for network operations. Use functional paradigms where appropriate. \nCode must be complete, runnable, correct. \n \nScraper design: \n1. Use cases (e.g., data extraction, site mapping) \n2. Types/roles (e.g., Crawler, Parser, DataStorage) \n3. Attributes, methods per type/role \n4. Type/role relations \n5. Access control \n6. Comment rationale \n7. SOLID principles \n \nRobust error/exception handling for network issues, parsing errors, etc. \nSecure against injection, XSS, CSRF. Validate and sanitize input and scraped data. \n \nUse Crawlee for robust crawling infrastructure. \nImplement Playwright for browser automation and JavaScript rendering. \nUtilize HTTPX for additional HTTP requests if needed. \nConfigure Chromium for

@jussker
jussker / demo_glm_batch.py
Last active September 12, 2024 04:01
ChatGLM Batch API CLI 例子
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Python Version: python 3.11.6
# @Time : 2024/09/11 23:40
# @Author : Jin
"""说明:
这个脚本是一个使用GLM Batch API的例子. 你可以使用这个脚本上传文件, 并且下载结果.
官网参考: https://bigmodel.cn/dev/howuse/batchapi
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jussker
jussker / GPT4VImageTokenCalculator.py
Created December 15, 2023 06:28
ChatGPT4V: Counting tokens for image
import math
from typing import Dict, Tuple
class GPT4VImageTokenCalculator:
def __init__(self, low_resolution: bool = False) -> None:
self.low_resolution = low_resolution
self.base_token_cost = 85
self.additional_token_cost = 170
def calculate_resize_dimensions(self, width: int, height: int) -> Tuple[int, int, int, int]: