alocus

"""
5-crawl_recursive_internal_links.py

Original file
https://github.com/coleam00/ottomator-agents/blob/main/crawl4AI-agent-v2/crawl4AI-examples/5-crawl_site_recursively.py

Modify 
Save MD content under "crawl_output" folder based on the URL structure as nested folders.
----------------------------------
Recursively crawls a site starting from a root URL, using Crawl4AI's arun_many and a memory-adaptive dispatcher.

alocus

import sys
import os
import json
from copy import deepcopy
import pandas

# Stackoverflow 
# https://stackoverflow.com/questions/41180960/convert-nested-json-to-csv-file-in-python/63500540#63500540
# Usage:
# 	python j2c.py input.json

alocus

How I set up bitbucket and github on the same computer

I always create new repo on github and bitbucket online and clone it locally.

• Follow github and bitbucket ssh setup procedure (except the fact saving the file with different name id_rsa_XXXXX) ++ Generate ssh key on machine and add them to the account

Mac

Set up ssh config for separate repo hosts

~/.ssh/config

alocus

#!/bin/bash

fn="$1"
of=`echo $1 | sed s/mp4/jpg/`

lf=`ffprobe -show_streams "$fn" 2> /dev/null | grep nb_frames | head -1 | cut -d \= -f 2`
rm -f "$of"
let "lf = $lf - 1"
ffmpeg -i $fn -vf select=\'eq\(n,$lf\) -vframes 1 $of