Created
March 20, 2021 10:30
-
-
Save r-rmcgibbo/a2042606246cbaacd904019a34df8a7c to your computer and use it in GitHub Desktop.
system: aarch64-linux | build_time: 2 seconds | https://github.com/NixOS/nixpkgs/pull/117016
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sourcing python-remove-tests-dir-hook | |
Sourcing python-catch-conflicts-hook.sh | |
Sourcing python-remove-bin-bytecode-hook.sh | |
Sourcing setuptools-build-hook | |
Using setuptoolsBuildPhase | |
Using setuptoolsShellHook | |
Sourcing pip-install-hook | |
Using pipInstallPhase | |
Sourcing python-imports-check-hook.sh | |
Using pythonImportsCheckPhase | |
Sourcing python-namespaces-hook | |
Sourcing setuptools-check-hook | |
@nix { "action": "setPhase", "phase": "unpackPhase" } | |
unpacking sources | |
unpacking source archive /nix/store/5kwg4kl38ayw512yivc9ar36azdqgzv2-source | |
source root is source | |
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/online-tests | |
@nix { "action": "setPhase", "phase": "patchPhase" } | |
patching sources | |
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" } | |
updateAutotoolsGnuConfigScriptsPhase | |
@nix { "action": "setPhase", "phase": "configurePhase" } | |
configuring | |
no configure script, doing nothing | |
@nix { "action": "setPhase", "phase": "buildPhase" } | |
building | |
Executing setuptoolsBuildPhase | |
running bdist_wheel | |
running build | |
running build_py | |
creating build | |
creating build/lib | |
creating build/lib/libgrabsite | |
copying libgrabsite/main.py -> build/lib/libgrabsite | |
copying libgrabsite/dump_urls.py -> build/lib/libgrabsite | |
copying libgrabsite/wpull_hooks.py -> build/lib/libgrabsite | |
copying libgrabsite/__init__.py -> build/lib/libgrabsite | |
copying libgrabsite/dashboard_client.py -> build/lib/libgrabsite | |
copying libgrabsite/wpull_tweaks.py -> build/lib/libgrabsite | |
copying libgrabsite/dupes.py -> build/lib/libgrabsite | |
copying libgrabsite/server.py -> build/lib/libgrabsite | |
copying libgrabsite/dupespotter.py -> build/lib/libgrabsite | |
copying libgrabsite/dashboard.html -> build/lib/libgrabsite | |
copying libgrabsite/404.html -> build/lib/libgrabsite | |
copying libgrabsite/favicon.ico -> build/lib/libgrabsite | |
copying libgrabsite/default_cookies.txt -> build/lib/libgrabsite | |
creating build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/global -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/reddit -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/pinterest -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/singletumblr -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/meetupeverywhere -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/nosortedindex -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/youtube -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/coppermine -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/facebook -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/blogs -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/noonion -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/twitter -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/mediawiki -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/imdb -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/forums -> build/lib/libgrabsite/ignore_sets | |
copying libgrabsite/ignore_sets/nogravatar -> build/lib/libgrabsite/ignore_sets | |
running build_scripts | |
creating build/scripts-3.7 | |
copying and adjusting grab-site -> build/scripts-3.7 | |
copying and adjusting gs-server -> build/scripts-3.7 | |
copying and adjusting gs-dump-urls -> build/scripts-3.7 | |
changing mode of build/scripts-3.7/grab-site from 644 to 755 | |
changing mode of build/scripts-3.7/gs-server from 644 to 755 | |
changing mode of build/scripts-3.7/gs-dump-urls from 644 to 755 | |
installing to build/bdist.linux-aarch64/wheel | |
running install | |
running install_lib | |
creating build/bdist.linux-aarch64 | |
creating build/bdist.linux-aarch64/wheel | |
creating build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/main.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/dump_urls.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/dashboard.html -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/wpull_hooks.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/404.html -> build/bdist.linux-aarch64/wheel/libgrabsite | |
creating build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/global -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/reddit -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/pinterest -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/singletumblr -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/meetupeverywhere -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/nosortedindex -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/youtube -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/coppermine -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/facebook -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/blogs -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/noonion -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/twitter -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/mediawiki -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/imdb -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/forums -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/ignore_sets/nogravatar -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets | |
copying build/lib/libgrabsite/__init__.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/dashboard_client.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/default_cookies.txt -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/wpull_tweaks.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/dupes.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/favicon.ico -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/server.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
copying build/lib/libgrabsite/dupespotter.py -> build/bdist.linux-aarch64/wheel/libgrabsite | |
running install_egg_info | |
running egg_info | |
creating grab_site.egg-info | |
writing grab_site.egg-info/PKG-INFO | |
writing dependency_links to grab_site.egg-info/dependency_links.txt | |
writing requirements to grab_site.egg-info/requires.txt | |
writing top-level names to grab_site.egg-info/top_level.txt | |
writing manifest file 'grab_site.egg-info/SOURCES.txt' | |
reading manifest file 'grab_site.egg-info/SOURCES.txt' | |
writing manifest file 'grab_site.egg-info/SOURCES.txt' | |
Copying grab_site.egg-info to build/bdist.linux-aarch64/wheel/grab_site-2.2.0-py3.7.egg-info | |
running install_scripts | |
creating build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data | |
creating build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts | |
copying build/scripts-3.7/gs-dump-urls -> build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts | |
copying build/scripts-3.7/grab-site -> build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts | |
copying build/scripts-3.7/gs-server -> build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts | |
changing mode of build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts/gs-dump-urls to 755 | |
changing mode of build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts/grab-site to 755 | |
changing mode of build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts/gs-server to 755 | |
adding license file "LICENSE" (matched pattern "LICEN[CS]E*") | |
creating build/bdist.linux-aarch64/wheel/grab_site-2.2.0.dist-info/WHEEL | |
creating 'dist/grab_site-2.2.0-py3-none-any.whl' and adding 'build/bdist.linux-aarch64/wheel' to it | |
adding 'grab_site-2.2.0.data/scripts/grab-site' | |
adding 'grab_site-2.2.0.data/scripts/gs-dump-urls' | |
adding 'grab_site-2.2.0.data/scripts/gs-server' | |
adding 'libgrabsite/404.html' | |
adding 'libgrabsite/__init__.py' | |
adding 'libgrabsite/dashboard.html' | |
adding 'libgrabsite/dashboard_client.py' | |
adding 'libgrabsite/default_cookies.txt' | |
adding 'libgrabsite/dump_urls.py' | |
adding 'libgrabsite/dupes.py' | |
adding 'libgrabsite/dupespotter.py' | |
adding 'libgrabsite/favicon.ico' | |
adding 'libgrabsite/main.py' | |
adding 'libgrabsite/server.py' | |
adding 'libgrabsite/wpull_hooks.py' | |
adding 'libgrabsite/wpull_tweaks.py' | |
adding 'libgrabsite/ignore_sets/blogs' | |
adding 'libgrabsite/ignore_sets/coppermine' | |
adding 'libgrabsite/ignore_sets/facebook' | |
adding 'libgrabsite/ignore_sets/forums' | |
adding 'libgrabsite/ignore_sets/global' | |
adding 'libgrabsite/ignore_sets/imdb' | |
adding 'libgrabsite/ignore_sets/mediawiki' | |
adding 'libgrabsite/ignore_sets/meetupeverywhere' | |
adding 'libgrabsite/ignore_sets/nogravatar' | |
adding 'libgrabsite/ignore_sets/noonion' | |
adding 'libgrabsite/ignore_sets/nosortedindex' | |
adding 'libgrabsite/ignore_sets/pinterest' | |
adding 'libgrabsite/ignore_sets/reddit' | |
adding 'libgrabsite/ignore_sets/singletumblr' | |
adding 'libgrabsite/ignore_sets/twitter' | |
adding 'libgrabsite/ignore_sets/youtube' | |
adding 'grab_site-2.2.0.dist-info/LICENSE' | |
adding 'grab_site-2.2.0.dist-info/METADATA' | |
adding 'grab_site-2.2.0.dist-info/WHEEL' | |
adding 'grab_site-2.2.0.dist-info/top_level.txt' | |
adding 'grab_site-2.2.0.dist-info/RECORD' | |
removing build/bdist.linux-aarch64/wheel | |
Finished executing setuptoolsBuildPhase | |
@nix { "action": "setPhase", "phase": "installPhase" } | |
installing | |
Executing pipInstallPhase | |
/build/source/dist /build/source | |
Processing ./grab_site-2.2.0-py3-none-any.whl | |
Requirement already satisfied: lmdb>=0.89 in /nix/store/ikzz3007ad3vbxjcn2g47rqv1xds6dmy-python3.7-lmdb-1.0.0/lib/python3.7/site-packages (from grab-site==2.2.0) (1.0.0) | |
Requirement already satisfied: fb-re2>=1.0.6 in /nix/store/w6krbb3zh6zkda2vgqphyh0aqmsrhrdz-python3.7-fb-re2-1.0.7/lib/python3.7/site-packages (from grab-site==2.2.0) (1.0.7) | |
Requirement already satisfied: websockets>=6.0 in /nix/store/1imfbbw0sk0ch86645dz1b8bbrpgl66l-python3.7-websockets-8.1/lib/python3.7/site-packages (from grab-site==2.2.0) (8.1) | |
Requirement already satisfied: autobahn>=0.12.1 in /nix/store/04p47nvwr9ijf21zh6havcs7y8d8jlh8-python3.7-autobahn-20.12.3/lib/python3.7/site-packages (from grab-site==2.2.0) (20.12.3) | |
Requirement already satisfied: manhole>=1.0.0 in /nix/store/cb704f0fwipkb5hxf6dn0ac63pq8ip6j-python3.7-manhole-1.6.0/lib/python3.7/site-packages (from grab-site==2.2.0) (1.6.0) | |
Requirement already satisfied: wpull in /nix/store/pij6va1ifvdg6f2ajzhm8xcnhpdl43r0-python3.7-ludios_wpull-3.0.7/lib/python3.7/site-packages (from grab-site==2.2.0) (3.0.7) | |
Requirement already satisfied: cchardet>=1.0.0 in /nix/store/2xn1v7rs8dyy6i8vws1diawnzbyw6gps-python3.7-cchardet-2.1.7/lib/python3.7/site-packages (from grab-site==2.2.0) (2.1.7) | |
Requirement already satisfied: click>=6.3 in /nix/store/v940s2qyplcyw9lclgh57162hvz314q0-python3.7-click-7.1.2/lib/python3.7/site-packages (from grab-site==2.2.0) (7.1.2) | |
Requirement already satisfied: txaio>=20.4.1 in /nix/store/h1qi617ijl69lc5lip9z1dxp2p9y7pm8-python3.7-txaio-20.4.1/lib/python3.7/site-packages (from autobahn>=0.12.1->grab-site==2.2.0) (20.4.1) | |
Requirement already satisfied: hyperlink>=20.0.1 in /nix/store/hla8xwg8zfaw6aqg1njsjgpvc4nn9sl1-python3.7-hyperlink-20.0.1/lib/python3.7/site-packages (from autobahn>=0.12.1->grab-site==2.2.0) (20.0.1) | |
Requirement already satisfied: cryptography>=2.9.2 in /nix/store/m1wggcxvnrlc20662r4ydagc9zk5ssj3-python3.7-cryptography-3.4.6/lib/python3.7/site-packages (from autobahn>=0.12.1->grab-site==2.2.0) (3.4.6) | |
Requirement already satisfied: cffi>=1.12 in /nix/store/34yr37a5f82l4wkzm5xmdpb8qpqr05fb-python3.7-cffi-1.14.5/lib/python3.7/site-packages (from cryptography>=2.9.2->autobahn>=0.12.1->grab-site==2.2.0) (1.14.5) | |
Requirement already satisfied: pycparser in /nix/store/8l293cm4bky3y2096qnbcig81xh74sgc-python3.7-pycparser-2.20/lib/python3.7/site-packages (from cffi>=1.12->cryptography>=2.9.2->autobahn>=0.12.1->grab-site==2.2.0) (2.20) | |
Requirement already satisfied: idna>=2.5 in /nix/store/bhdm0z7m8g4n6mr0zr8g2rpy5s5v0rnr-python3.7-idna-2.10/lib/python3.7/site-packages (from hyperlink>=20.0.1->autobahn>=0.12.1->grab-site==2.2.0) (2.10) | |
Requirement already satisfied: yapsy in /nix/store/p8x18ks0lz5zfsi4ia7yd3vk46cvf8mk-python3.7-Yapsy-1.12.2/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (1.12.2) | |
Requirement already satisfied: html5-parser in /nix/store/swd3vw0xk61x6yzbx46kqcjavzf3f10m-python3.7-html5-parser-0.4.9/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (0.4.9) | |
Requirement already satisfied: tornado==4.5.3 in /nix/store/d5jaq9igr46wqrp0s0jlhx0zrpilnqxv-python3.7-tornado-4.5.3/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (4.5.3) | |
Requirement already satisfied: dnspython in /nix/store/p9nwam8kcvvml24nc4fswm7hm9vmzl1x-python3.7-dnspython-2.0.0/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (2.0.0) | |
Requirement already satisfied: lxml in /nix/store/i0lkkmh4dzwjk6y33xax0gqkgkx46cdm-python3.7-lxml-4.6.2/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (4.6.2) | |
Requirement already satisfied: namedlist in /nix/store/ayfrb8a0fgcwyb8c639cayszyaga7k2b-python3.7-namedlist-1.8/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (1.8) | |
Requirement already satisfied: chardet in /nix/store/7hmb6d7chdq6kazsv39cvkp7q120vq2x-python3.7-chardet-3.0.4/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (3.0.4) | |
Requirement already satisfied: sqlalchemy in /nix/store/2f14irjdmksbkwmcafgxr0ffp38058i3-python3.7-SQLAlchemy-1.3.23/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (1.3.23) | |
Installing collected packages: grab-site | |
Successfully installed grab-site-2.2.0 | |
/build/source | |
Finished executing pipInstallPhase | |
@nix { "action": "setPhase", "phase": "fixupPhase" } | |
post-installation fixup | |
shrinking RPATHs of ELF executables and libraries in /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0 | |
strip is /nix/store/h5wgppbyv8vkla58v8zi535j5i9akly5-binutils-2.35.1/bin/strip | |
stripping (with command strip and flags -S) in /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/lib /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/bin | |
patching script interpreter paths in /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0 | |
checking for references to /build/ in /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0... | |
Rewriting #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10/bin/python3.7 to #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10 | |
wrapping `/nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/bin/gs-dump-urls'... | |
Rewriting #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10/bin/python3.7 to #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10 | |
wrapping `/nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/bin/grab-site'... | |
Rewriting #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10/bin/python3.7 to #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10 | |
wrapping `/nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/bin/gs-server'... | |
Executing pythonRemoveTestsDir | |
Finished executing pythonRemoveTestsDir | |
@nix { "action": "setPhase", "phase": "installCheckPhase" } | |
running install tests | |
grab-site --help | |
Usage: grab-site [OPTIONS] [START_URL]... | |
Runs a crawl on one or more URLs. For | |
additional help, see | |
https://github.com/ArchiveTeam/grab- | |
site/blob/master/README.md#usage | |
Options: | |
--concurrency NUM Use this many | |
connections to | |
fetch in | |
parallel | |
(default: 2). | |
--concurrent NUM Alias for | |
--concurrency. | |
--delay DELAY Time to wait | |
between | |
requests, in | |
milliseconds | |
(default: 0). | |
Can be "NUM", or | |
"MIN-MAX" to use | |
a random delay | |
between MIN and | |
MAX for each | |
request. Delay | |
applies to each | |
concurrent | |
fetcher, not | |
globally. | |
--recursive / --1 --recursive | |
(default: true) | |
to crawl under | |
last /path/ | |
component | |
recursively, or | |
--1 to get just | |
START_URL. | |
--offsite-links / --no-offsite-links | |
--offsite-links | |
(default: true) | |
to grab all | |
links to a depth | |
of 1 on other | |
domains, or | |
--no-offsite- | |
links to | |
disable. | |
--igsets LIST Comma-separated | |
list of ignore | |
sets to use in | |
addition to | |
"global". | |
--ignore-sets LIST Alias for | |
--igsets. | |
--import-ignores FILE Copy this file | |
to DIR/ignores | |
before the crawl | |
begins. | |
--igon / --igoff --igon (default: | |
false) to print | |
all URLs being | |
ignored to the | |
terminal and | |
dashboard. | |
--debug Print a lot of | |
debugging | |
information. | |
--video / --no-video --no-video | |
(default: false) | |
to skip the | |
download of | |
videos by both | |
mime type and | |
file extension. | |
Skipped videos | |
are logged to DI | |
R/skipped_videos | |
-i, --input-file TEXT Load list of | |
URLs-to-grab | |
from a local | |
file or from a | |
URL; like wget | |
-i. File must be | |
a newline- | |
delimited list | |
of URLs. Combine | |
with --1 to | |
avoid a | |
recursive crawl | |
on each URL. | |
--max-content-length N Skip the | |
download of any | |
response that | |
claims a | |
Content-Length | |
larger than N | |
(default: -1, | |
don't skip | |
anything). | |
--level NUM Recurse this | |
many levels | |
(default: inf). | |
--page-requisites-level NUM Recursive this | |
many levels for | |
page requisites | |
(default: 5). | |
--warc-max-size BYTES Try to limit | |
each WARC file | |
to around BYTES | |
bytes before | |
rolling over to | |
a new WARC file | |
(default: | |
5368709120, | |
which is 5GiB). | |
--ua STRING Send User-Agent: | |
STRING instead | |
of pretending to | |
be Firefox on | |
Windows. | |
--wpull-args ARGS String | |
containing | |
additional | |
arguments to | |
pass to wpull; | |
see ~/.local/bin | |
/wpull --help. | |
ARGS is split | |
with shlex.split | |
and individual | |
arguments can | |
contain spaces | |
if quoted, e.g. | |
--wpull-args="-- | |
youtube-dl \"-- | |
youtube-dl- | |
exe=/My Document | |
s/youtube-dl\"" | |
--sitemaps / --no-sitemaps --sitemaps | |
(default: true) | |
to queue URLs | |
from sitemap.xml | |
at the root of | |
the site, or | |
--no-sitemaps to | |
disable. | |
--dupespotter / --no-dupespotter | |
--dupespotter | |
(default: true) | |
to skip the | |
extraction of | |
links from pages | |
that look like | |
duplicates of | |
earlier pages, | |
or --no- | |
dupespotter to | |
disable. | |
Disable this for | |
sites that are | |
directory | |
listings. | |
--id ID Use id ID for | |
the crawl | |
instead of a | |
random 128-bit | |
id. This must be | |
unique for every | |
crawl. | |
--dir DIR Put control | |
files, temporary | |
files, and | |
unfinished WARCs | |
in DIR (default: | |
a directory name | |
based on the | |
URL, date, and | |
first 8 | |
characters of | |
the id). | |
--finished-warc-dir FINISHED_WARC_DIR | |
Absolute path to | |
a directory into | |
which finished | |
.warc.gz and | |
.cdx files will | |
be moved. | |
--permanent-error-status-codes STATUS_CODES | |
A comma- | |
separated list | |
of HTTP status | |
codes to treat | |
as a permanent | |
error and | |
therefore *not* | |
retry (default: 4 | |
01,403,404,405,4 | |
10) | |
--which-wpull-args-partial Print a partial | |
list of wpull | |
arguments that | |
would be used | |
and exit. | |
Excludes grab- | |
site-specific | |
features, and | |
removes DIR/ | |
from paths. | |
Useful for | |
reporting bugs | |
on wpull without | |
grab-site | |
involvement. | |
--which-wpull-command Populate DIR/ | |
but don't start | |
wpull; instead | |
print the | |
command that | |
would have been | |
used to start | |
wpull with all | |
of the grab-site | |
functionality. | |
--version Print version | |
and exit. | |
--help Show this | |
message and | |
exit. | |
grab-site --version | |
2.2.0 | |
gs-dump-urls --help | |
Usage: gs-dump-urls [OPTIONS] WPULL_DB_FILE [done| | |
error|in_progress|skipped|todo | |
] | |
Dumps URLs of a particular crawl status from a | |
wpull.db file. | |
WPULL_DB_FILE is the path to the wpull.db | |
file. | |
STATUS is one of "done", "error", | |
"in_progress", "skipped", or "todo". | |
Options: | |
--version Print version and exit. | |
--help Show this message and exit. | |
python -c 'import libgrabsite.server' | |
@nix { "action": "setPhase", "phase": "pythonCatchConflictsPhase" } | |
pythonCatchConflictsPhase | |
@nix { "action": "setPhase", "phase": "pythonRemoveBinBytecodePhase" } | |
pythonRemoveBinBytecodePhase | |
@nix { "action": "setPhase", "phase": "pythonImportsCheckPhase" } | |
pythonImportsCheckPhase | |
Executing pythonImportsCheckPhase |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment