Skip to content

Instantly share code, notes, and snippets.

@r-rmcgibbo

r-rmcgibbo/grab-site

Created March 20, 2021 10:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save r-rmcgibbo/a2042606246cbaacd904019a34df8a7c to your computer and use it in GitHub Desktop.
Save r-rmcgibbo/a2042606246cbaacd904019a34df8a7c to your computer and use it in GitHub Desktop.
Download ZIP
system: aarch64-linux | build_time: 2 seconds | https://github.com/NixOS/nixpkgs/pull/117016
Raw
grab-site
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing setuptools-check-hook
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/5kwg4kl38ayw512yivc9ar36azdqgzv2-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/online-tests
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing setuptoolsBuildPhase
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/libgrabsite
copying libgrabsite/main.py -> build/lib/libgrabsite
copying libgrabsite/dump_urls.py -> build/lib/libgrabsite
copying libgrabsite/wpull_hooks.py -> build/lib/libgrabsite
copying libgrabsite/__init__.py -> build/lib/libgrabsite
copying libgrabsite/dashboard_client.py -> build/lib/libgrabsite
copying libgrabsite/wpull_tweaks.py -> build/lib/libgrabsite
copying libgrabsite/dupes.py -> build/lib/libgrabsite
copying libgrabsite/server.py -> build/lib/libgrabsite
copying libgrabsite/dupespotter.py -> build/lib/libgrabsite
copying libgrabsite/dashboard.html -> build/lib/libgrabsite
copying libgrabsite/404.html -> build/lib/libgrabsite
copying libgrabsite/favicon.ico -> build/lib/libgrabsite
copying libgrabsite/default_cookies.txt -> build/lib/libgrabsite
creating build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/global -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/reddit -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/pinterest -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/singletumblr -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/meetupeverywhere -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/nosortedindex -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/youtube -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/coppermine -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/facebook -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/blogs -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/noonion -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/twitter -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/mediawiki -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/imdb -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/forums -> build/lib/libgrabsite/ignore_sets
copying libgrabsite/ignore_sets/nogravatar -> build/lib/libgrabsite/ignore_sets
running build_scripts
creating build/scripts-3.7
copying and adjusting grab-site -> build/scripts-3.7
copying and adjusting gs-server -> build/scripts-3.7
copying and adjusting gs-dump-urls -> build/scripts-3.7
changing mode of build/scripts-3.7/grab-site from 644 to 755
changing mode of build/scripts-3.7/gs-server from 644 to 755
changing mode of build/scripts-3.7/gs-dump-urls from 644 to 755
installing to build/bdist.linux-aarch64/wheel
running install
running install_lib
creating build/bdist.linux-aarch64
creating build/bdist.linux-aarch64/wheel
creating build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/main.py -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/dump_urls.py -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/dashboard.html -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/wpull_hooks.py -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/404.html -> build/bdist.linux-aarch64/wheel/libgrabsite
creating build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/global -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/reddit -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/pinterest -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/singletumblr -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/meetupeverywhere -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/nosortedindex -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/youtube -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/coppermine -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/facebook -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/blogs -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/noonion -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/twitter -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/mediawiki -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/imdb -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/forums -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/ignore_sets/nogravatar -> build/bdist.linux-aarch64/wheel/libgrabsite/ignore_sets
copying build/lib/libgrabsite/__init__.py -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/dashboard_client.py -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/default_cookies.txt -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/wpull_tweaks.py -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/dupes.py -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/favicon.ico -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/server.py -> build/bdist.linux-aarch64/wheel/libgrabsite
copying build/lib/libgrabsite/dupespotter.py -> build/bdist.linux-aarch64/wheel/libgrabsite
running install_egg_info
running egg_info
creating grab_site.egg-info
writing grab_site.egg-info/PKG-INFO
writing dependency_links to grab_site.egg-info/dependency_links.txt
writing requirements to grab_site.egg-info/requires.txt
writing top-level names to grab_site.egg-info/top_level.txt
writing manifest file 'grab_site.egg-info/SOURCES.txt'
reading manifest file 'grab_site.egg-info/SOURCES.txt'
writing manifest file 'grab_site.egg-info/SOURCES.txt'
Copying grab_site.egg-info to build/bdist.linux-aarch64/wheel/grab_site-2.2.0-py3.7.egg-info
running install_scripts
creating build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data
creating build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts
copying build/scripts-3.7/gs-dump-urls -> build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts
copying build/scripts-3.7/grab-site -> build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts
copying build/scripts-3.7/gs-server -> build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts
changing mode of build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts/gs-dump-urls to 755
changing mode of build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts/grab-site to 755
changing mode of build/bdist.linux-aarch64/wheel/grab_site-2.2.0.data/scripts/gs-server to 755
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
creating build/bdist.linux-aarch64/wheel/grab_site-2.2.0.dist-info/WHEEL
creating 'dist/grab_site-2.2.0-py3-none-any.whl' and adding 'build/bdist.linux-aarch64/wheel' to it
adding 'grab_site-2.2.0.data/scripts/grab-site'
adding 'grab_site-2.2.0.data/scripts/gs-dump-urls'
adding 'grab_site-2.2.0.data/scripts/gs-server'
adding 'libgrabsite/404.html'
adding 'libgrabsite/__init__.py'
adding 'libgrabsite/dashboard.html'
adding 'libgrabsite/dashboard_client.py'
adding 'libgrabsite/default_cookies.txt'
adding 'libgrabsite/dump_urls.py'
adding 'libgrabsite/dupes.py'
adding 'libgrabsite/dupespotter.py'
adding 'libgrabsite/favicon.ico'
adding 'libgrabsite/main.py'
adding 'libgrabsite/server.py'
adding 'libgrabsite/wpull_hooks.py'
adding 'libgrabsite/wpull_tweaks.py'
adding 'libgrabsite/ignore_sets/blogs'
adding 'libgrabsite/ignore_sets/coppermine'
adding 'libgrabsite/ignore_sets/facebook'
adding 'libgrabsite/ignore_sets/forums'
adding 'libgrabsite/ignore_sets/global'
adding 'libgrabsite/ignore_sets/imdb'
adding 'libgrabsite/ignore_sets/mediawiki'
adding 'libgrabsite/ignore_sets/meetupeverywhere'
adding 'libgrabsite/ignore_sets/nogravatar'
adding 'libgrabsite/ignore_sets/noonion'
adding 'libgrabsite/ignore_sets/nosortedindex'
adding 'libgrabsite/ignore_sets/pinterest'
adding 'libgrabsite/ignore_sets/reddit'
adding 'libgrabsite/ignore_sets/singletumblr'
adding 'libgrabsite/ignore_sets/twitter'
adding 'libgrabsite/ignore_sets/youtube'
adding 'grab_site-2.2.0.dist-info/LICENSE'
adding 'grab_site-2.2.0.dist-info/METADATA'
adding 'grab_site-2.2.0.dist-info/WHEEL'
adding 'grab_site-2.2.0.dist-info/top_level.txt'
adding 'grab_site-2.2.0.dist-info/RECORD'
removing build/bdist.linux-aarch64/wheel
Finished executing setuptoolsBuildPhase
@nix { "action": "setPhase", "phase": "installPhase" }
installing
Executing pipInstallPhase
/build/source/dist /build/source
Processing ./grab_site-2.2.0-py3-none-any.whl
Requirement already satisfied: lmdb>=0.89 in /nix/store/ikzz3007ad3vbxjcn2g47rqv1xds6dmy-python3.7-lmdb-1.0.0/lib/python3.7/site-packages (from grab-site==2.2.0) (1.0.0)
Requirement already satisfied: fb-re2>=1.0.6 in /nix/store/w6krbb3zh6zkda2vgqphyh0aqmsrhrdz-python3.7-fb-re2-1.0.7/lib/python3.7/site-packages (from grab-site==2.2.0) (1.0.7)
Requirement already satisfied: websockets>=6.0 in /nix/store/1imfbbw0sk0ch86645dz1b8bbrpgl66l-python3.7-websockets-8.1/lib/python3.7/site-packages (from grab-site==2.2.0) (8.1)
Requirement already satisfied: autobahn>=0.12.1 in /nix/store/04p47nvwr9ijf21zh6havcs7y8d8jlh8-python3.7-autobahn-20.12.3/lib/python3.7/site-packages (from grab-site==2.2.0) (20.12.3)
Requirement already satisfied: manhole>=1.0.0 in /nix/store/cb704f0fwipkb5hxf6dn0ac63pq8ip6j-python3.7-manhole-1.6.0/lib/python3.7/site-packages (from grab-site==2.2.0) (1.6.0)
Requirement already satisfied: wpull in /nix/store/pij6va1ifvdg6f2ajzhm8xcnhpdl43r0-python3.7-ludios_wpull-3.0.7/lib/python3.7/site-packages (from grab-site==2.2.0) (3.0.7)
Requirement already satisfied: cchardet>=1.0.0 in /nix/store/2xn1v7rs8dyy6i8vws1diawnzbyw6gps-python3.7-cchardet-2.1.7/lib/python3.7/site-packages (from grab-site==2.2.0) (2.1.7)
Requirement already satisfied: click>=6.3 in /nix/store/v940s2qyplcyw9lclgh57162hvz314q0-python3.7-click-7.1.2/lib/python3.7/site-packages (from grab-site==2.2.0) (7.1.2)
Requirement already satisfied: txaio>=20.4.1 in /nix/store/h1qi617ijl69lc5lip9z1dxp2p9y7pm8-python3.7-txaio-20.4.1/lib/python3.7/site-packages (from autobahn>=0.12.1->grab-site==2.2.0) (20.4.1)
Requirement already satisfied: hyperlink>=20.0.1 in /nix/store/hla8xwg8zfaw6aqg1njsjgpvc4nn9sl1-python3.7-hyperlink-20.0.1/lib/python3.7/site-packages (from autobahn>=0.12.1->grab-site==2.2.0) (20.0.1)
Requirement already satisfied: cryptography>=2.9.2 in /nix/store/m1wggcxvnrlc20662r4ydagc9zk5ssj3-python3.7-cryptography-3.4.6/lib/python3.7/site-packages (from autobahn>=0.12.1->grab-site==2.2.0) (3.4.6)
Requirement already satisfied: cffi>=1.12 in /nix/store/34yr37a5f82l4wkzm5xmdpb8qpqr05fb-python3.7-cffi-1.14.5/lib/python3.7/site-packages (from cryptography>=2.9.2->autobahn>=0.12.1->grab-site==2.2.0) (1.14.5)
Requirement already satisfied: pycparser in /nix/store/8l293cm4bky3y2096qnbcig81xh74sgc-python3.7-pycparser-2.20/lib/python3.7/site-packages (from cffi>=1.12->cryptography>=2.9.2->autobahn>=0.12.1->grab-site==2.2.0) (2.20)
Requirement already satisfied: idna>=2.5 in /nix/store/bhdm0z7m8g4n6mr0zr8g2rpy5s5v0rnr-python3.7-idna-2.10/lib/python3.7/site-packages (from hyperlink>=20.0.1->autobahn>=0.12.1->grab-site==2.2.0) (2.10)
Requirement already satisfied: yapsy in /nix/store/p8x18ks0lz5zfsi4ia7yd3vk46cvf8mk-python3.7-Yapsy-1.12.2/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (1.12.2)
Requirement already satisfied: html5-parser in /nix/store/swd3vw0xk61x6yzbx46kqcjavzf3f10m-python3.7-html5-parser-0.4.9/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (0.4.9)
Requirement already satisfied: tornado==4.5.3 in /nix/store/d5jaq9igr46wqrp0s0jlhx0zrpilnqxv-python3.7-tornado-4.5.3/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (4.5.3)
Requirement already satisfied: dnspython in /nix/store/p9nwam8kcvvml24nc4fswm7hm9vmzl1x-python3.7-dnspython-2.0.0/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (2.0.0)
Requirement already satisfied: lxml in /nix/store/i0lkkmh4dzwjk6y33xax0gqkgkx46cdm-python3.7-lxml-4.6.2/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (4.6.2)
Requirement already satisfied: namedlist in /nix/store/ayfrb8a0fgcwyb8c639cayszyaga7k2b-python3.7-namedlist-1.8/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (1.8)
Requirement already satisfied: chardet in /nix/store/7hmb6d7chdq6kazsv39cvkp7q120vq2x-python3.7-chardet-3.0.4/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (3.0.4)
Requirement already satisfied: sqlalchemy in /nix/store/2f14irjdmksbkwmcafgxr0ffp38058i3-python3.7-SQLAlchemy-1.3.23/lib/python3.7/site-packages (from wpull->grab-site==2.2.0) (1.3.23)
Installing collected packages: grab-site
Successfully installed grab-site-2.2.0
/build/source
Finished executing pipInstallPhase
@nix { "action": "setPhase", "phase": "fixupPhase" }
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0
strip is /nix/store/h5wgppbyv8vkla58v8zi535j5i9akly5-binutils-2.35.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/lib /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/bin
patching script interpreter paths in /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0
checking for references to /build/ in /nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0...
Rewriting #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10/bin/python3.7 to #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10
wrapping `/nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/bin/gs-dump-urls'...
Rewriting #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10/bin/python3.7 to #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10
wrapping `/nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/bin/grab-site'...
Rewriting #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10/bin/python3.7 to #!/nix/store/3r74dnylfb9m6k22w4mxlb2m9drwyvm6-python3-3.7.10
wrapping `/nix/store/p7grwa1p61iz41x894iiqbshna5y29bi-grab-site-2.2.0/bin/gs-server'...
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
@nix { "action": "setPhase", "phase": "installCheckPhase" }
running install tests
grab-site --help
Usage: grab-site [OPTIONS] [START_URL]...
Runs a crawl on one or more URLs. For
additional help, see
https://github.com/ArchiveTeam/grab-
site/blob/master/README.md#usage
Options:
--concurrency NUM Use this many
connections to
fetch in
parallel
(default: 2).
--concurrent NUM Alias for
--concurrency.
--delay DELAY Time to wait
between
requests, in
milliseconds
(default: 0).
Can be "NUM", or
"MIN-MAX" to use
a random delay
between MIN and
MAX for each
request. Delay
applies to each
concurrent
fetcher, not
globally.
--recursive / --1 --recursive
(default: true)
to crawl under
last /path/
component
recursively, or
--1 to get just
START_URL.
--offsite-links / --no-offsite-links
--offsite-links
(default: true)
to grab all
links to a depth
of 1 on other
domains, or
--no-offsite-
links to
disable.
--igsets LIST Comma-separated
list of ignore
sets to use in
addition to
"global".
--ignore-sets LIST Alias for
--igsets.
--import-ignores FILE Copy this file
to DIR/ignores
before the crawl
begins.
--igon / --igoff --igon (default:
false) to print
all URLs being
ignored to the
terminal and
dashboard.
--debug Print a lot of
debugging
information.
--video / --no-video --no-video
(default: false)
to skip the
download of
videos by both
mime type and
file extension.
Skipped videos
are logged to DI
R/skipped_videos
-i, --input-file TEXT Load list of
URLs-to-grab
from a local
file or from a
URL; like wget
-i. File must be
a newline-
delimited list
of URLs. Combine
with --1 to
avoid a
recursive crawl
on each URL.
--max-content-length N Skip the
download of any
response that
claims a
Content-Length
larger than N
(default: -1,
don't skip
anything).
--level NUM Recurse this
many levels
(default: inf).
--page-requisites-level NUM Recursive this
many levels for
page requisites
(default: 5).
--warc-max-size BYTES Try to limit
each WARC file
to around BYTES
bytes before
rolling over to
a new WARC file
(default:
5368709120,
which is 5GiB).
--ua STRING Send User-Agent:
STRING instead
of pretending to
be Firefox on
Windows.
--wpull-args ARGS String
containing
additional
arguments to
pass to wpull;
see ~/.local/bin
/wpull --help.
ARGS is split
with shlex.split
and individual
arguments can
contain spaces
if quoted, e.g.
--wpull-args="--
youtube-dl \"--
youtube-dl-
exe=/My Document
s/youtube-dl\""
--sitemaps / --no-sitemaps --sitemaps
(default: true)
to queue URLs
from sitemap.xml
at the root of
the site, or
--no-sitemaps to
disable.
--dupespotter / --no-dupespotter
--dupespotter
(default: true)
to skip the
extraction of
links from pages
that look like
duplicates of
earlier pages,
or --no-
dupespotter to
disable.
Disable this for
sites that are
directory
listings.
--id ID Use id ID for
the crawl
instead of a
random 128-bit
id. This must be
unique for every
crawl.
--dir DIR Put control
files, temporary
files, and
unfinished WARCs
in DIR (default:
a directory name
based on the
URL, date, and
first 8
characters of
the id).
--finished-warc-dir FINISHED_WARC_DIR
Absolute path to
a directory into
which finished
.warc.gz and
.cdx files will
be moved.
--permanent-error-status-codes STATUS_CODES
A comma-
separated list
of HTTP status
codes to treat
as a permanent
error and
therefore *not*
retry (default: 4
01,403,404,405,4
10)
--which-wpull-args-partial Print a partial
list of wpull
arguments that
would be used
and exit.
Excludes grab-
site-specific
features, and
removes DIR/
from paths.
Useful for
reporting bugs
on wpull without
grab-site
involvement.
--which-wpull-command Populate DIR/
but don't start
wpull; instead
print the
command that
would have been
used to start
wpull with all
of the grab-site
functionality.
--version Print version
and exit.
--help Show this
message and
exit.
grab-site --version
2.2.0
gs-dump-urls --help
Usage: gs-dump-urls [OPTIONS] WPULL_DB_FILE [done|
error|in_progress|skipped|todo
]
Dumps URLs of a particular crawl status from a
wpull.db file.
WPULL_DB_FILE is the path to the wpull.db
file.
STATUS is one of "done", "error",
"in_progress", "skipped", or "todo".
Options:
--version Print version and exit.
--help Show this message and exit.
python -c 'import libgrabsite.server'
@nix { "action": "setPhase", "phase": "pythonCatchConflictsPhase" }
pythonCatchConflictsPhase
@nix { "action": "setPhase", "phase": "pythonRemoveBinBytecodePhase" }
pythonRemoveBinBytecodePhase
@nix { "action": "setPhase", "phase": "pythonImportsCheckPhase" }
pythonImportsCheckPhase
Executing pythonImportsCheckPhase
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment