Skip to content

Instantly share code, notes, and snippets.

@ma7dev
Last active September 5, 2022 13:23
Show Gist options
  • Star 6 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ma7dev/677682ec3b7a8e61b1954c845bf2f07a to your computer and use it in GitHub Desktop.
Save ma7dev/677682ec3b7a8e61b1954c845bf2f07a to your computer and use it in GitHub Desktop.
takes requirements.txt file without module versions and annotates it with the latest set of module versions that won't result in build/runtime errors.
#!/bin/bash
# TODO: you will need to have conda installed
# create a python environment
conda create -n env_tmp python=3.8.13 -y
# activate environment
conda activate env_tmp
# download a sample for requirements.txt
wget https://raw.githubusercontent.com/delip/PyTorchNLPBook/master/requirements.txt
# install poetry
pip install poetry
# download a simple template for pyproject.toml (just so poetry can install the packages)
wget https://gist.githubusercontent.com/sudomaze/7298ffc4409032edd4d18a57b4c38f3a/raw/1c32efcbde31aaf896c6d47b32dac19ed44d14a4/pyproject.toml
# install dependencies from requirements.txt
cat requirements.txt | xargs poetry add
# export to requirements-like format
poetry export -f requirements.txt --output long_requirements.txt --without-hashes
# remove unwanted python versioning
cat long_requirements.txt | cut -d ";" -f 1 > with_dep_requirements.txt
# remove unwated dependencies
cat requirements.txt | while read line
do echo $(grep -n $line'==' with_dep_requirements.txt | cut -d ":" -f 2 ) >> final_requirements.txt
done
# output
cat final_requirements.txt
tqdm==4.64.0
matplotlib==3.5.3
nltk==3.7
jupyter==1.0.0
pandas==1.4.3
seaborn==0.11.2
scipy==1.9.0
annoy==1.17.1
numpy==1.23.2
requests==2.28.1
tqdm
matplotlib
nltk
jupyter
pandas
seaborn
scipy
annoy
numpy
requests
@ma7dev
Copy link
Author

ma7dev commented Aug 27, 2022

I developed a package out of this script: https://github.com/sudomaze/pyreqpp

It is currently working for the explicit calling of the package: python -m pyreqpp, where requirements.txt is at the same level as your root directory.

Integrating with pre-commit and publishing it is Work-In-Progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment