Vikram VikramTiwari

## including_external_package_in_dataflow.md

      
              1 file
            
          
              1 fork
            
          
              6 comments
            
          
              26 stars
            
          
                inchoate
                / including_external_package_in_dataflow.md
            
            
              Last active
              February 2, 2024 11:40
            
              
                Adding an extra package to a Python Dataflow project to run on GCP
              
          
    The Problem

The documentation for how to deploy a pipeline with extra, non-PyPi, pure Python packages on GCP is missing some detail. This gist shows how to package and deploy an external pure-Python, non-PyPi dependency to a managed dataflow pipeline on GCP.
TL;DR: You external package needs to be a python (source/binary) distro properly packaged and shipped alongside your pipeline. It is not enough to only specify a tar file with a setup.py.
Preparing the External Package

Your external package must have a proper setup.py. What follow is an example setup.py for our ETL package. This is used to package version 1.1.1 of the etl library. The library requires 3 native PyPi packages to run. These are specified in the install_requires field. This package also ships with custom external JSON data, declared in the package_data section. Last, the setuptools.find_packages function searches for all available packages and returns that

  
## how-to-publish-to-npm.md

      
              1 file
            
          
              210 forks
            
          
              75 comments
            
          
              1114 stars
            
          
                coolaj86
                / how-to-publish-to-npm.md
            
            
              Last active
              June 9, 2024 23:19
            
              
                How to publish packages to NPM
              
          
    Getting Started with NPM (as a developer)

As easy as 1, 2, 3!
Updated:

Aug, 08, 2022 update config docs for npm 8+
Jul 27, 2021 add private scopes
Jul 22, 2021 add dist tags
Jun 20, 2021 update for --access=public
Sep 07, 2020 update docs for npm version