Here is a list of tools I find worth a try if you are going to set up a new ML project. This list is not intended to be exhaustive overview and it does not include any ML frameworks or libraries. It is focused on auxiliary tools that can make development easier and experiments reproducible. Some of this tools I have used in real projects, others I just tried on a toy example, but found interesting to use in future.
cookiecutter - a scaffold generator for all sorts of projects. It allows generating boilerplate code for an empty project. Useful for the unification of project directories. It also creates basic configurations for packaging and testing (may also include configured code coverage tools).
- But I highly recommend NOT to use https://github.com/drivendata/cookiecutter-data-science cause it contains a whole bunch of controversial python practices like using
src
as a module - It's better to use https://github.com/audreyr/cookiecutter-pypackage and, but borrow the directory structure from the
cookiecutter-data-science
.
MLFlow - is a tool for keeping track of performed experiments and their parameters. Has own web-view for accessing experiment data. Require a running server to store experiment runs.
DVC - a tool that can be thought of as git
for large or binary files.
It does not try to calculate a diff between versions and relies on third-party file storage (for example - S3). In essence, it consists of 2 parts - the data versioning part and the data pipeline part. The first one is definitely a must-have thing. It could be very useful for keeping track of your trained models near the code.
But the second part - DVC Pipeline, looks redundant for me and it is better to replace it with good old Makefile.
- Makefile - a very old tool. You define target files and steps on how to obtain them.
- Luigi or Airflow - distributed versions of the Makefile. Allows building a Directed Acyclic Graphs of tasks in python. They also have a web-view and built-in scheduler.
- kedro - yet another data pipeline framework which also provides it's own project generation tool. It may be interesting because it provides a high level of abstraction. Kedo-pipelines is abstract enough to be converted into Airflow-pipeline. However they heavily depend on
Dataset
abstraction, so it might be tricky to implement lazy data loaders. - metaflow - a server-less variation of Airflow, it allows to build pipelines and DAGs, but does not have a scheduler or server-side. Instead, it can launch pipelines in the cloud.
- Flask-RESTPlus + gunicorn - slightly more sophisticated setup then a just simple Flask. RESTPlus restricts you with better API development practices than pure Flask does. It also allows to easily document your API with Swagger.
- FastAPI - even more compact version of Flask + Swagger which builds API documentation automatically.
- Docker + Kubernetes - There are tons of training materials and reviews on the Internet. Little advice - don't try to deploy your own K8s control plane.
- cortex - allows you to quickly deploy your models as a service on AWS. It automatically wraps your
predict
function into API endpoint and instantiates EC2 instances with for serving it. But in my opinion, this framework is only suitable for prototypes, as the solution is not transferable and no containers are created.