Our README process for versioning files under DVC - is this right?

README.md

Models and their Files

In addition to storing and versioning data under the data/ directory, we store and version models in the models/ directory and version them using DVC. On PySpark we store models on the nodes of the cluster at /tmp/models/ so you will need to copy them there and use spark-specific paths when using PySpark.

In order to add a file to DVC, you need to run this series of commands. Failure to run them all can result in the deletion of files recently added by your co-workers [:(]. It is very important that you git pull and dvc pull, one after the other, before you add files and dvc push. You can add new files to DVC this way or re-add them to update the version.

# Get the latest .dvc files and configuration for the project
git pull origin <branch>

# Pull the latest datasets from DVC before adding our new ones
dvc pull

# Version a new file under DVC - "dvc add" is analagous to "git add"
dvc add models/pig_classifier/logistic_regression.pkl

# This creates the file models/pig_classifier/logistic_regression.pkl.dvc and pushes it to our branch
git add -f models/pig_classifier/logistic_regression.pkl.dvc
git commit -m "DVC versioned the latest logistic regression model for classifying happy and sad pigs"

# Commit the file to DVC
dvc commit

# Push the files to git
git push origin <branch>

# Push the files to DVC
dvc push

In this specific example I think we don't need dvc pull and dvc commit. Otherwise it should work fine!

@shcheklein I’m confused. The docs say that dvc add is like git add. Does adding the file push it to our GCS bucket?

hey, @rjurney , sorry for the delay.

when our docs saying "dvc something is like git something", unfortunately it's never precise. And I see that it can be misleading. Our intention was to give some idea about the command, and it felt that git add was the best to explain.

dvc add indeed is similar to git add. It adds file into Git, but if go further we can say that dvc add is git add file + git commit file. So, after dvc add you don't need to do dvc commit. In this case I would even say that dvc commit's name similarity to git commit is very misleading.

dvc commit in DVC is not meant to be used often. It forcefully updates all .dvc and dvc.lock files. That's why if you don't have data pulled with dvc pull initially it will update all previous .dvc files as if they were pointing to empty data (or it actually should even error out?)

Does adding the file push it to our GCS bucket?

No, it adds it into .dvc/cache (similar to git add saving some info into .git locally). dvc push is needed to send data from cache to the remote storage.

Please let me know if this clarifies the flow a bit. Happy to discuss other details here.