template for readme.md file

README_template.md

Project Name

General info

Short description what it is for.

Stakeholders: for external - list departments, for internal - projects or ppl who depend. (To evaluate how critical if it goes down in production once and to inform about problems).

Input Data (if applicable):

• name, hdfs/external path;
• ...

Output Data (if applicable):

• hdfs path;
• someDB

If it is a data pipeline, describe main stages of it, diagram or text like: Drop previously created tables in Impala. Ingest data from Oracle <name_db> to hive metastore as PARQUET, store it as table_name. Associate it with external impala tables with corresponding names. Write a SUCCESS_ file on <flag_path>.

Scheduled to run: daily at 22 o'clock/ once a week/ every 5 days/ etc.

Features

• send logs to kibana <link_to_kibana_index>
• failure alerts are sent to <email_address>

Ownership

• project owner: name here
• substitute person: name here
• most domains knowleadge people are: names

Developing

Pre-requisites

• java8, python 3.x at $PATH
• sbt/mvn version etc
• system variabels: —list all system variables introduced by the project, if applicable
• something else

Package dependences

• link on requirements file (for python projects)
• conda env, if applicable.

For contributers

--step by step guide how to prepare for developing

1. cd your_path/to_projects_dir
2. git clone <project link>
3. instructions to install depended packages, if applicable, or requirements file for python projects
4. commands how to build and run unit tests, depends on project build system sbt clean build test
5. git checkout -b feature/<new_branch_name>
6. develop your thing, with local commits
7. when done, create PR to master branch

Testing

Locally

How to run unit tests

--if applicable, either no unit tests

Lab Environment

Integration testing

--step by step how to run integrational tests, or no integrational tests

Functional testing

--step by step instruction or no functional tests available

Deployment

--step by step guide to deploy for different environments

Lab Environment

1. deploy with idea on <server_address> under <user_name>
2. ssh to <server_name>
3. cd <path_to_project>
4. perform in CLI command_to_perform or run script.

Flags for successfull run:

• creates new directory <path_to_dir>
• success files on a path <path_to_success_flag_file>
• on <http_link> something will appear
• select * from <table_name> limit 1; will return result
• something else

Else:

• review logs <path_to_logs>
• rerun in debug mode: command --debug
• something else

Production Environment

1. create PR, choose reviewers
2. after approve merge with master
3. follow jenkins <project_name> build progess

Flags for sucessfull run:

• creates new directory <path_to_dir>
• success files on a path <path_to_success_flag_file>
• on <http_link> something will appear
• something else

Else:

• description for alert system (sending leter of failure on that address)
• logs for production are kept
• work scheduler is
• something else

First Aid Documentation

1. Error scenarios: —describe all possible and known error scenarios and their resolution

• error or misbehavior been seen (detailed description with errors messages)
• solution (step-by-step)

2. Link to page with credentials;
3. Path to all logs and monitoring available, what to look for;
4. Link to postmortems;
5. Links to other useful things;
6. Something what is not listed in readme, but is essential to add.