Skip to content

Instantly share code, notes, and snippets.

Last active April 9, 2024 18:36
Show Gist options
  • Save pkuczynski/8665367 to your computer and use it in GitHub Desktop.
Save pkuczynski/8665367 to your computer and use it in GitHub Desktop.
Read YAML file from Bash script
parse_yaml() {
local prefix=$2
local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
sed -ne "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" \
-e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p" $1 |
awk -F$fs '{
indent = length($1)/2;
vname[indent] = $2;
for (i in vname) {if (i > indent) {delete vname[i]}}
if (length($3) > 0) {
vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
# include parse_yaml function
# read yaml file
eval $(parse_yaml zconfig.yml "config_")
# access yaml content
echo $config_development_database
adapter: mysql2
encoding: utf8
database: my_database
username: root
Copy link

I tried to use this in Jenkins for job config parsing.
I ran successfully with Python 2.6.6 but not with Python 2.7.5
Help is really appreciated.

Ran with Python 2.6.6

  • echo 'Get Job Properties'
    Get Job Properties
  • /usr/bin/python -V
    Python 2.6.6
  • cd /bms/webapps/jenkins/workspace/pro-group/test/yaml_parser
  • .
    ++ parse_yaml access_info.yaml accessconfig_

Ran with Python 2.7.5

  • echo 'Get Job Properties'
    Get Job Properties
  • /usr/bin/python -V
    Python 2.7.5
  • cd /bms/webapps/jenkins/workspace/pro-group/test/yaml_parser
  • .
    /bms/webapps/jenkins/ line 29: .: file not found
    Build step 'Execute shell' marked build as failure

Copy link

Thank you for publishing! Helped save alot of time 👍

Copy link

4383 commented Oct 9, 2019


niet now support eval output format so you can do things like that:

$ pip install -U niet
$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval .
 foo_biz="bar";fizz__buzz=( zero one two three )
$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval .)
$ echo ${foo_biz}
$ echo ${fizz__buzz}
zero one two three
$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval '"foo-biz"'); echo ${foo_biz}
$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval
fizz_buzz=( zero one two three );
$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval
$ for el in ${fizz_buzz}; do echo $el; done

niet work with JSON and YAML input, also you can convert each format to each other, just by using pip install -U niet.

Copy link

mundrusatish commented Nov 21, 2019

Thank you. Can we validate that the username as below is mandatory and have value, non empty field. Right now it passed.
adapter: mysql2
encoding: utf8
database: my_database
if anyone tried, please share. Thanks much

Copy link

bukowa commented Jan 26, 2020

Copy link

@pkuczynski, It's work for me as well to read the key in yaml file. Thanks for that. Do you have the script to replace the value after reading? i have to replace the each value which i reads .

Copy link

inieves commented Apr 25, 2020

Hi! What is the expected output from running Perhaps listing an output file above would be useful.

Copy link

Works great.

Copy link

nice script!

Copy link

niekwit commented Feb 14, 2021

Works really well, thank you!

Copy link

description: sample message here
may overflow next line

I have a similar issue how could we store description variable that may overflow into the next line in the same

1_2_1_description="sample message here"
1_2_1_description="sample message here may overflow next line"

Thank you!

Copy link

aahnik commented Apr 3, 2021

I think python is superior in this scripting situation. why on earth anyone would use a bash script to do such a thing? python is generally pre-installed in mac and Linux.

python is so much clean and readable.

pip install pyyaml

import yaml

FILENAME = 'your_file.yml'

with open(FILENAME) as file:
    data:dict = yaml.full_load(file)

# data is a python dictionary

Copy link

The traversal logic is super. Love it !! Sometimes its just fun to do these crazy stuff with Bash.

Copy link

jq for JSON and yq for YAML

Copy link

kokosowy commented Sep 8, 2021

Hi all! This is very interested thread. Anybody knows how to make it working where string spans several lines (through "|" pipe), example:

key1: |
   1st line
   2nd line
   3rd line
key2: |
   1st line
   2nd line
   3rd line

Copy link

sonjz commented Oct 28, 2021

thanks for this.
added this to end after awk as a hack not to interpret $ interpolation (application: reading serverless.yml config into bash)

 | tr "$" "#" # don't want to interpret $

Copy link

i cant figure out why its not running succesfully for me whenever i execute ./ it gives me error
./ line 4: .: file not found
./ line 7: parse_yaml: command not found

Copy link

has anybody else faced this issue please help

Copy link

potes74 commented Jan 30, 2022

@Nigam8972 try with
. ./
if they are in the same folder (it's what I use)

Copy link

ppenguin commented Mar 2, 2023

Cool, just discovered this; of course the more recent suggestions are more robust and feature-rich, but I found myself needing this on constrained (i.e. busybox, sh and no python) systems where going POSIX-sh only is the way to go.

The method here by @wadewegner (or similar) to pre-process problematic keys (but not their values) was the way to go, i.e. by using "advanced" sed hold-space etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment