Standard for -env.sh

bach-env-sh-standard.md

Goal

Define a standard for how configuration files similar to hadoop *-env.sh should be defined in chef-bach.

Requirement

• Should be able to dynamically add/remove properties to the configuration file
• Should be able to override default values defined in chef-bach through a wrapper cookbook
• Should be able to conditionally add/remove properties to the configuration file

Proposal

• Define the property/value pairs to be stored in the configuration file as chef node attributes in the relevant attribute file
  • relevant -> if the config file is for the hadoop component yarn, the attribute need to be defined in the yarn.rb attribute file in bcpc-hadoop cookbook.
  • The attribute should be defined using the standard default["bach"]["PRODUCT_GROUP"]["COMPONENT_NAME"]["env_sh"]["property"] = value
    • For e.g. default["bach"]["hadoop"]["yarn"]["env_sh"]["YARN_OPTS"] = "-Dcom.sun.management.jmxremote.ssl=false" where PRODUCT_GROUP is hadoop, the hadoop component to which this property is applicable is yarn and YARN_OPTS is one of the property which need to be added to the env.sh file
  • If the component is not part of a "PRODUCT_GROUP" for e.g. kafka or spark then the standard which need to be followed is default["bach"]["COMPONENT_NAME"]["env_sh"]["property"] = value
    • For e.g. ``default["bach"]["kafka"]["env_sh"]["KAFKA_OPTS"] = "-Dcom.sun.management.jmxremote.ssl=false"
• Define a recipe in the cookbook where the attribute was added with the name COMPONENT_NAME_config_filename.rb. For e.g. for yarn env.sh config file the recipe name will be yarn_env.rb
  • For any property which need to be included or updated in the config file based on certain conditions, add the logic to this recipe
    • for e.g. using the .tap ruby method

    if node[:bcpc][:hadoop][:kerberos][:enable] == true then
      default["bach"]["hadoop"]["yarn"]["env_sh"].tap do |env|
        env[:HBASE_OPTS] = '"$HBASE_OPTS -Djava.security.auth.login.config=/etc/hbase/conf/hbase-client.jaas"'
        env[:HBASE_MASTER_OPTS] = '$HBASE_MASTER_OPTS -Djava.security.auth.login.config=/etc/hbase/conf/hbase-server.jaas'
        env[:HBASE_REGIONSERVER_OPTS] += ' -Djava.security.auth.login.config=/etc/hbase/conf/regionserver.jaas'
      end
    end
    

  • At the end of the recipe use a template to generate the config file using generic_env.sh.erb chef-bach template resource.
    • Note generic_env.sh.erb will need to be updated to handle different key/value seperators like = or spaces
• Include this recipe in the recipe which installs the component and before the service gets started

Certain attributes cannot be evaluated unless in a recipe, which is how we are currently winding up with some attributes being defined in attribute files and others in _config recipes. Why not have a recipe per file hive_site_recipe.rb and hive_env_recipe.rb as an example?

Thanks @pu239ppy for the comments. The idea is to have one recipe for each config file. As a start we are trying to define the standard for -env.sh file and as mentioned in the write-up it will have its own recipe. Hope we are on the same page.

I must have missed that, I agree with that notion

Why do we need product groups? we are trying to reduce internal dependencies between components

Went with how the code is currently structured to minimize change to bring the code to common standard. It is not a must we need to have it since the same component existing under different product groups is nil.