CodeAmp Project Configuration Design Doc

project_configuration_design_doc_v1.md

The CodeAmp Project Configuration

Problem

Currently, there is a high cost to migrating a project configuration from one environment to another. As a result, projects with large configurations rarely get migrated to other environments for development since there is so much friction in doing so.

Solution

A project-wide, environment-scoped configuration that can be applied to different environments of that project or to a completely different project as well. It will have the following properties:

Server

• resources within configuration are project and environment scoped
• supports both import and export of a configuration.
  • with import, a configuration is appled for a specific project and environment
  • With export, a configuration is built from a specific project and environment
• supported resource types include:
  • Secrets
  • Services
  • ProjectExtensions
  • ProjectSettings

Client

• option in project settings page to import and export a project configuration

YAML Project Spec Format

project:
  secrets:
    - key: KEY
      value: value
  services:
    - name: foo-service
      otherAttributes: ..
  extensions:
    - name: extension-name
      customConfig: customconfig
  settings:
    branch: master                                                                                                                                               

Design Challenges

1. Isolating child object dependencies from the project configuration when we build it for exporting
2. When importing a configuration, it should be easy to process child components without coupling the overall behavior with the specific child handlers
3. Project should be able to handle n resource types in its configuration

Note: other design challenges but will not be solved in this iteration include:

• backwards compatibility with previous config versions

Implementation

Server

• new files. A file for defining the ResourceConfig

  • plugins/codeamp/resourceconfig/service.go
  • plugins/codeamp/resourceconfig/project.go
  • plugins/codeamp/resourceconfig/secret.go
  • plugins/codeamp/resourceconfig/service.go
  • plugins/codeamp/resourceconfig/projectsettings.go
  • plugins/codeamp/resourceconfig/extension.go
  • plugins/codeamp/resourceconfig/interfaces.go

• new interfaces

  type ResourceConfig interface {
      Import() error
      ExportYAML() (string, error)
      GetResource() interface{}
      GetConfig() string
      SetConfig(string)
      GetChildResourceConfigs() []ResourceConfig    
  }

• example struct implementing ResourceConfig

  // example struct that implements ResourceConfig
  type ProjectConfig struct {
      ResourceConfig // inherits methods from ResourceConfig
      db *gorm.DB
      project *Project
      environment *Environment
      config string
  }
  
  func createProjectConfig(db *gorm.DB, project *Project, env *Environment) *ProjectResourceConfig {
      return &ProjectConfig{
          db: db,
          project: project,
          environment: environment,
      }
  }
  
  
  func (p *ProjectConfig) Import() error {
      childConfigs := p.GetChildResourceConfigs()
      for _, config := range childConfigs {
          err := config.Import()
          if err != nil {
              return err
          }
      }
  }
  
  func (p *ProjectConfig) ExportYAML() (string, error) {
      aggregateConfigString := ``
      project := Project{}
      
      childConfigs := p.GetChildResourceConfigs()
      for _, config := range childConfigs {
          configString, err = config.ExportYAML()
          if err != nil {
            return ``, err
          }
          
          aggregateConfigString += configString
      }
      
      // test if can be unmarshaled
      err := yaml.Unmarshal(&project, aggregateConfigString)
      if err != nil {
        return ``, err
      }
      
      return aggregateConfigString, nil
  }  
  
  func (p *ProjectConfig) GetResource() interface{} {
      return p.project
  }
  
  func (p *ProjectConfig) GetConfig() string {
      return p.config
  }
  
  func (p *ProjectConfig) SetConfig(config string) string {
      p.config = config
  }
  
  func (p *ProjectConfig) GetChildResourceConfigs() []ResourceConfig {
      childResourceConfigs := []ResourceConfig{}
      // unmarshal config and get resource configs for each child object
      unmarshaledProjectConfig := ...
      for _, service := range unmarshaledProject.Services {
        svcConfig := createProjectServiceConfig(p.db, service, p.environment)
        // ignore err since it's coming from an unmarshaled yaml obj
        marshaledYamlSvcConfig, _ := yaml.Marshal(svcConfig)
        svcConfig.SetConfig(marshaledYamlSvcConfig)
        childResourceConfigs = append(childResourceConfigs, serviceConfig)
      }  
      ...    
      return childResourceConfigs
  }
  
  // ProjectConfig specific method
  func (p *ProjectConfig) GetEnvironment() Environment {
      return p.environment
  }

• ImportProject pseudo-code

  func (r *Resolver) ImportProject(config string, envID string) error {
    projectConfig := createProjectConfig(r.db, r.project, r.env)
    err := projectConfig.Import()
    if err != nil {
        return err
    }
  }

Client

• add import button in the Project/Settings page
  • applies a given project configuration to the project scoped within the chosen environment
• add export button in the Project/Settings page
  • produces a project configuration spec for download to the user, all resources in the config scoped for the chosen environment

The Future

This is a first step towards an infrastructure-as-code initiative in which we'll be able to store our entire pipeline configuration and resources all within a YAML file. Using that as a single source of truth, we'll be able to inspect the file in corresponding repository and reflect that onto our clients.