AppConfig: Data loss with strongly typed settings

problem-statement.md

AppConfiguration: Data loss with strongly typed settings

Overview

The general form of settings stored in AppConfiguration are simple key/value string pairs, where both have no formal structure. AppConfiguration also has the notion of "feature flag" and "secret reference" configuration settings, which are loosely structured data, conforming to a minimal and extensible JSON schema. These settings are expected to include all elements from the JSON schema but are also permitted to include additional data in the form of simple or complex elements.

Primary champion scenario

• A developer creates settings in the portal, using a combination of the schema entry form and advanced JSON-centric view for custom attributes.
• The settings are read by a client application using a platform integration package owned by the service team owns (e.g. ASP.NET, Spring).
• The application does not directly reference or interact with the Azure SDK package.

Problem statement

Data that exists as part of the setting in the service is inaccessible to applications using the integration packages or the client for .NET and Java. Any updates made using the client will cause data loss.

When using the Azure SDK packages for AppConfiguration in .NET or Java, a setting is considered to be structured data if the content type identifies data as one of the structured settings and the requisite JSON schema elements are present. An undocumented assumption is made that only the elements identified by the schema are valid. Developers using the settings types in the client library are intended to have the ability to read only the schema data, ignoring any extensions. If any updates are made using the client, they include only the schema elements and remove any additional data that was present.

Proposed behavioral change

• When the client parses a feature flag or secret reference configuration setting, it extracts the known schema elements into strongly-typed properties.
• The original JSON form is retained and accessible using the setting's Value property.
• The schema elements can be read and updated using the strongly-typed properties.
• Any updates using the stongly-typed properties cause an in-place update to the raw JSON, preserving the structure and unknown elements.
• Any updates to the raw JSON Value are treated and parsed as a new value. The strongly-typed property values are overwritten.

Note: The above assumes the setting value conforms to the schema structure and contains the required elements. For scenarios where that is not true, the current behavior of treating the data as a raw string will be maintained. This is described in more detail in What happens when a setting is not valid to the schema.

What the other languages do

Python

The client library in Python already conforms to the proposed change. The library returns both the schema fields and extensions, preserving non-schema data when updates are made.

JavaScript

The client library in JS has a slightly different experience than others. Because JSON is a concept embedded into the language itself, the core developer experience for working with settings involves interacting with the JSON returned by the service directly. If developers wish, they can use a parse function to produce an immutable view of the schema fields. This view is only for developer convenience and not used by the client library for updates. As a result, the full set of AppConfiguration service data is accessible to developers and non-schema extensions are preserved during updates.

Addendum: Current implementation details

What happens when a setting is not valid to the schema

• If a setting value is not valid to the schema, it is treated as an opaque string value with no structure assumed.
• Strongly-typed properties associated with the schema are not accessible, throwing an exception when called.
• The raw string value may be read and written by callers, the model does not attempt to enforce the schema.
• If the setting value is set by a caller and the new value conforms to the schema, behavior changes to Valid Behavior.

What happens when a setting is valid to the schema

• If the setting value conforms to the schema, it is parsed into the strongly-typed schema properties.
• If a property value is updated, the corresponding item in the raw string JSON value is updated.
• Unknown values in the JSON not represented by the strongly-typed properties are silently dropped.
• The raw string value of the setting may be read and written by callers, but reflects only the JSON schema structure.
• If the raw string value is set by a caller and the new value does not conform to the schema, behavior changes to Invalid Behavior.

What "valid to the schema" means

In a nutshell, this means that the setting must have the expected structure of the schema, whether an element is populated or not. If there is data for an element, it must be of the expected type. The setting may have additional or empty elements, which are ignored.

For example, with the Feature Flag setting type, this means:

Must have:
  - Id
  - Enabled
  - Conditions

May have:
  - Arbitrary user-defined properties (will be dropped)
  - Description
  - Display Name
  - Client Filters
      - Any item must have Name, may have Parameters

References

• Feature flag JSON schema
• AppConfiguration feature management overview

PORTAL - original value which contains "requirement_type": "All" that is not a valid schema property (which is an unknown/extensible property)

{
    "id": "hello",
    "enabled": true,
    "conditions": {
        "requirement_type": "All",
        "client_filters": [
            {
                "name": "Microsoft.Percentage",
                "parameters": {
                    "Value": 30
                }
            }
        ]
    }
}

GET response (include "requirement_type") which is what we expect

{
  "id": "hello",
  "enabled": true,
  "conditions": {
    "requirement_type": "All",
    "client_filters": [
      {
        "name": "Microsoft.Percentage",
        "parameters": {
          "Value": 30
        }
      }
    ]
  }
}

UPDATE request - (updated the percentage value to 99 but lose data "requirement_type" before send it out to service)

{
  "id": "hello",
  "enabled": true,
  "conditions": {
    "client_filters": [
      {
        "name": "Microsoft.Percentage",
        "parameters": {
          "Value": 99
        }
      }
    ]
  }
}

GET after UPDATE (data loss - "requirement_type", it validates the send request is updated in the portal)

{
  "id": "hello",
  "enabled": true,
  "conditions": {
    "client_filters": [
      {
        "name": "Microsoft.Percentage",
        "parameters": {
          "Value": 99
        }
      }
    ]
  }
}