Skip to content

Instantly share code, notes, and snippets.

@gaborgsomogyi

gaborgsomogyi/Spark parameter uppercase or lowercase magic in 3.0.md

Last active April 9, 2019 09:02
Show Gist options
  • Save gaborgsomogyi/4239c804a3264f680112d366cae46fd1 to your computer and use it in GitHub Desktop.
Save gaborgsomogyi/4239c804a3264f680112d366cae46fd1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Spark parameter uppercase or lowercase magic in 3.0.md

Parameters are arriving to source/sink lowercase.

maxOffsetsPerTrigger parameter as an example:

  • KafkaSourceProvider uses caseInsensitiveParams which converts keys to lowercase
  • KafkaMicroBatchStream uses CaseInsensitiveStringMap where get operation uses lowercase conversion
  • KafkaSource uses CaseInsensitiveMap where get operation uses lowercase conversion

In the last case CaseInsensitiveMap extends Map and as said it provides lowercase key lookup but in the interface only Map is passed which doesn't enforce this implementation detail. The possible future problem is that there are code parts which are depending on this implementation detail. An example for this is groupidprefix (there are others so don't want to finger-point). If the implementation behind the scenes change all such code parts will break.

The final statement it works but brittle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment