yaauie/unicode-and-control-chars.txt

Created December 5, 2022 19:53

Star 0 You must be signed in to star a gist
Fork 0 You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/yaauie/1a4018fb51dcd6a60d6926f3c24a3ba5.js"></script>
Save yaauie/1a4018fb51dcd6a60d6926f3c24a3ba5 to your computer and use it in GitHub Desktop.

Download ZIP

Raw

unicode-and-control-chars.txt

	literal: 👍
	u-encoded: \u001c\u{1F44D}
	inspected: "\u001C👍"
	dumped: "\x1C\u{1F44D}"
	json-encoced: "\u001c👍"

Author

yaauie commented Dec 5, 2022

Running the above through three outputs with different codecs (line, json_lines, rubydebug), we see that both non-printing characters and multibyte unicode characters make it through as-provided.

╰─● curl --silent https://gist.githubusercontent.com/yaauie/1a4018fb51dcd6a60d6926f3c24a3ba5/raw/7ef86f5e5c649b74f2e5ec8cd128ac0439e83452/unicode-and-control-chars.txt | bin/logstash -e 'output { stdout { codec => line { format => "line codec: %{message}"} } stdout { codec => json_lines } stdout { codec => rubydebug } }'
Using system java: /Users/yaauie/.jenv/shims/java
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /Users/yaauie/src/elastic/ls/logs which is now configured via log4j2.properties
[2022-12-05T20:25:00,734][INFO ][logstash.runner          ] Log4j configuration path used is: /Users/yaauie/src/elastic/ls/config/log4j2.properties
[2022-12-05T20:25:00,742][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-12-05T20:25:00,743][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.6.0", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 11.0.16.1+0 on 11.0.16.1+0 +indy +jit [arm64-darwin]"}
[2022-12-05T20:25:00,744][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-12-05T20:25:00,761][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-12-05T20:25:00,976][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-12-05T20:25:01,065][INFO ][org.reflections.Reflections] Reflections took 48 ms to scan 1 urls, producing 132 keys and 462 values
[2022-12-05T20:25:01,148][INFO ][logstash.codecs.jsonlines] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2022-12-05T20:25:01,187][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-12-05T20:25:01,197][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1250, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x1ad932a8@/Users/yaauie/src/elastic/ls/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2022-12-05T20:25:01,532][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.33}
[2022-12-05T20:25:01,553][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-12-05T20:25:01,563][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
line codec: literal: 👍
line codec: u-encoded: \u001c\u{1F44D}
line codec: json-encoced: "\u001c👍"
line codec: inspected: "\u001C👍"
line codec: dumped: "\x1C\u{1F44D}"
{"message":"literal: \u001C👍","@version":"1","@timestamp":"2022-12-05T20:25:01.561936Z","event":{"original":"literal: \u001C👍"},"type":"stdin","host":{"hostname":"maybe.lan"}}
{"message":"dumped: \"\\x1C\\u{1F44D}\"","@version":"1","@timestamp":"2022-12-05T20:25:01.563495Z","event":{"original":"dumped: \"\\x1C\\u{1F44D}\""},"type":"stdin","host":{"hostname":"maybe.lan"}}
{"message":"inspected: \"\\u001C👍\"","@version":"1","@timestamp":"2022-12-05T20:25:01.563428Z","event":{"original":"inspected: \"\\u001C👍\""},"type":"stdin","host":{"hostname":"maybe.lan"}}
{"message":"json-encoced: \"\\u001c👍\"","@version":"1","@timestamp":"2022-12-05T20:25:01.563559Z","event":{"original":"json-encoced: \"\\u001c👍\""},"type":"stdin","host":{"hostname":"maybe.lan"}}
{"message":"u-encoded: \\u001c\\u{1F44D}","@version":"1","@timestamp":"2022-12-05T20:25:01.563311Z","event":{"original":"u-encoded: \\u001c\\u{1F44D}"},"type":"stdin","host":{"hostname":"maybe.lan"}}
{
       "message" => "literal: \u001C👍",
      "@version" => "1",
    "@timestamp" => 2022-12-05T20:25:01.561936Z,
         "event" => {
        "original" => "literal: \u001C👍"
    },
          "type" => "stdin",
          "host" => {
        "hostname" => "maybe.lan"
    }
}
{
       "message" => "u-encoded: \\u001c\\u{1F44D}",
      "@version" => "1",
    "@timestamp" => 2022-12-05T20:25:01.563311Z,
         "event" => {
        "original" => "u-encoded: \\u001c\\u{1F44D}"
    },
          "type" => "stdin",
          "host" => {
        "hostname" => "maybe.lan"
    }
}
{
       "message" => "json-encoced: \"\\u001c👍\"",
      "@version" => "1",
    "@timestamp" => 2022-12-05T20:25:01.563559Z,
         "event" => {
        "original" => "json-encoced: \"\\u001c👍\""
    },
          "type" => "stdin",
          "host" => {
        "hostname" => "maybe.lan"
    }
}
{
       "message" => "inspected: \"\\u001C👍\"",
      "@version" => "1",
    "@timestamp" => 2022-12-05T20:25:01.563428Z,
         "event" => {
        "original" => "inspected: \"\\u001C👍\""
    },
          "type" => "stdin",
          "host" => {
        "hostname" => "maybe.lan"
    }
}
{
       "message" => "dumped: \"\\x1C\\u{1F44D}\"",
      "@version" => "1",
    "@timestamp" => 2022-12-05T20:25:01.563495Z,
         "event" => {
        "original" => "dumped: \"\\x1C\\u{1F44D}\""
    },
          "type" => "stdin",
          "host" => {
        "hostname" => "maybe.lan"
    }
}
[2022-12-05T20:25:01,772][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-12-05T20:25:02,072][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2022-12-05T20:25:02,079][INFO ][logstash.runner          ] Logstash shut down.
[success (00:00:06)]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment