frsyuki

Performance degradation due to too many active tasks

When we ran a stress testing tool on Digdag server, the server stopped running tasks. The cause was that propagateAllPlannedToDone and propagateBlockedChildrenToReady methods of io.digdag.core.workflow.WorkflowExecutor class were too slow when there're too many active tasks.

Here is the scenario:

1. Many workflows submit many tasks.

2. Eventually, there're a lot of tasks in PLANNED or BLOCKED tasks.

frsyuki

import java.util.HashMap;
import java.util.Map;
import javax.script.Bindings;
import javax.script.Invocable;
import javax.script.ScriptContext;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
import javax.script.SimpleBindings;
import jdk.nashorn.api.scripting.NashornScriptEngineFactory;

frsyuki

An experimental patch.

diff --git a/ext/msgpack/unpacker.c b/ext/msgpack/unpacker.c
index 8b1f699..6b2f7af 100644
--- a/ext/msgpack/unpacker.c
+++ b/ext/msgpack/unpacker.c
@@ -690,6 +690,12 @@ int msgpack_unpacker_read_map_header(msgpack_unpacker_t* uk, uint32_t* result_si
     return 0;
 }
 

frsyuki

import it.sauronsoftware.cron4j.Predictor;
import it.sauronsoftware.cron4j.SchedulingPattern;
import java.util.Date;
import java.util.TimeZone;
import java.time.Instant;

public class Cron4jTzTest {
    public static void main(String[] args) {
        final TimeZone scheduleTimeZone = TimeZone.getTimeZone("UTC");

frsyuki

Problems

• Deserialization of Ruby objects is slow.
• Filter plugins are slow.

Optimimzation of a record object

A Record class is implemented in C. It has Ruby objects as a serialized binary using MessagePack. When a field is accessed (e.g. record['key']), it returns a deserialized object. This deserializes only necessary value. Thus, if only a few fields are accessed, this improves performance.

frsyuki

Fluentd running on ServerEngine

Logging

• ServerEngine initializes a logger
  • config[:log] config parameter is required. Fluentd will set it.
• Fluentd::Log writes logs to ServerEngine's logger so that it can use ServerEngine::DaemonLogger's log rotation config.
  • Fluentd::Log#puts(msg) calls ::Logger#<<(msg) (note: ServerEngine::DaemonLogger extends ::Logger)
• Fluentd::Log#initialize takes initialize(logger, opts={}) as the arguments (logger is ::Logger) instead of initialize(out, level, opts).
• Fluentd::Log#initialize copies @level from logger.level.

frsyuki

frsyuki

Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows.
java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
    at jnr.ffi.provider.InvalidProvider$1.loadLibrary(InvalidProvider.java:48)
    at jnr.ffi.LibraryLoader.load(LibraryLoader.java:290)
    at jnr.ffi.Library.loadLibrary(Library.java:127)
    at jnr.posix.POSIXFactory$DefaultLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:279)
    at jnr.posix.POSIXFactory$DefaultLibCProvider.getLibC(POSIXFactory.java:283)
    at jnr.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:36)
    at jnr.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:18)
    at jnr.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:143)

frsyuki

2016-02-08_06:12:25.56117 2016-02-08 06:12:25.558 +0000 [WARN] (qtp54440148-21): Unprocessable Entity: ParserPlugin 'query_string' is not found.
2016-02-08_06:12:25.56118 (RuntimeError) can't add a new key into hash during iteration
2016-02-08_06:12:25.56119 org.embulk.config.ConfigException: ParserPlugin 'query_string' is not found.
2016-02-08_06:12:25.56119 (RuntimeError) can't add a new key into hash during iteration
2016-02-08_06:12:25.56120 	at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:72) ~[embulk-core-0.7.10.jar:na]
2016-02-08_06:12:25.56120 	at org.embulk.plugin.PluginManager.newPluginWithoutWrapper(PluginManager.java:57) ~[embulk-core-0.7.10.jar:na]
2016-02-08_06:12:25.56120 	at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:30) ~[embulk-core-0.7.10.jar:na]
2016-02-08_06:12:25.56121 	at org.embulk.spi.ExecSession.newPlugin(ExecSession.java:166) ~[embulk-core-0.7.10.jar:na]
2016-02-08_06:12:25.56121 	at org.embulk.spi.Exec.newPlugin(Exec.java:74) ~[e

frsyuki

diff -r 5bfdcfdb618a src/main/java/org/yaml/snakeyaml/resolver/Resolver.java
--- a/src/main/java/org/yaml/snakeyaml/resolver/Resolver.java	Fri Aug 29 15:53:34 2014 +0300
+++ b/src/main/java/org/yaml/snakeyaml/resolver/Resolver.java	Tue Jan 12 19:46:01 2016 -0800
@@ -37,7 +37,7 @@
      * added to keep backwards compatibility
      */
     public static final Pattern FLOAT = Pattern
-            .compile("^([-+]?(\\.[0-9]+|[0-9_]+(\\.[0-9_]*)?)([eE][-+]?[0-9]+)?|[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*|[-+]?\\.(?:inf|Inf|INF)|\\.(?:nan|NaN|NAN))$");
+            .compile("^([-+]?(\\.[0-9]+|[1-9][0-9_]*(\\.[0-9_]*)?)([eE][-+]?[0-9]+)?|[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*|[-+]?\\.(?:inf|Inf|INF)|\\.(?:nan|NaN|NAN))$");
     public static final Pattern INT = Pattern