Faktory polyglot speed tests

readme.md

Background

Testing out some simple workers with Ruby, Python, and Node. Was surprised to see Ruby was the fastest and Node was the slowest.

Setup

• Ruby 2.7.2
• Node 14.9.0
• Python 3.7.5
• Faktory 1.4.2
• macOS 10.15.7

Steps

Vanilla install of Faktory via Homebrew.

Start Faktory

$ faktory
Faktory 1.4.2
Copyright © 2021 Contributed Systems LLC
Licensed under the GNU Public License 3.0
I 2021-02-28T15:31:12.296Z Initializing redis storage at /Users/josh/.faktory/db, socket /Users/josh/.faktory/db/redis.sock
I 2021-02-28T15:31:12.322Z Web server now listening at localhost:7420
I 2021-02-28T15:31:12.322Z PID 23332 listening at localhost:7419, press Ctrl-C to stop

Start a worker for Ruby, Python, and Node.js

$ faktory-worker -r ./worker.rb -l ruby -q ruby
Starting ruby worker...
2021-02-28T15:35:01.284Z 23609 TID-z0 INFO: Running in ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-darwin19]
2021-02-28T15:35:01.285Z 23609 TID-z0 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2021-02-28T15:35:01.285Z 23609 TID-z0 INFO: Starting processing, hit Ctrl-C to stop

$ python worker.py
INFO:faktory.worker:Registered task: python_worker
INFO:root:Starting python worker...
INFO:faktory.connection:Connecting to localhost:7419
INFO:faktory.worker:Queues: python
INFO:faktory.worker:Labels: python

$ node worker.js
Starting node.js worker...

Start 10,000 jobs each

$ ruby app.rb 10000
Starting 10000 jobs for each (ruby, node, python)
2021-02-28 10:36:13 -0500
Done.

Note the last timestamp for each worker:

Ruby: 2021-02-28T10:37:06-05:00 (~ 53 sec)

Python: 2021-02-28T10:37:25.452204 (~ 72 sec)

Node: 2021-02-28T15:37:38.289Z (~ 85 sec)

app.rb

require 'connection_pool'
require 'faktory'

T = ARGV[0].to_i
puts "Starting #{T} jobs for each (ruby, node, python)"
puts Time.now()

SomeRubyWorker = Faktory::Job.set(queue: 'ruby', jobtype: 'RubyWorker')
SomeNodeWorker = Faktory::Job.set(queue: 'node', jobtype: 'nodeWorker')
SomePythonWorker = Faktory::Job.set(queue: 'python', jobtype: 'python_worker')

T.times do |x|
  SomeRubyWorker.perform_async("this is ruby job #{x}") 
  SomePythonWorker.perform_async("this is python job #{x}") 
  SomeNodeWorker.perform_async("this is node job #{x}") 
end

puts "Done."

worker.js

const faktory = require('faktory-worker')

// Node worker
faktory.register('nodeWorker', async (data) => {
  const f = 1000000
  const arr = Array(f).fill(0) // waste some time
  console.log(`NODE: ${data} -> (${new Date().toISOString()})`)
})

console.log('Starting node.js worker...')
faktory.work({
  queues: ['node'],
  labels: ['node'],
  concurrency: 20
})

worker.py

from faktory import Worker
from datetime import datetime
import logging

logging.basicConfig(level=logging.INFO)

# Python worker
def worker(data):
    buckets = [0] * 1000000  # waste some time
    logging.info("PYTHON: %s -> (%s)", data, datetime.now().isoformat())


w = Worker(queues=["python"], concurrency=20, use_threads=True)
w.register("python_worker", worker)
logging.info("Starting python worker...")
w.run()

worker.rb

# frozen_string_literal: true

require 'faktory'

# Ruby Worker
class RubyWorker
  include Faktory::Job
  faktory_options queue: 'ruby'

  def perform(args)
    buckets = Array.new(1000000, 0)   # waste some time
    puts "RUBY: #{args} -> (#{Time.now.iso8601})"
  end
end

puts "Starting ruby worker..."