rails-performance.md

Performance

1. Browsre -> Load balancer
2. Load blancer to web server nginx
3. nginx to App ser
4. app server to Cache
5. app server to DB

pagespeed insights

https://pagespeed.web.dev/

Memory

processes * (static_memory + (threads * processing_memory))

So in a perfect world, our memory usage formula would now be:

static_memory + (processes * threads * processing_memory)

cat /proc/$PID/smaps_rollup to check memory usage

rbtrace

rbtrace -p 39 -e 'Thread.list'
>> Thread.list

btrace -p 39 -h

bundle exec rbtrace --pid $PID -e 'Thread.new{GC.start;require "objspace";io=File.open("/tmp/ruby-heap.dump", "w"); ObjectSpace.dump_all(output: io); io.close

ps -p $PID -o pid,vsz=MEMORY -o user,group=GROUP -o comm,args=ARGS
ps -w -p $PID -o pid,vsz=MEMORY -o user,group=GROUP -o comm,args=ARGS

* RUBY_GC_HEAP_INIT_SLOTS: Initial allocation slots.
* RUBY_GC_HEAP_FREE_SLOTS: Prepare at least this amount of slots after GC. Allocate slots if there aren’t enough slots.
* RUBY_GC_HEAP_GROWTH_FACTOR: Allocate slots by this factor. (next slots number) = (current slots number) * (this factor)
* RUBY_GC_HEAP_GROWTH_MAX_SLOTS: The allocation rate is limited to this number of slots.
* RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO: Allocate additional pages when the number of free slots is lower than the value (total_slots * (this ratio)).
* RUBY_GC_HEAP_FREE_SLOTS_GOAL_RATIO: Allocate slots to satisfy this formula: free_slots = total_slots * goal_ratio. In other words, prepare (total_slots * goal_ratio) free slots. If this value is 0.0, then use RUBY_GC_HEAP_GROWTH_FACTOR directly.
* RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO: Allow to free pages when the number of free slots is greater than the value (total_slots * (this ratio)).
* RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR: Do full GC when the number of old objects is more than R * N where R is this factor and N is the number of old objects just after last full GC.

```bash
rbtrace -p 40 -e 'Thread.new{require "objspace"; ObjectSpace.trace_object_allocations_start; GC.start(); ObjectSpace.dump_all(output: File.open("/tmp/heap3.json", "w"))}.join'

-e 'Thread.new{ require "stackprof"; StackProf.start(mode: :cpu); sleep 2; StackProf.stop; StackProf.results("/tmp/perf-1"); }'

rbtrace -p 40 -e 'ObjectSpace.memsize_of Prometheus::Client.config.data_store'

ps -xm -o %mem,rss,comm -p $(pgrep puma)

top -b -d $delay -p $pid | awk -v OFS="," '$1+0>0 {
print strftime("%Y-%m-%d %H:%M:%S"),$1,$NF,$9,$10; fflush() }'

siege -b -c200 --header="Authorization: Basic " https://dev.com/health/postgresql.json

GC.count
GC.stat
RubyVM.stat[:global_constant_state]
GC.stat.keys



## APM

https://docs.newrelic.com/docs/apm/new-relic-apm/apdex/apdex-measure-user-satisfaction/
Apdex score

<img width="762" alt="1571648647-queuing" src="https://user-images.githubusercontent.com/82757613/250695360-54b8a346-9c9f-480f-b0d7-354bb6f253b8.png">

* request queueing: hit nginx and pick up by app server(puma)
* redis
* postgres
* ruby


<img width="427" alt="ruby2_0" src="https://user-images.githubusercontent.com/82757613/250695393-5f31237a-647a-42b4-8341-beda58d28af7.png">

## Ruby related metrics
* memory
* objects
* threads
* Ruby GC numbers

```bash
!/bin/bash

run_ab() {
    result=$(ab -q -n "$1" -c "$2" http://localhost:4080/api/v4/projects)
    time_taken=$(echo "$result" | grep "Time taken for tests:" | cut -d" " -f7)
    time_per_req=$(echo "$result" | grep "Time per request:" | grep "(mean)" | cut -d" " -f10)

    echo -e "$1\t$2\t$time_taken\t$time_per_req"
}

for i in 1 2 3 4 5 6 7 8; do
    run_ab $((i*100)) $i
    sleep 1
done

---

HTTP

HTTP2

SSL offloading

gzip with nginx

Cache

1. compile assets and serve from nginx + cloudflare
2. compile assets and copy to bucket to share

HTTP Headers

• If-None-Match
• Expire

ETag: cache with client

CDN

• cloudflare
• gcp bucket

---

Ruby

Different memory allocator

jmalloc

FROM ruby:2.7-slim

RUN apt-get update ; \
    apt-get install -y --no-install-recommends
      libjemalloc2 ; \
    rm -rf /var/lib/apt/lists/*

ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

Docker Image: quay.io/evl.ms/fullstaq-ruby3.2.2-jemalloc

YJIT (>3.2)

https://blog.appsignal.com/2022/09/07/jit-compilers-for-ruby-and-rails-an-overview.html

Just-in-time compilation is a method of running computer code that requires compilation while running a program.

This could entail translating the source code, but it's most frequently done by converting the bytecode to machine code, which is then run directly.

YJIT is a lightweight, minimalistic Ruby JIT built inside CRuby. It lazily compiles code using a Basic Block Versioning (BBV) architecture.

RUBY_YJIT_ENABLE=1

pools

1. DB connection pools
2. Redis connection pools for Sidekiq and cache

Async/multiple threads

puma

Single Mode vs. Cluster Mode Ractors and Fibers

concurrent-ruby

1. CPU bound
2. IO bound

load_async (Rails 7)

Fiber

---

HTTP

HTTP2

SSL offloading

gzip with nginx

Cache

1. compile assets and serve from nginx + cloudflare
2. compile assets and copy to bucket to share

HTTP Headers

• If-None-Match
• Expire

ETag: cache with client

CDN

• cloudflare
• gcp bucket

---

Rails

Faster JSON decoder/encoder

1. Oj

gem 'oj'

# config/initializers/oj.rb
require 'oj'

Oj.optimize_rails()

# Disable encoding of HTML entities in JSON.
ActiveSupport.escape_html_entities_in_json = false

2. redis connection with C

• read heavy as a cache
• read write half/half as job queue

3. Use sidekiq for background jobs

Rails cache at server

• Page Caching
• Action Caching
• Fragment Caching
• Russian Doll Caching
• Shared Partial Caching

redis memcached

1. Rails.cache with redis memory cache
2. IdentityCache

---

DB

conf.echo = false # for irb pry_instance.config.print = proc {} # for pry Avoid hit the DB.

1. Read/write splitting
2. Indexing

• cache hit
• indexing missing
• right type of index: GIN, Hash gem: rails-pg-extras

SELECT
  'index hit rate' AS name,
  (sum(idx_blks_hit)) / nullif(sum(idx_blks_hit + idx_blks_read),0) AS ratio
FROM pg_statio_user_indexes
WHERE schemaname = 'public'
UNION ALL
SELECT
 'table hit rate' AS name,
  sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read),0) AS ratio
FROM pg_statio_user_tables
WHERE schemaname = 'public';

RailsPgExtras.null_indexes
RailsPgExtras.index_info
RailsPgExtras.index_size

SELECT 
  sum(heap_blks_read) as heap_read,
  sum(heap_blks_hit)  as heap_hit,
  sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM 
  pg_statio_user_tables;
  

4. SQL Explain

• Explain
• ? = ANY (linked_ids) -> linked_ids @> '{?}\'

5. N+1 queries

• Sentry
• Gems: Bullet Prosopite

Cronjob to restart deployment

---
# Service account the client will use to reset the deployment,
# by default the pods running inside the cluster can do no such things.
kind: ServiceAccount
apiVersion: v1
metadata:
  name: deployment-restart
  namespace: <namespace>
---
# allow getting status and patching only the one deployment you want
# to restart
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployment-restart
  namespace: <namespace>
rules:
  - apiGroups: ["apps", "extensions"]
    resources: ["deployments"]
    resourceNames: ["<deployment-name>"]
    verbs: ["get", "patch", "list", "watch"] # "list" and "watch" are only needed
                                             # if you want to use `rollout status`
---
# bind the role to the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deployment-restart
  namespace: <namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: deployment-restart
subjects:
  - kind: ServiceAccount
    name: deployment-restart
    namespace: <namespace>
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: deployment-restart
  namespace: <namespace>
spec:
  concurrencyPolicy: Forbid
  schedule: '0 22 */2 * *' # cron spec of time, here, 22 o'clock
  jobTemplate:
    spec:
      backoffLimit: 2 # this has very low chance of failing, as all this does
                      # is prompt kubernetes to schedule new replica set for
                      # the deployment
      activeDeadlineSeconds: 600 # timeout, makes most sense with 
                                 # "waiting for rollout" variant specified below
      template:
        spec:
          serviceAccountName: deployment-restart # name of the service
                                                 # account configured above
          restartPolicy: Never
          containers:
            - name: kubectl
              image: bitnami/kubectl # probably any kubectl image will do,
                                     # optionaly specify version, but this
                                     # should not be necessary, as long the
                                     # version of kubectl is new enough to
                                     # have `rollout restart`
              command:
                - 'kubectl'
                - 'rollout'
                - 'restart'
                - 'deployment/<deployment-name>'

---

Summary

• Usable -> Reliable -> Enjoyable -> Client Happy.
• Observer -> feedback -> make
• Focus on some solution works.

References

1. https://github.com/jemalloc/jemalloc
2. Ruby 3.2’s YJIT is Production-Ready
3. https://storck.io/posts/psa-switch-ruby-docker-to-jemalloc-now/
4. https://dev.to/appaloosastore/active-record-sidekiq-pools-and-threads-18d5
5. https://shopify.engineering/ruby-execution-models