Skip to content

Instantly share code, notes, and snippets.

View renatocron's full-sized avatar

Renato Santos renatocron

  • eokoe
  • Sao Paulo, Brazil
View GitHub Profile

Title: Batching requests and processing in parallel using NestJS

Hello, I have a use case where I want to batch incoming requests from multiple users in my NestJS application to reduce network usage, database queries, and optimize processing. Ideally, I'd like to store the requests and their response objects in an in-memory queue within the same process.

My requirements are:

  • Group incoming requests into a single transaction (e.g., 1000 requests within a second).
  • Perform batch operations in parallel to reduce the number of database queries.
  • Wait for a quorum to be reached for successful operations before sending responses.
  • Handle errors and timeouts without affecting other users' requests.
[druid-gl-brokers-0] 2022-06-09T01:22:09,157 WARN [sql[a8e73b73-b187-4046-9e57-485f16da18bc]] org.apache.druid.sql.http.SqlResource - Failed to handle query: SqlQuery{query='SELECT
[druid-gl-brokers-0] MAX("added") as last_added,
[druid-gl-brokers-0] case when cityName IN ('Abbotsford','Aglayan' ) then '-1' else 0 end as MyFavCitiesFirst
[druid-gl-brokers-0] FROM wikipedia
[druid-gl-brokers-0] WHERE
[druid-gl-brokers-0] __time >= CURRENT_TIMESTAMP - INTERVAL '7' YEAR
[druid-gl-brokers-0] AND
[druid-gl-brokers-0] (
[druid-gl-brokers-0] cityName IN ('Abbotsford', 'Aglayan')
[druid-gl-brokers-0] )
apiVersion: "druid.apache.org/v1alpha1"
kind: "Druid"
metadata:
name: druid-cluster
spec:
image: apache/druid:0.21.1
# Optionally specify image for all nodes. Can be specify on nodes also
# imagePullSecrets:
# - name: tutu
startScript: /druid.sh
@renatocron
renatocron / sample.pl
Created January 19, 2021 23:25
Send SMS Transactional using Amazon::SNS (perl)
use strict;
use Amazon::SNS;
use LWP::ConsoleLogger::Everywhere ();
my $message = 'test' . time;
my $phonenumber = '+55119000000';
$ENV{AWS_SNS_KEY} = 'xxxxx';
$ENV{AWS_SNS_SECRET} = 'yyy';
$ENV{AWS_SNS_ENDPOINT} = '';
SELECT
p.name,
c.id as cliente_id,
coalesce(cp.value, p.initial_value) as value
FROM preferences p
CROSS JOIN clientes c
LEFT JOIN clientes_preferences cp ON cp.cliente_id = c.id AND cp.preference_id = p.id
where c.id=180 and p.name='NOTIFY_COMMENTS_POSTS_CREATED';
/*
SELECT
p.name,
c.id as cliente_id,
coalesce(cp.value, p.initial_value) as value
FROM preferences p,
clientes c
LEFT JOIN clientes_preferences cp ON cp.cliente_id = c.id AND cp.preference_id = c.id
where c.id=180 and p.name='NOTIFY_COMMENTS_POSTS_CREATED';
/*
@renatocron
renatocron / mojo.pl
Created September 24, 2020 23:30
mojo-pretty-user-agent
package Mojo::Transaction::Role::PrettyDebug {
use Mojo::Base -role;
use Mojo::Util 'term_escape';
use DDP;
use constant PRETTY => $ENV{TRACE} || $ENV{MOJO_CLIENT_PRETTY_DEBUG} || 0;
after client_read => sub {
my ($self, $chunk) = @_;
my $url = $self->req->url->to_abs;
$t->get_ok(
'/me/audios/' . $event_id,
{'x-api-key' => $session}
)->status_is(200)->json_has('/audios/0/waveform', 'has waveform')->json_has('/audios/0/bytes', 'row 0 has bytes')
->json_is('/audios/0/id', $audio_1->id, 'row 0 is the right id')
->json_is('/audios/1/id', $audio_2_dup->id, 'row 1 is the right id')
->json_is('/audios/0/event_sequence', 1, 'row 0 is the right sequence')
->json_is('/audios/1/event_sequence', 2, 'row 1 is the right sequence')
->json_is('/audios/2', undef, 'row 2 does not exists')
->json_is('/audios/0/audio_duration', '16s', 'row 0 duration')
package TouchBase::Schema::ResultSet::Voucher;
use common::sense;
use Moose;
use namespace::autoclean;
extends 'DBIx::Class::ResultSet';
with 'TouchBase::Role::Verification';
with 'TouchBase::Role::Verification::TransactionalActions::DBIC';
with 'TouchBase::Schema::Role::ResultsetFind';
package TouchBase::Schema::ResultSet::Voucher;
use common::sense;
use Moose;
use namespace::autoclean;
extends 'DBIx::Class::ResultSet';
with 'TouchBase::Role::Verification';
with 'TouchBase::Role::Verification::TransactionalActions::DBIC';
with 'TouchBase::Schema::Role::ResultsetFind';