Skip to content

Instantly share code, notes, and snippets.

Avatar

Sergei Morozov morozov

  • SugarCRM
  • San Francisco Bay Area
  • @srgmrzv
View GitHub Profile
@morozov
morozov / README.md
Last active Mar 3, 2021
DBZ-2975: Support for Multiple Databases per SQL Server Connector
View README.md

Motivation

Currently, it is challenging to use Debezium Connector for SQL Server in a multi-tenanted environment where a single SQL Server instance hosts tens to hundreds of similar tenant-scoped databases due to the fact that a given connector instance can capture changes only from a single database. The challenges are:

  1. Kafka Connect resources. Deploying hundreds of connector instances will require a Kafka Connect cluster with significant CPU and memory resources.
  2. Cluster stability. If the resources above are insufficient (e.g. memory), it may cause a Connect worker crash which in turn will fail all connector tasks scheduled to the worker. In the best case scenario, those tasks can be just restarted. In a worse case scenario, the restarted connectors will produce duplicates since the source connectors currently don't implement exactly-once processing semantics. In an even worse case, a worker can fail during a long-running snapshot, handling which will require additional cleanup.
  3. SQL Server res
@morozov
morozov / README.md
Created Jan 23, 2021
Support for Multiple Databases per SQL Server Connector
View README.md

Support for Multiple Databases per SQL Server Connector

Background

Unlike the Debezium connector for MySQL which consumes the changes from all databases via a single binlog, the SQL Server connector interacts with the CDC schema which is deployed individually for each database.

At SugarCRM, we have a number of SQL Server instances hosting more than a hundred databases each, the changes from which we'd like to capture using the Debezium connector for SQL Server. Having to deploy a hundred connectors per instance may be suboptimal both from the resourcing and the operations standpoints.

Research

@morozov
morozov / benchmark.php
Last active Nov 12, 2020
SQL Parser Benchmark
View benchmark.php
<?php
use Doctrine\DBAL\Connection;
use Doctrine\DBAL\ExpandArrayParameters;
use Doctrine\DBAL\ParameterType;
use Doctrine\DBAL\SQL\Parser;
use Doctrine\DBAL\SQLParserUtils;
require __DIR__ . '/vendor/autoload.php';
@morozov
morozov / diff-class-map.sh
Created Aug 12, 2020
Generates the diff between Composer autload files from a pull request
View diff-class-map.sh
#!/usr/bin/env bash
set -eu
if [ $# -ne 1 ]; then
echo "Usage: `basename $0` <pull>";
exit 1;
fi
PULL="$1"
@morozov
morozov / .gitignore
Last active Jun 20, 2020
PHPUnit and Psalm
@morozov
morozov / .gitignore
Last active May 26, 2020
composer/package-versions-deprecated incompatible with humbug/box
View .gitignore
/example.phar
/vendor
View README.md
sudo ./sockdump.py --format string --output dump.txt ~/.phpbrew/php/php-7.4.5/var/run/php-fpm.sock
@morozov
morozov / watch-and-diff.sh
Last active Oct 10, 2019
Watch and Diff. A shell script for monitoring changes in configuration files
View watch-and-diff.sh
#!/usr/bin/env bash
set -euo pipefail
function __get_missing_keys {
php -- "$@" << 'EOF'
<?php
$args = $GLOBALS['argv'];
array_shift($args);
@morozov
morozov / backport.sh
Last active Dec 6, 2018
A shell script for back-porting Doctrine DBAL pull requests from master to older branches
View backport.sh
#!/usr/bin/env bash
set -eu
if [ $# -ne 2 ]; then
echo "Usage: `basename $0` <pull> <branch>";
exit 1;
fi
PULL="$1"
@morozov
morozov / disallow-equal-operators.diff
Created Nov 24, 2018
Applied SlevomatCodingStandard.ControlStructures.DisallowEqualOperators to ba4c9a2
View disallow-equal-operators.diff
diff --git a/lib/Doctrine/DBAL/Connection.php b/lib/Doctrine/DBAL/Connection.php
index 2a1d9a1b9..4a9a9231a 100644
--- a/lib/Doctrine/DBAL/Connection.php
+++ b/lib/Doctrine/DBAL/Connection.php
@@ -1254,7 +1254,7 @@ class Connection implements DriverConnection
$logger = $this->_config->getSQLLogger();
- if ($this->_transactionNestingLevel == 1) {
+ if ($this->_transactionNestingLevel === 1) {