Skip to content

Instantly share code, notes, and snippets.

@atengberg
Last active March 4, 2024 18:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save atengberg/f992d873eeccf5a212f1b52328ff622b to your computer and use it in GitHub Desktop.
Save atengberg/f992d873eeccf5a212f1b52328ff622b to your computer and use it in GitHub Desktop.
(Somewhat naive) Approach to using ZX for managing redis server during node project development
// Using zx to manage background redis server (for development purposes mostly, and doesn't require sudo (except prior to, when disabling redis-server autoboot initially)).
// ( cli user input loop logic via zx question likely suggests its own reusable component TBD; also something similar for zx error handling).
// notice while redis-server cmd is used to start, to cleanly stop redis-cli shutdown is used. Avoids manipulating systemctl (as I saw on several guides) which when doing so tended to spontaneousily crash my system).
// in reverse order for readible coherence, all relative directories flattened for simplicity.
// assume npm esm project... with a script defined:
// npm script: "redis-server": "node index.js" (see below)
// New File!
// Called by the npm script (RedisServer.js/literalConfig.js) would usually be nested in src/* while src/ and this index.js cohabitate for typical namespace resoluton).
//index.js
import RedisServer from "./interactive_redis_server.js";
RedisServer();
// New File!
// Since ESM project, provide common util script for obtaining project base __dirname, configuring zx, dotenv (in direct proximity to actual .env in case of nested subprojects) | would be located in base directory
//commonit.js
process.env.FORCE_COLOR='1' // Supposed to force recolorize terminal output when handled by zx processp's.
// import 'zx/globals' for vscode autocomplete?
import * as zx from 'zx';
const { $, chalk } = zx;
$.verbose = false;
import __dirname from './dirname.js';
import 'dotenv/config';
const blue = s => chalk.blue(s);
export {
zx,
__dirname,
blue
}
//ZX note to log output of a command and not the command themsleves
//$.verbose = false
//await $`program 2>&1`.pipe(process.stdout)
// new File!
// Not super clean, but zx script for managing background redis server instance.
// interactive-redis-server.js
import getLiteralConfig from './literalConfig.js';
import { zx, __dirname, blue } from './commonit.js';
const { $, question, path, fs, chalk } = zx;
const redisDir = path.join(__dirname); // omitted compelte relative path building
const redisServerDir = path.join(redisDir, 'server'); // etc
const configFile = path.join(redisServerDir, 'config', 'redisconfiguration.conf');
const dbPath = path.join(redisServerDir, 'db');
const serverStdOutLogFilePath = path.join(redisServerDir, 'log', 'redis_server_stdoutnerr.txt');
const port = process.env.redis_PORT;
const dbFileName = process.env.redis_DB_NAME;
console.log(chalk.yellow(`Running interactive-redis-server.js...`));
const logAlreadyRunning = () => console.log(chalk.bgYellow("Redis server already running, didn't you see the status message?"))
const logNotRunning = () => console.log(chalk.bgYellowBright("Redis server IS NOT running, didn't you see the status message?"))
const INFO_options = [
'server',
'clients',
'memory',
'persistence',
'stats',
'replication',
'cpu',
'commandstats',
'latencystats',
'sentinel',
'cluster',
'modules',
'keyspace',
'errorstats',
'everything',
'all',
'default'
]
const INFO_options_tty = INFO_options.reduce((a, v) => a + '\t\t' + v + '\n', "");
// redo to use [fcn] instead of switch? also iterate so numbers are automatically derived.
const options = [
`1 - write config file`,
`2 - start redis server`,
`3 - 1 & 2`,
"4 - show query INFO options",
"5 - shutdown redis server",
"6 - 5 & quit this already!",
].reduce((a, v) => a + '\t' + v + '\n', '');
const colorStatus = status => {
switch (status.toLowerCase()) {
case 'offline':
return chalk.grey(status);
case 'online':
return chalk.green(status);
case 'unknown':
return chalk.yellow(status);
default:
return chalk.bgYellow(status);
}
}
// Note the if else else if else... could be a while input loop but easier to handle while sorting out zx process promise redirects.
async function RedisServer() {
// also note if you did not catch at beginning, sudo systemctl disable for redis-server was already run since while devin, wanted to manually start and stop redis-server
let serverRunning = false;
const writeConfigFile = async () => {
try {
await fs.writeFile(configFile, getLiteralConfig({ dbPath, dbFileName, port }));
console.log(`Successfully wrote config file to ${blue(configFile)}`);
} catch (e) {
console.error(chalk.red('Failed to write config file due to \n'), e?.stderr || e?.msg || e);
}
}
const startRedisServer = async () => {
try {
// There should be a way to more simply redirect with ZX's pipe, but piping stdout&stderr to file served equaly well here.
await $`redis-server ${configFile} >> ${serverStdOutLogFilePath} 2>&1 &`;
console.log(`Successfully started redis server writing output to ${blue(serverStdOutLogFilePath)}`);
} catch (e) {
console.error("Failed to start redis server due to ", e?.stderr || e?.msg || e);
}
}
/** Notice, cleanly handles closing the process started by zx above! */
const shutdownRedisServerViaCli = async () => {
try {
await $`redis-cli shutdown`;
console.log(`Successfully shut down redis server! Db dumped to ${dbPath}`);
} catch (e) {
console.error("Failed to shutdown redis server via redis-cli due to ", e?.stderr || e?.msg || e);
}
}
const pingRedisServer = async () => {
try {
const output = (await $`redis-cli ping`).stdout.toString().trimEnd();
console.log(output);
console.log(chalk.yellow(`Redis server says ${chalk.cyanBright(output)}`));
console.log(`Redis server status - ${colorStatus('ONLINE')}`);
serverRunning = true;
} catch (e) {
serverRunning = false;
const lit = `${e.stderr}`;
if (lit.includes('Could not connect to Redis at') && lit.includes('Connection refused')) {
console.log(`Redis server status - ${colorStatus('OFFLINE')}!`)
} else {
console.log(`Redis server status - ${colorStatus('UNKNOWN')}: `, e?.stderr || e?.msg || e);
}
}
}
const queryInfo = async (which = '') => {
try {
const output = (await $`redis-cli INFO ${which}`).stdout.toString();
console.log(`Redis server INFO query ${which} results:\n`, output);
} catch (e) {
const lit = `${e.stderr}`;
if (lit.includes('Could not connect to Redis at') && lit.includes('Connection refused')) {
console.log(`Could not query INFO, the redis server status is ${colorStatus('OFFLINE')}!`)
} else {
console.log(`Redis server status - ${colorStatus('UNKNOWN')}: `, e?.stderr || e?.msg || e);
}
}
}
const getUserInput = async () => {
console.log('\n');
await pingRedisServer();
const userInput = (await question(chalk.bgYellow(`What to do?\n${options}`))).trim().toLowerCase();
if (userInput === '1') {
await writeConfigFile();
await getUserInput();
} else if (userInput === '2') {
if (serverRunning) {
logAlreadyRunning();
} else {
await startRedisServer();
}
await getUserInput();
} else if (userInput === '3') {
await writeConfigFile();
if (serverRunning) {
logAlreadyRunning();
} else {
await startRedisServer();
}
await getUserInput();
} else if (userInput === '4') {
if (serverRunning) {
console.log('\n');
const infoOptionInput = (await question(`What query INFO option (enter nothing for default)?\n${INFO_options_tty}`)).trim().toLowerCase();
if (infoOptionInput === '') {
await queryInfo();
} else {
if (INFO_options.includes(infoOptionInput)) {
await queryInfo(infoOptionInput);
} else {
console.log("\nUnrecognized query INFO option, try again?")
}
}
} else {
logNotRunning();
}
await getUserInput();
} else if (userInput === '5') {
if (serverRunning) {
await shutdownRedisServerViaCli();
} else {
logNotRunning();
}
await getUserInput();
} else if (userInput === '6') {
if (serverRunning) {
await shutdownRedisServerViaCli();
}
console.log("Now is as good a time as any, all systems shutdown. Goodbye!");
process.exit(0);
} else {
console.log(`User input ${userInput} not recognized. Try again?`);
await getUserInput();
}
}
await getUserInput();
}
export default RedisServer;
// new File!
//literalConfig.js
// Not properly setup for streaming/serializing snapshots yet.
/** Generates the literal that can be written as the redis server config file */
function getLiteralConfig({ dbPath, port, maxMemory, dbFileName } = {}) {
if (!dbPath) throw new Error("Can't produce redis server config literal without specified dbPath");
const _maxMemory = maxMemory ?? '50mb';
const maxClients = 10;
const dbPathSansFileName = dbPath;
const dbFileNameSansPath = dbFileName ?? 'redis_dump.rdb';
const _port = port ?? 6379;
const logLevel = 'verbose'; // debug | verbose | notice | warning
const dbCount = 3;
return [
`bind 127.0.0.1`,
`protected-mode yes`,
`port ${_port}`,
`tcp-backlog 511`,
`timeout 0`,
`tcp-keepalive 300`,
`daemonize no`,
`supervised no`,
`pidfile /var/run/redis_${_port}.pid`,
`loglevel ${logLevel}`,
`logfile ""`,
`databases ${dbCount}`,
`always-show-logo yes`,
`save 900 1`,
`save 300 10`,
`save 60 10000`,
`stop-writes-on-bgsave-error yes`,
`rdbcompression yes`,
`rdbchecksum yes`,
`dbfilename ${dbFileNameSansPath}`,
`rdb-del-sync-files no`,
`dir ${dbPathSansFileName}`,
`replica-serve-stale-data yes`,
`replica-read-only yes`,
`repl-diskless-sync no`,
`repl-diskless-sync-delay 5`,
`repl-diskless-load disabled`,
`repl-disable-tcp-nodelay no`,
`replica-priority 0`,
`min-replicas-to-write 0`,
`min-replicas-max-lag 0`,
`acllog-max-len 128`,
`aclfile ""`,
`maxclients ${maxClients}`,
`maxmemory ${_maxMemory}`,
`maxmemory-policy noeviction`,
`lazyfree-lazy-eviction no`,
`lazyfree-lazy-expire no`,
`lazyfree-lazy-server-del no`,
`replica-lazy-flush no`,
`lazyfree-lazy-user-del no`,
`oom-score-adj no`,
`oom-score-adj-values 0 200 800`,
`appendonly no`,
`appendfilename "appendonly.aof"`,
`appendfsync no`,
`no-appendfsync-on-rewrite no`,
`auto-aof-rewrite-percentage 100`,
`auto-aof-rewrite-min-size 64mb`,
`aof-load-truncated yes`,
`aof-use-rdb-preamble yes`,
`lua-time-limit 5000`,
`slowlog-log-slower-than 10000`,
`slowlog-max-len 128`,
`latency-monitor-threshold 0`,
`notify-keyspace-events ""`,
`hash-max-ziplist-entries 512`,
`hash-max-ziplist-value 64`,
`list-max-ziplist-size -2`,
`list-compress-depth 0`,
`set-max-intset-entries 512`,
`zset-max-ziplist-entries 128`,
`zset-max-ziplist-value 64`,
`hll-sparse-max-bytes 3000`,
`stream-node-max-bytes 4096`,
`stream-node-max-entries 100`,
`activerehashing yes`,
`client-output-buffer-limit normal 0 0 0`,
`client-output-buffer-limit replica 256mb 64mb 60`,
`client-output-buffer-limit pubsub 32mb 8mb 60`,
`hz 10`,
`dynamic-hz yes`,
`aof-rewrite-incremental-fsync yes`,
`rdb-save-incremental-fsync yes`,
`jemalloc-bg-thread yes`
].reduce((a, v) => a + v + '\n', '');
}
export default getLiteralConfig;
// new File!
//sampleclients.js
import { Redis } from 'ioredis';
import { zx, __dirname, blue } from './commonit.js';
const { $, question, path, fs, chalk } = zx;
const producer = new Redis();
const consumer = new Redis();
consumer.subscribe('send-data', (err, count) => {
if (err) console.error(err.message);
console.log(`Subscribed to ${count} channels`);
})
consumer.on("message", (channel, message) => {
console.log(`on consumer message with ${channel} and ${JSON.stringify(message)}`);
})
setTimeout(() => {
producer.publish("send-data", JSON.stringify({ data: { data: "data" } }))
}, 1000);
/*
// STREAMS
TO WRITE/PUBLISH:
XADD [stream name] [id | * for AIID/ulid ] [key]-[value] [key]-[value] -> eg XADD order-invoice * id 1 price "$25" created "1698190109" -> "1698189556511-0"
- returns message-id ("1698189556511-0")
TO READ:
XREAD COUNT [n] BLOCK [millie seconds] STREAMS [stream-name] [id] -> XREAD COUNT 10 BLOCK 60000 STREAMS order-invoice $
-? COUNT - limit the number of messages consumed at once
-? BLOCK - timeout to wait for new messages
-STREAMS <stream-name>- name of streams to subscribe too
-ID - [ "0-0" to read all messages from the begining | message-id | $ to recieve only new messages after XREAD is executed]
*/
@atengberg
Copy link
Author

Anyone who knows what they are doing and seeing obvious improvements in the above, comments much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment