Skip to content

Instantly share code, notes, and snippets.

@vsemozhetbyt
Last active July 16, 2017 09:48
Show Gist options
  • Save vsemozhetbyt/0efd0e3d6abc511e95968f9f38120156 to your computer and use it in GitHub Desktop.
Save vsemozhetbyt/0efd0e3d6abc511e95968f9f38120156 to your computer and use it in GitHub Desktop.
'use strict';
const fs = require('fs');
const readline = require('readline');
const files = [];
const [, , target = '.'] = process.argv;
if (fs.statSync(target).isDirectory()) {
files.push(...fs.readdirSync(target)
.map(file => `${target}/${file}`)
.filter(file => !fs.statSync(file).isDirectory()));
} else {
files.push(target);
}
const whitelisted = 'θ'; // used in some transcriptions
const latinScript = `[\\p{Script_Extensions=Latin}\\p{Script_Extensions=Common}${whitelisted}]`;
const anyLetter = `\\p{General_Category=Letter}`;
const latinLetter = `(?=${latinScript})${anyLetter}`;
const otherLetters = RegExp(`(?:(?!${latinScript})${anyLetter})+`, 'gu');
const mix = `(?:${latinLetter}${otherLetters.source}|${otherLetters.source}${latinLetter})`;
const mixedWord = RegExp(`${anyLetter}*${mix}${anyLetter}*`, 'gu');
const wrapAt = /(.{1,75}) /gu;
let mixedWords = 0;
check(files.shift());
function check(file) {
let lineNum = 0;
let withErrors = false;
readline.createInterface({
input: fs.createReadStream(file, { encoding: 'utf8' }),
}).on('line', (line) => {
lineNum++;
const matches = line.match(mixedWord);
if (matches) {
mixedWords += matches.length;
if (!withErrors) {
withErrors = true;
console.log(`\n${file}`);
}
console.log(` Line ${lineNum}: ${
matches.map(str => `${str}: ${str.replace(otherLetters, '($&)')}`)
.join(', ')}`.replace(wrapAt, wrap));
}
}).on('close', () => {
if (files.length) check(files.shift());
else console.log(`\nMixed words: ${mixedWords}.`);
});
}
function wrap(match, chunk, offset, string) {
const rest = string.length - (offset + chunk.length);
return `${chunk}${rest + chunk.length > 75 ? '\n ' : ' '}`;
}
@vsemozhetbyt
Copy link
Author

vsemozhetbyt commented Jul 9, 2017

Usage: node --harmony_regexp_property check-mixed-latin-script.js [path-to-file-or-directory]

If the path argument is not provided, the current directory is checked. Directories are checked without recursion or filters (all files are checked within only defined directory).

Example:

cyrillic-in-latin.txt:

Lоrеm ірsum dоlоr sіt аmеt, соnsесtеtur аdіріsсіng еlіt,
sеd dо еіusmоd tеmроr іnсіdіdunt ut lаbоrе еt dоlоrе mаgnа аlіquа.
Ut еnіm аd mіnіm vеnіаm, quіs nоstrud ехеrсіtаtіоn ullаmсо lаbоrіs nіsі
ut аlіquір ех еа соmmоdо соnsеquаt. Duіs аutе іrurе dоlоr іn rерrеhеndеrіt
іn vоluрtаtе vеlіt еssе сіllum dоlоrе еu fugіаt nullа раrіаtur.
Ехсерtеur sіnt оссаесаt сuріdаtаt nоn рrоіdеnt,
sunt іn сulра quі оffісіа dеsеrunt mоllіt аnіm іd еst lаbоrum. 

node --harmony_regexp_property check-mixed-latin-script.js cyrillic-in-latin.txt:

cyrillic-in-latin.txt
  Line 1: Lоrеm: L(о)r(е)m, ірsum: (ір)sum, dоlоr: d(о)l(о)r, sіt: s(і)t,
    аmеt: (а)m(е)t, соnsесtеtur: (со)ns(ес)t(е)tur, аdіріsсіng:
    (а)d(ірі)s(сі)ng, еlіt: (е)l(і)t
  Line 2: sеd: s(е)d, dо: d(о), еіusmоd: (еі)usm(о)d, tеmроr: t(е)m(ро)r,
    іnсіdіdunt: (і)n(сі)d(і)dunt, lаbоrе: l(а)b(о)r(е), еt: (е)t, dоlоrе:
    d(о)l(о)r(е), mаgnа: m(а)gn(а), аlіquа: (а)l(і)qu(а)
  Line 3: еnіm: (е)n(і)m, аd: (а)d, mіnіm: m(і)n(і)m, vеnіаm: v(е)n(іа)m,
    quіs: qu(і)s, nоstrud: n(о)strud, ехеrсіtаtіоn: (ехе)r(сі)t(а)t(іо)n,
    ullаmсо: ull(а)m(со), lаbоrіs: l(а)b(о)r(і)s, nіsі: n(і)s(і)
  Line 4: аlіquір: (а)l(і)qu(ір), соmmоdо: (со)mm(о)d(о), соnsеquаt:
    (со)ns(е)qu(а)t, Duіs: Du(і)s, аutе: (а)ut(е), іrurе: (і)rur(е), dоlоr:
    d(о)l(о)r, іn: (і)n, rерrеhеndеrіt: r(ер)r(е)h(е)nd(е)r(і)t
  Line 5: іn: (і)n, vоluрtаtе: v(о)lu(р)t(а)t(е), vеlіt: v(е)l(і)t, еssе:
    (е)ss(е), сіllum: (сі)llum, dоlоrе: d(о)l(о)r(е), еu: (е)u, fugіаt:
    fug(іа)t, nullа: null(а), раrіаtur: (ра)r(іа)tur
  Line 6: Ехсерtеur: (Ехсер)t(е)ur, sіnt: s(і)nt, оссаесаt: (оссаеса)t,
    сuріdаtаt: (с)u(рі)d(а)t(а)t, nоn: n(о)n, рrоіdеnt: (р)r(оі)d(е)nt
  Line 7: іn: (і)n, сulра: (с)ul(ра), quі: qu(і), оffісіа: (о)ff(ісіа),
    dеsеrunt: d(е)s(е)runt, mоllіt: m(о)ll(і)t, аnіm: (а)n(і)m, іd: (і)d, еst:
    (е)st, lаbоrum: l(а)b(о)rum

Mixed words: 63.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment