Skip to content

Instantly share code, notes, and snippets.

@kgrabs
Last active January 7, 2024 07:08
  • Star 2 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save kgrabs/b042214d715176ac9ab563c188995457 to your computer and use it in GitHub Desktop.
Desampling functions for Avisynth+, replaces DebilinearM and lineart_rpow2
/*Desample.avsi - v1.3
---Included script functions---
DesampleX: Wrapper for doing various things involving DesampleMT, deinterlacers, resizers, and masks
Default behavior identical to DebilinearM, can also function as lineart_rpow2
DeCrossConversionMT: Replacement for ccc, ccc_720
---Requirements---
Avisynth+ r2455-MT or newer
Formats: YUV, Y
Plugins: masktools2-pfmod, ResampleMT
Plugins (DesampleX): RgTools-pfmod, Dither, nnedi3, nnedi3ocl, eedi2, eedi3
Scripts (DesampleX): edi_rpow2, ResizeX
---Parameters---
dw, dh | "Native" width & height
kernel | Desampling kernel. Takes string or integer. Mostly corresponds to the kernel strings for Dither Tools, plus "ccc" for cross-conversions
b, c, p, taps, cc | Parameters for Debicubic (b, c), Degauss (p), Delanczos/blackman/sinc (taps), and DeCrossConversion (cc)
| Note that "c" will be calculated automatically based on "b" according to b+2c=1, if not supplied (i.e. 0/0.5, 0.2/0.4, 0.333/0.333)
ow, oh | Output width & height. Default is same as dw, dh
kerneld | Post-Desample/edi kernel. Mostly like Dither
a1d, a2d, tapsd | Parameters to go with kerneld
edi | Set deinterlacer for height doubling
ratiothr | If scaling factor >= ratiothr, use edi_rpow2 to upscale, otherwise scale normally. This was copied from nnedi3_resize16 so assume this works the same
^ these parameters can be abused to do antialiasing while still downscaling.
ratiothr=0.5 will usually work right, don't set it to 0.000001 or it'll try to double the height a million or so times
kernely, kernelc | When output is resized, this controls the resizer(s) for the base clip
a1y, a2y, tapsy, a1c, a2c, tapsc | Parameters for kernely/kernelc
yuv444 | Scale chroma to 4:4:4 - this does nothing when output and input resolutions are the same. Use a boolean or set to "nnedi3" to upscale chroma with nnedi3_rpow2 (don't use nnedi3 if your input is float!)
edgemask | Enable or disable edge masking. Default: enabled only when edi_rpow2 is used
thr | Cutoff for the absolute error of descaling -> rescaling for a pixel to be considered distorted and included in protection mask. Set to 0 to disable
expand, inflate | Iterations of mt_expand/inflate. mt_expand uses the pattern both/both/square for "mode"
paramscale | masktools2 parameter, applies to "thr". Basically, mt_binarize defaults to an 8 bit scale, regardless of input depth, unless you change this
rgmode | Which removegrain mode to use to soften the line mask
blur_first | Whether to use removegrain on the line mask before or after subtracting the error mask
exmask | External edgemask string/clip. Renders rgmode and blur_first moot. Error mask is still subtracted from this clip
| i.e. a clip: exmask=last.mt_edge("hprewitt") or a string: exmask="""des.mt_edge("hprewitt")""" - desampled clip is "des", or normal clip is "src" (needs ExtractY)
show | Display mask (0 = normal output, 1 = credits mask, 2 = full mask) Mask is overlayed 50% onto the output clip: set negative values for the raw mask clip
(DesampleMask)
ow, oh | Mask "output" width & height, for masking at a different resolution
dw, dh | Descaled clip width & height. You don't usually need to set this
---Notes---
Desample can have a sharpening effect on low detail areas; grain added after upscaling, blocking, noise, etc. It may be a good idea to enable the edgemask even when not using edi_rpow2
Due to its very destructive nature, using an error mask with ccc can give a very high error, causing false positives which will be much more noticeable than usual. To deal with this, I would recommend scene-
filtering things like text overlays manually to only mask those scenes, and if possible limiting the mask to those black and white areas only, using the input clip's luma, DesampleMask with thr=0, and mt_lutxy
---Changelog---
v1.3.0 - DeCrossConversionMT actually works now
v1.2.2 - Set DeCrossConversionMT "order" default to 1 (vertical first)
v1.2.1 - Fix YV12 chroma center shift
v1.2.0 - Add nnedi3 YV12 -> YV24
v1.1.0 - Fixes and "exmask" enhancement
v1.0.0 - Initial release
*/
# TODO: saner controls over antialiasing features. maybe rfactor override
# allow setting a second, lower thr value to smooth 0 - 255 from thr1 -> thr2
Function DesampleX(clip src, int "dw", int "dh", val "kernel", float "b", float "c", float "p", int "taps", float "cc", \
int "ow", int "oh", string "kerneld", float "a1d", float "a2d", int "tapsd", string "edi", float "ratiothr", \
string "kernely", float "a1y", float "a2y", int "tapsy", string "kernelc", float "a1c", float "a2c", int "tapsc", val "yuv444", \
bool "edgemask", float "thr", int "expand", int "inflate", string "paramscale", int "rgmode", bool "blur_first", val "exmask", int "show", \
int "nsize", int "nns", int "qual", int "etype", int "pscrn", int "edi_threads", int "opt", int "fapprox", \
float "alpha", float "beta", float "gamma", int "nrad", int "mdis", bool "hp", bool "ucubic", bool "cost3", \
int "vcheck", float "vthresh0", float "vthresh1", float "vthresh2", string "sclip", string "sclip_params", clip "mclip", \
int "mthresh", int "lthresh", int "vthresh", int "estr", int "dstr", int "maxd", int "map", int "nt", int "pp", bool "mt", string "mt_params", \
int "threads", bool "logicalCores", bool "MaxPhysCore", bool "SetAffinity", bool "sleep", int "prefetch", int "range", int "accuracy", int "order"){
# Row 1: Desample parameters
# Row 2: Post-Desample/edi resizing parameters
# Row 3: Base clip resizing parameters
# Row 4: masking parameters
# Rows 5-8: edi parameters
# Row 9: More Desample parameters
assert(!src.IsRGB, "DesampleX: YUV and Y format only")
bits = src.BitsPerComponent
none = Undefined()
ratiothr = Default(ratiothr, 1.125)
iw = src.Width
w = default(dw,1280)
ow = default(ow, w)
wratio = (float(ow) / float(w))/ratiothr
rfactorX = wratio > 1 ? Round(Pow(2, Ceil( log(wratio) / log(2) ))) : 1
ih = src.Height
h = default(dh,720)
oh = default(oh, h)
hratio = (float(oh) / float(h))/ratiothr
rfactorY = hratio > 1 ? Round(Pow(2, Ceil( log(hratio) / log(2) ))) : 1
kernel = default(kernel, "bilinear")
kernel = isint(kernel) ? select(kernel, "ccc", "bilinear", "bicubic", "lanczos", "spline16", "spline36", "spline64", "blackman", "gauss", "sinc") : kernel
kerneld = default(kerneld, "spline36")
kernely = default(kernely, "spline36")
kernelc = default(kernelc, kernely)
b = default(b, 1.0/3.0)
c = default(c, (1.0-b)/2.0)
p = default(p, 30.0)
yuv444 = default(yuv444, ow%2==1 || oh%2==1)
edi = default(edi, "nnedi3")
edgemask = default(edgemask, edi!="" && (rfactorX>1 || rfactorY>1))
thr = default(thr, 10)
expand = default(expand, min(max((iw - 960)/320, (ih - 540)/180), 6))
inflate = default(inflate, max(expand, 2))
rgmode = default(rgmode, iw < 960 || ih < 720 ? 11 : 20)
blur_first = default(blur_first, true)
show = default(show,0)
usemask = Defined(exmask) || edgemask || thr>0
show!=0 ? assert(usemask, "DesampleX: Mask cannot be previewed while disabled") : NOP()
srcy = src.ExtractY()
src = iw==ow && ih==oh ? src : src.Resize(ow, oh, kernely, a1y, a2y, tapsy, kernelc, a1c, a2c, tapsc, yuv444, !usemask)
down = srcy.Desample_internal(w, h, none, none, none, none, kernel, b, c, p, taps, cc, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
errmask = thr>0 ? down.DesampleMask(srcy, ow, oh, w, h, thr, expand, inflate, kernel, b, c, p, taps, cc, paramscale) : NOP()
(rfactorX<2 && rfactorY<2) || edi=="" ? Eval("""
des = ow==w && oh==h ? down : down.Resize(ow, oh, kerneld, a1d, a2d, tapsd)
""") : Eval("""
des = bits>8 ? down.ConvertBits(8, dither=1) : down
des = des.edi_rpow2(rfactorX, rfactorY, edi, kerneld+"resize", ow, oh, tapsd, a1d, a2d, none, true, false, false, bits>8, true, none,
\nsize, nns, qual, etype, pscrn, edi_threads, opt, fapprox, alpha, beta, gamma, nrad, mdis, hp, ucubic, cost3,
\vcheck, vthresh0, vthresh1, vthresh2, sclip, sclip_params, mclip, mthresh, lthresh, vthresh, estr, dstr, maxd,
\map, nt, pp, mt, mt_params)
des = bits>8 ? des.ConvertFromStacked(16) : des
des = bits==8 || bits==16 ? des : des.ConvertBits(bits)
""")
Defined(exmask) ? Eval("""
exmask = IsString(exmask) ? Eval(exmask) : exmask
mask = thr>0 ? exmask.mt_lutxy(errmask, "x y -") : exmask
""") : edgemask ? Eval("""
lines = des.mt_edge("prewitt", 4, 24)
lines = (kernel=="bicubic" && c > 0.5) || kernel=="lanczos" ? lines.mt_expand() : lines
lines = kernel=="blackman" || kernel=="spline16" || kernel=="spline36" || kernel=="spline64" ? lines.mt_inflate() : lines
lines = kernel=="sinc" ? lines.mt_expand().mt_expand().mt_inflate().mt_inflate() : lines
lines = blur_first ? lines.removegrain(rgmode) : lines
mask = thr>0 ? blur_first || inflate>0 ? lines.mt_lutxy(errmask, "x y -") : errmask.mt_logic(lines, "andn") : lines
mask = blur_first ? mask : mask.removegrain(rgmode)
""") : thr > 0 ? Eval("""
mask = errmask
""") : NOP()
return usemask ? show>0 ? mt_merge(src, src.BlankClip(color=$35FF11), show==1?errmask.clamp():mask.clamp(), luma=true)
\ : show<0 ? show==-1 ? errmask : mask
\ : edgemask || Defined(exmask) ? mt_merge(src, des, mask)
\ : mt_merge(des.csp(src), src, mask, u=4, v=4)
\ : src.IsYUV ? CombinePlanes(des, src, planes="YUV", source_planes="YUV", sample_clip=src) : des
Function clamp(clip clip){clip.mt_clamp(clip.BlankClip(color=$808080), clip.BlankClip(), overshoot=0, undershoot=0)}
Function csp(clip clip, clip ref){return ref.IsY ? clip : ref.Is420 ? clip.converttoyuv420() : ref.Is422 ? clip.converttoyuv422() : ref.Is444 ? clip.converttoyuv444() : clip.converttoyv411()}
Function Desample_internal(clip src, int w, int h, float "src_left", float "src_top", float "src_width", float "src_height", string "kernel", float "b", float "c", float "p", int "taps", float "cc", \
int "threads", bool "logicalCores", bool "MaxPhysCore", bool "SetAffinity", bool "sleep", int "prefetch", int "range", int "accuracy", int "order"){
src
return kernel=="point" ? pointresize(w, h, src_left, src_top, src_width, src_height)
\ : kernel=="bilinear" || kernel=="linear" ? DeBilinearResizeMT(w, h, src_left, src_top, src_width, src_height, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="bicubic" || kernel=="cubic" ? DeBicubicResizeMT(w, h, b, c, src_left, src_top, src_width, src_height, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="lanczos" ? DeLanczosResizeMT(w, h, src_left, src_top, src_width, src_height, taps, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="lanczos4" ? DeLanczos4ResizeMT(w, h, src_left, src_top, src_width, src_height, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="spline16" ? DeSpline16ResizeMT(w, h, src_left, src_top, src_width, src_height, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="spline36" ? DeSpline36ResizeMT(w, h, src_left, src_top, src_width, src_height, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="spline64" ? DeSpline64ResizeMT(w, h, src_left, src_top, src_width, src_height, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="blackman" ? DeBlackmanResizeMT(w, h, src_left, src_top, src_width, src_height, taps, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="gauss" || kernel=="gaussian" ? DeGaussResizeMT(w, h, src_left, src_top, src_width, src_height, p, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="sinc" ? DeSincResizeMT(w, h, src_left, src_top, src_width, src_height, taps, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : kernel=="ccc" ? DeCrossConversionMT(w, h, cc, taps, threads, logicalCores, MaxPhysCore, SetAffinity, sleep, prefetch, range, accuracy, order)
\ : assert(false, "DesampleX: Invalid kernel setting")
}
}
# TODO: Implement option for luma-matching to catch only black/white credits
# see if abusing floats when binarizing and downscaling the mask works in Avisynth
# Add src_left/top
Function DesampleMask(clip descaled, clip input, int "ow", int "oh", float "dw", float "dh", float "thr", int "expand", int "inflate", val "kernel", float "b", float "c", float "p", int "taps", float "cc", string "paramscale"){
iw = input.Width
ih = input.Height
dw = default(dw, descaled.Width)
dh = default(dh, descaled.Height)
ow = default(ow, round(dw))
oh = default(oh, round(dh))
thr = default(thr, 10)
expand = default(expand, min(max((iw - 960)/320, (ih - 540)/180), 6))
inflate = default(inflate, max(expand, 2))
kernel = default(kernel, "bilinear")
b = default(b, 1.0/3.0)
c = default(c, (1.0-b)/2.0)
paramscale = default(paramscale, "i8")
kernel = isint(kernel) ? select(kernel, "ccc", "bilinear", "bicubic", "lanczos", "spline16", "spline36", "spline64", "blackman", "gauss", "sinc") : kernel
resc = descaled.DeDesample(iw, ih, dw, dh, kernel, b, c, p, taps, cc)
error = mt_lutxy(input, resc, "x y - abs", chroma="process")
mask = thr>0 ? error.mt_binarize(thr, chroma="process", paramscale=paramscale) : error
mask = ow==iw && oh==ih ? mask : mask.bilinearresize(ow, oh).mt_binarize(3, chroma="process")
mask = expand>0 ? mask.expand_rec(expand) : mask
mask = inflate>0 ? mask.inflate_rec(inflate) : mask
return mask
Function expand_rec(clip clp, int iter){return iter==0 ? clp : clp.mt_expand(chroma="process", mode=iter%3==0?"square":"both").expand_rec(iter-1)}
Function inflate_rec(clip clp, int iter){return iter==0 ? clp : clp.mt_inflate(chroma="process").inflate_rec(iter-1)}
Function DeDesample(clip src, int iw, int ih, float dw, float dh, string "kernel", float "b", float "c", float "p", int "taps", float "cc"){
src
return kernel=="point" ? pointresize(iw, ih, 0, 0, dw, dh)
\ : kernel=="bilinear" || kernel=="linear" ? bilinearresize(iw, ih, 0, 0, dw, dh)
\ : kernel=="bicubic" || kernel=="cubic" ? bicubicresize(iw, ih, b, c, 0, 0, dw, dh)
\ : kernel=="lanczos" ? lanczosresize(iw, ih, 0, 0, dw, dh, taps)
\ : kernel=="lanczos4" ? lanczos4resize(iw, ih, 0, 0, dw, dh)
\ : kernel=="spline16" ? spline16resize(iw, ih, 0, 0, dw, dh)
\ : kernel=="spline36" ? spline36resize(iw, ih, 0, 0, dw, dh)
\ : kernel=="spline64" ? spline64resize(iw, ih, 0, 0, dw, dh)
\ : kernel=="blackman" ? blackmanresize(iw, ih, 0, 0, dw, dh, taps)
\ : kernel=="gauss" || kernel=="gaussian" ? gaussresize(iw, ih, 0, 0, dw, dh, p)
\ : kernel=="sinc" ? sincresize(iw, ih, 0, 0, dw, dh, taps)
\ : kernel=="ccc" ? crossconversion(iw, ih, cc)
\ : assert(false, "DesampleMask: invalid kernel")
}
}
# Unlike CCC, this doesn't scale chroma normally, so you'll have to do that yourself
Function DeCrossConversionMT(clip src, int "target_width", int "target_height", float "cc", \
int "threads", bool "logicalCores", bool "MaxPhysCore", bool "SetAffinity", \
bool "sleep", int "prefetch", int "range", int "accuracy", int "order"){
target_width = default(target_width, 1280)
target_height = default(target_height, 720)
order = default(order, 1)
cc = default(cc, 0.0)
assert(target_height % 2 == 0, "DeCrossConversionMT: Height must be even")
fh = float(target_height/2)
sfac = float(target_height) / float(src.Height)
cc = -cc * sfac
sf = src.separatefields()
e = sf.selecteven().DeBilinearResizeMT(target_width, target_height/2, src_top=-cc, src_height=fh-cc, \
threads=threads, logicalCores=logicalCores, MaxPhysCore=MaxPhysCore, SetAffinity=SetAffinity, \
sleep=sleep, prefetch=prefetch, range=range, accuracy=accuracy, order=order)
o = sf.selectodd().DeBilinearResizeMT(target_width, target_height/2, src_top=cc, src_height=fh+cc, \
threads=threads, logicalCores=logicalCores, MaxPhysCore=MaxPhysCore, SetAffinity=SetAffinity, \
sleep=sleep, prefetch=prefetch, range=range, accuracy=accuracy, order=order)
return interleave(e,o).weave()
}
Function crossconversion(clip src, int w, int h, float cc){
src.separatefields()
fh = float(last.Height)
sfac = (fh * 2.) / float(h)
cc = -cc * sfac
e = selecteven().bilinearresize(w, h/2, src_top=-cc, src_height=fh-cc)
o = selectodd().bilinearresize(w, h/2, src_top=cc, src_height=fh+cc)
return interleave(e, o).weave()
}
Function Resize(clip src, int "w", int "h", string "kernely", float "a1y", float "a2y", int "tapsy", string "kernelc", float "a1c", float "a2c", int "tapsc", val "yuv444", bool "trashluma"){
bits = src.BitsPerComponent
yuv444 = Default(yuv444, false)
trashluma = Default(trashluma, false)
nnedi3_uv = IsString(yuv444)
yuv444 = nnedi3_uv ? true : yuv444
# love that trend of never adding plane parameters to core filters
src8 = nnedi3_uv && bits>8 ? bits==16 ? src.ConvertToStacked().DitherPost(mode=6, y=1)
\ : src.ConvertBits(8, dither=1)
\ : Undefined()
u_clip = nnedi3_uv && bits>8 ? src8.ExtractU() : src.ExtractU()
v_clip = nnedi3_uv && bits>8 ? src8.ExtractV() : src.ExtractV()
yuv = src.IsYUV
hss = src.IsYV411 ? 4 : src.Is420 || src.Is422 ? 2 : 1
vss = src.Is420 ? 2 : 1
cw = yuv444 ? w : w/hss
ch = yuv444 ? h : h/vss
sx = hss>1 ? yuv444 ? 0.25 : calign(src.Width, w, hss) : 0
return trashluma ? yuv ? CombinePlanes(u_clip.Resize_internal_uv(cw, ch, sx, kernelc, a1c, a2c, tapsc, nnedi3_uv, bits),
\v_clip.Resize_internal_uv(cw, ch, sx, kernelc, a1c, a2c, tapsc, nnedi3_uv, bits),
\planes="UV", source_planes="YY", sample_clip=src)
\ : Undefined()
\ : !yuv || hss==1 ? src.Resize_internal(w, h, 0, 0, kernely, a1y, a2y, tapsy)
\ : CombinePlanes(src.ExtractY().Resize_internal(w, h, 0, 0, kernely, a1y, a2y, tapsy),
\u_clip.Resize_internal_uv(cw, ch, sx, kernelc, a1c, a2c, tapsc, nnedi3_uv, bits),
\v_clip.Resize_internal_uv(cw, ch, sx, kernelc, a1c, a2c, tapsc, nnedi3_uv, bits),
\planes="YUV", source_planes="YYY", sample_clip=src)
Function calign(float in, float out, int hss){return hss==2 ? 0.25*(1.0-in/out) : 0.375*(1.0-in/out)}
Function Resize_internal_uv(clip src, int w, int h, float sx, string "kernel", float "a1", float "a2", int "taps", bool "usennedi3", int "bits"){
src
usennedi3 ? nnedi3_rpow2(2) : last
usennedi3 && bits>8 ? ConvertBits(bits) : last
Resize_internal(w, h, usennedi3 ? 0 : sx, usennedi3 ? -0.5 : 0, kernel, a1, a2, taps)
}
Function Resize_internal(clip src, int w, int h, float sx, float sy, string "kernel", float "a1", float "a2", int "taps"){
src
return kernel=="point" ? pointresize(w, h, sx, sy)
\ : kernel=="bilinear" || kernel=="linear" ? bilinearresize(w, h, sx, sy)
\ : kernel=="bicubic" || kernel=="cubic" ? bicubicresize(w, h, a1, a2, sx, sy)
\ : kernel=="lanczos" ? lanczosresize(w, h, sx, sy, taps=taps)
\ : kernel=="lanczos4" ? lanczos4resize(w, h, sx, sy)
\ : kernel=="spline16" ? spline16resize(w, h, sx, sy)
\ : kernel=="spline36" ? spline36resize(w, h, sx, sy)
\ : kernel=="spline64" ? spline64resize(w, h, sx, sy)
\ : kernel=="blackman" ? blackmanresize(w, h, sx, sy, taps=taps)
\ : kernel=="gauss" || kernel=="gaussian" ? gaussresize(w, h, sx, sy, p=a1)
\ : kernel=="sinc" ? sincresize(w, h, sx, sy, taps=taps)
\ : assert(false, "DesampleX: invalid kernely or kernelc")
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment