Skip to content

Instantly share code, notes, and snippets.

@UnaiM

UnaiM/the-magic-of-kronos.md

Created November 7, 2023 09:20
Show Gist options
  • Save UnaiM/2056fae6429ab6328751f7cbbd1590a5 to your computer and use it in GitHub Desktop.
Save UnaiM/2056fae6429ab6328751f7cbbd1590a5 to your computer and use it in GitHub Desktop.
Download ZIP
Why my rolling shutter fix didn’t match with RollingShutter
Raw
the-magic-of-kronos.md

The magic of Kronos

(AKA Why My Rolling Shutter Fix Didn’t Match with RollingShutter) – 20th December 2014
WARNING: doesn’t come to any solution.

My last post was about trying to recreate The Foundry’s RollingShutter plugin, using the new Kronos 2 technology instead of the old one, which the discontinued plugin was left with. And while the proposed solution benefitted from Regularized motion estimation, when switching back to Local it was clear that RollingShutter still won over it. So there’s something missing that I didn’t recreate, and have yet to figure out.

The first thing that came to my mind is that maybe RollingShutter uses a blend between two warped frames instead of just one. But since Kronos is the father of this technology, and it’s way clearer to see how it works, I made a blend with it, and tried to mimic its effect with IDistorts and a Dissolve.

Nuke node graph trying to mimic Kronos’s result with a blend between two IDistorted frames.

Nuke copy/paste 
set cut_paste_input [stack 0]
version 9.0 v1
BackdropNode {
 inputs 0
 name BackdropNode1
 label "Has a user knob\nthat drives other nodes.\nGoes from 0 at frame 1\nto 1 at frame 11."
 note_font_size 13
 selected true
 xpos -639
 ypos -298
 bdwidth 134
 bdheight 133
}
BackdropNode {
 inputs 0
 name BackdropNode2
 label "Disable it to see how both\nmethods fade between frames."
 note_font_size 13
 selected true
 xpos -447
 ypos -230
 bdwidth 190
 bdheight 116
}
BackdropNode {
 inputs 0
 name BackdropNode3
 label "Warp frame 2\nwith frame 3’s\nbackward vectors."
 note_font_size 13
 selected true
 xpos -482
 ypos -47
 bdwidth 117
 bdheight 140
}
BackdropNode {
 inputs 0
 name BackdropNode4
 label "Warp frame 3\nwith frame 2’s\nforward vectors."
 note_font_size 13
 selected true
 xpos -339
 ypos -47
 bdwidth 117
 bdheight 140
}
BackdropNode {
 inputs 0
 name BackdropNode5
 label "Source and FgVecs\nconnected to make\nsure it uses same\nvectors as method 2."
 note_font_size 13
 selected true
 xpos -637
 ypos 20
 bdwidth 130
 bdheight 147
}
Read {
 inputs 0
 file seq/#.jpg
 format "1920 1080 0 0 1920 1080 1 HD_1080"
 last 4
 origlast 4
 origset true
 name Read1
 selected true
 xpos -391
 ypos -278
 postage_stamp false
}
VectorGenerator {
 motionEstimation Regularized
 name VectorGenerator1
 selected true
 xpos -391
 ypos -155
}
set N5d27d830 [stack 0]
FrameHold {
 first_frame 2
 name FrameHold1
 selected true
 xpos -464
 ypos -89
}
set Na272400 [stack 0]
push $N5d27d830
FrameHold {
 first_frame 3
 name FrameHold2
 selected true
 xpos -317
 ypos -89
}
set N4c82dc00 [stack 0]
clone node15ccca4e0|ShuffleCopy|36687 ShuffleCopy {
 inputs 2
 in motion
 in2 none
 red red
 green green
 blue blue
 out motion
 name ShuffleCopy2
 selected true
 xpos -317
 ypos 34
}
set C5ccca4e0 [stack 0]
IDistort {
 channels rgb
 uv forward
 uv_scale {{1-parent.NoOp1.mix}}
 name IDistort2
 selected true
 xpos -317
 ypos 58
}
push $N4c82dc00
push $Na272400
clone $C5ccca4e0 {
 inputs 2
 xpos -464
 ypos 34
 selected true
}
IDistort {
 channels rgb
 uv backward
 uv_scale {{parent.NoOp1.mix}}
 name IDistort1
 selected true
 xpos -464
 ypos 58
}
Dissolve {
 inputs 2
 which {{parent.NoOp1.mix}}
 name Dissolve1
 selected true
 xpos -395
 ypos 123
}
push $cut_paste_input
NoOp {
 name NoOp1
 selected true
 xpos -614
 ypos -202
 addUserKnob {20 User}
 addUserKnob {7 mix}
 mix {{clamp((t-1)/10)}}
}
push $N5d27d830
Dot {
 name Dot2
 selected true
 xpos -580
 ypos -148
}
set N5d2d7760 [stack 0]
push 0
push 0
push $N5d2d7760
Kronos {
 inputs 4
 input.last 4
 retimedChannels rgb
 timing2 Frame
 timingOutputSpeed 0.2
 timingFrame2 {{parent.NoOp1.mix+2}}
 showLegacyMode false
 motionEstimation Regularized
 legacyModeNuke9 false
 name Kronos1
 selected true
 xpos -614
 ypos 125
}

Download sequence

It’s clearly different. I also tried blacking out frames 1 and 4 from the source, to confirm that Kronos doesn’t use any data from other frames than the two it’s blending between. By disabling VectorGenerator, that is, blacking the motion vectors out, we can see that Kronos’ result is a simple dissolve, just the same as what I used, so it doesn’t do any fancy stuff once the frames are warped. The problem is in the IDistorts then.

The next thing I thought about was that there could be some cross-vector math involved before warping, maybe a combination of forward and backward vectors, or even from different frames… So I tried looking at just the first frame getting warped, by blacking out everything else. This gave me a fading to black that I undid with a Multiply and an expression. After that, I could black all the motion vectors except the backwards ones of the second frame, and guess what, the results were still the same.

Nuke node graph trying to recrate what Kronos does to a single source frame/set of vectors, using IDistort.

Nuke copy/paste 
set cut_paste_input [stack 0]
version 9.0 v1
BackdropNode {
 inputs 0
 name BackdropNode1
 label "Source: everything\nblacked out except\nframe 2."
 note_font_size 13
 selected true
 xpos -621
 ypos 28
 bdwidth 117
 bdheight 112
}
BackdropNode {
 inputs 0
 name BackdropNode2
 label "FgVecs: everything\nblacked out except\nframe 3."
 note_font_size 13
 selected true
 xpos -488
 ypos 28
 bdwidth 117
 bdheight 112
}
BackdropNode {
 inputs 0
 name BackdropNode3
 label "Black out\nforward vectors."
 note_font_size 13
 selected true
 xpos -401
 ypos -107
 bdheight 108
}
BackdropNode {
 inputs 0
 name BackdropNode4
 label "Divide colour\nto undo fading."
 note_font_size 13
 selected true
 xpos -559
 ypos 223
 bdwidth 112
 bdheight 103
}
BackdropNode {
 inputs 0
 name BackdropNode5
 label "Frame 2 warped\nby frame 3’s\nbackward vectors."
 note_font_size 13
 selected true
 xpos -315
 ypos 185
 bdwidth 117
 bdheight 139
}
BackdropNode {
 inputs 0
 name BackdropNode6
 label "0 = live action example\n1 = exaggerated example"
 note_font_size 13
 selected true
 xpos -430
 ypos -244
 bdwidth 158
 bdheight 103
}
push $cut_paste_input
NoOp {
 name NoOp1
 selected true
 xpos -419
 ypos 240
 addUserKnob {20 User}
 addUserKnob {7 mix}
 mix {{clamp((t-1)/10)}}
}
Read {
 inputs 0
 file seq2/#.exr
 format "640 480 0 0 640 480 1 PC_Video"
 first 2
 last 3
 origfirst 2
 origlast 3
 origset true
 name Read2
 selected true
 xpos -244
 ypos -185
 postage_stamp false
}
Read {
 inputs 0
 file seq/#.jpg
 format "1920 1080 0 0 1920 1080 1 HD_1080"
 last 4
 origlast 4
 origset true
 name Read1
 selected true
 xpos -540
 ypos -244
 postage_stamp false
}
VectorGenerator {
 motionEstimation Regularized
 name VectorGenerator1
 selected true
 xpos -540
 ypos -185
}
Switch {
 inputs 2
 name Switch1
 label "\[value which]"
 selected true
 xpos -391
 ypos -185
}
Shuffle {
 in none
 out forward
 name Shuffle1
 selected true
 xpos -391
 ypos -27
}
set N1e1cda30 [stack 0]
Dot {
 name Dot1
 selected true
 xpos -501
 ypos -24
}
set N1e1e9d10 [stack 0]
Multiply {
 channels backward
 value {{t==3}}
 name Multiply2
 selected true
 xpos -470
 ypos 97
}
push 0
push 0
push $N1e1e9d10
Multiply {
 channels rgb
 value {{t==2}}
 name Multiply1
 selected true
 xpos -606
 ypos 101
}
Kronos {
 inputs 4
 input.last 4
 retimedChannels rgb
 timing2 Frame
 timingOutputSpeed 0.2
 timingFrame2 {{parent.NoOp1.mix+2}}
 showLegacyMode false
 motionEstimation Regularized
 legacyModeNuke9 false
 name Kronos1
 selected true
 xpos -541
 ypos 176
}
Multiply {
 channels rgb
 value {{1/(1-parent.NoOp1.mix)}}
 name Multiply3
 selected true
 xpos -541
 ypos 287
}
push $N1e1cda30
Dot {
 name Dot2
 selected true
 xpos -259
 ypos -24
}
set N1e1d4010 [stack 0]
FrameHold {
 first_frame 3
 name FrameHold2
 selected true
 xpos -248
 ypos 108
}
push $N1e1d4010
FrameHold {
 first_frame 2
 name FrameHold1
 selected true
 xpos -339
 ypos 109
}
ShuffleCopy {
 inputs 2
 in motion
 in2 none
 red red
 green green
 blue blue
 out motion
 name ShuffleCopy2
 selected true
 xpos -297
 ypos 263
}
IDistort {
 channels rgb
 uv backward
 uv_scale {{parent.NoOp1.mix}}
 name IDistort1
 selected true
 xpos -297
 ypos 287
}

So now we know Kronos only uses the backwards vectors of frame 3 to warp frame 2, just like we did with IDistort. And the same happens with frame 3 and the forward vectors of frame 2. Yet the results between both methods are strikingly different, with Kronos giving smoother warps than IDistort, while both using exactly the same input images and vectors.

If you switch to the exaggerated example, you can see the smearing it produces as if the vectors themselves were moving into place as well, which is impossible with IDistort (or at least with it alone). The answer is clear: Kronos doesn’t just use IDistort’s technology to warp images. Although I can’t prove it, I do believe RollingShutter uses this same mysterious technology.

Great. What now? Well, it comes to the same thing we thought about in the last post: if only we could change Kronos’ Frame knob along the height of the image… Unfortunately, this parameter isn’t just a multiplier like IDistort’s UV Scale, and you can clearly see this by trying to multiply the vectors before Kronos.

Nuke node graph showing the difference between tweaking Kronos’s Frame knob and multiplying motion vectors.

Nuke copy/paste 
set cut_paste_input [stack 0]
version 9.0 v1
BackdropNode {
 inputs 0
 name BackdropNode1
 tile_color 0x88aa88ff
 label "<left>DRAG MY SLIDER\nCan’t be animated this time, because the 2nd Kronos is static and it would freeze the effect."
 note_font_size 13
 selected true
 xpos -669
 ypos -88
 bdwidth 139
 bdheight 131
}
BackdropNode {
 inputs 0
 name BackdropNode2
 label "REGULAR KRONOS\nSame as before."
 note_font_size 13
 selected true
 xpos -774
 ypos 226
 bdwidth 133
 bdheight 146
}
BackdropNode {
 inputs 0
 name BackdropNode3
 label "<left>IDISTORT\nSame as before, with BlackOutside because Kronos also does it internally."
 note_font_size 13
 selected true
 xpos -292
 ypos 229
 bdwidth 178
 bdheight 142
}
BackdropNode {
 inputs 0
 name BackdropNode4
 label "<left>ANIMATED VECTOR INTENSITY, STATIC KRONOS\nMostly like IDistort.\nThe 0.001 shift is so Kronos still reads frame 2, and not just frame 3 (black)."
 note_font_size 13
 selected true
 xpos -616
 ypos 159
 bdwidth 299
 bdheight 212
}
push $cut_paste_input
NoOp {
 name NoOp1
 selected true
 xpos -641
 ypos 8
 addUserKnob {20 User}
 addUserKnob {7 mix}
 mix 0.5
}
Read {
 inputs 0
 file seq2/#.exr
 format "640 480 0 0 640 480 1 PC_Video"
 first 2
 last 3
 origfirst 2
 origlast 3
 origset true
 name Read2
 selected true
 xpos -282
 ypos 19
 postage_stamp false
}
Read {
 inputs 0
 file seq/#.jpg
 format "1920 1080 0 0 1920 1080 1 HD_1080"
 last 4
 origlast 4
 origset true
 name Read1
 selected true
 xpos -502
 ypos -40
 postage_stamp false
}
VectorGenerator {
 motionEstimation Regularized
 name VectorGenerator1
 selected true
 xpos -502
 ypos 19
}
Switch {
 inputs 2
 name Switch1
 label "\[value which]"
 selected true
 xpos -392
 ypos 19
}
Shuffle {
 in none
 out forward
 name Shuffle1
 selected true
 xpos -392
 ypos 64
}
set N577ed510 [stack 0]
Dot {
 name Dot1
 selected true
 xpos -607
 ypos 67
}
set N24d1ac20 [stack 0]
Multiply {
 channels backward
 value {{t==3}}
 name Multiply2
 selected true
 xpos -510
 ypos 113
}
set N24d1d170 [stack 0]
push 0
push 0
push $N24d1ac20
Multiply {
 channels rgb
 value {{t==2}}
 name Multiply1
 selected true
 xpos -749
 ypos 166
}
set N24d221f0 [stack 0]
Kronos {
 inputs 4
 input.last 4
 retimedChannels rgb
 timing2 Frame
 timingOutputSpeed 0.2
 timingFrame2 {{parent.NoOp1.mix+2}}
 showLegacyMode false
 motionEstimation Regularized
 legacyModeNuke9 false
 name Kronos1
 selected true
 xpos -749
 ypos 305
}
Multiply {
 channels rgb
 value {{1/(1-parent.NoOp1.mix)}}
 name Multiply3
 selected true
 xpos -749
 ypos 337
}
push $N24d1d170
Multiply {
 channels backward
 value {{parent.NoOp1.mix/0.999}}
 name Multiply5
 selected true
 xpos -510
 ypos 239
}
push 0
push 0
push $N24d221f0
Kronos {
 inputs 4
 input.last 4
 retimedChannels rgb
 timing2 Frame
 timingOutputSpeed 0.2
 timingFrame2 2.999
 showLegacyMode false
 motionEstimation Regularized
 legacyModeNuke9 false
 name Kronos2
 selected true
 xpos -510
 ypos 303
}
Multiply {
 channels rgb
 value 1000
 name Multiply4
 selected true
 xpos -510
 ypos 335
}
push $N577ed510
Dot {
 name Dot2
 selected true
 xpos -207
 ypos 67
}
set N577f41f0 [stack 0]
FrameHold {
 first_frame 3
 name FrameHold2
 selected true
 xpos -196
 ypos 122
}
push $N577f41f0
FrameHold {
 first_frame 2
 name FrameHold1
 selected true
 xpos -287
 ypos 123
}
ShuffleCopy {
 inputs 2
 in motion
 in2 none
 red red
 green green
 blue blue
 out motion
 name ShuffleCopy2
 selected true
 xpos -242
 ypos 191
}
BlackOutside {
 name BlackOutside1
 selected true
 xpos -242
 ypos 314
}
IDistort {
 channels rgb
 uv backward
 uv_scale {{parent.NoOp1.mix}}
 name IDistort1
 selected true
 xpos -242
 ypos 338
}

Basically we’re losing all the Kronosness when we multiply the vector, instead of using its Frame knob. We’re converting it into a regular IDistort. What’s the mystery behind that Frame knob then? I don’t know. Sorry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment