Implementation of a Smudge tool in Pythonista, tutorial-ish, and sensitive to the Apple Pencil force by setting applePencil=True

PythoniSmudge.py

import ui
import math
import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
import Image
import ImageOps
import ImageChops
import io
import scene
from objc_util import ObjCInstance
import time


#WARNING: DON'T FORGET TO MANUALLY STOP THE PROGRAM AFTER CLOSING THE UI WINDOW, BY PRESSING THE CROSS ON THE TOP RIGHT CORNER OF THE CODE EDITOR, OTHERWISE THE CODE WILL KEEP RUNNING FOREVER


'''
This is a Pythonista implementation of a Smudge tool.

The way it works:
	
	The actual computation of the image happens at the very end of the code, in the while(True) infinite loop. For this computation to be fast, I used the GPU accelerated numpy library allowing to manipulate/add/multiply big vectors elementwise very fast.
	
	The Smudge tool can be seen as infintely iterative operation on the image consisting in mixing a region of the picture with another one. More precisely, when a smudge tool is stroking the image, at any time the circular region it was before is mixed with the region it is now.
	
	 A slightly more complex, but a lot more beautiful, smudge tool will actually mix something with the current region of the picture that depends more on the whole strokes than just the previous region (think of a piece of a tissue you stroke on a drawing, the tissue will not only move the colors it passes on, it will actually also absorb them, get dirty, and "remember" them). This is implemented with another vector, called "stamp", alsoin the end of the code.
	
	The only data the algorithm in the while(true) loop needs is the actual strokes. These are made by the user on the screen, so I used a ui.View to get the touch_began, touch_moved, and touch_ended data.
	
	This was really all was needed to compute the smudge result.
	
	Then I needed to display the result image on the screen so that the user can see the image changing as is finger smudges on the screen.
	
	The challenge here was to display the image in real time.
	
	The first simple idea was to convert the numpy array into an ui.Image, that can be displayed in real time by an ui.ImageView. But the conversion could not happen in perfect real time.
	
	A solution that was used was to redraw only the portion around the cursor in real time, using a separate view on top of the picture, and then only when the finger releases the screen, redraw the whole picture behind.
	
	It worked well except when the finger left the area around the cursor with a long stroke. So I had to move the separate view when the finger was going to far away from its initial position, but I had to update the big imageView before that, otherwise the separate view would make the unchanged image reappear as it leaves its initial position to follow the cursor.
	
	Because of these regular updates of the big image, There was some time glitches during long strokes.
	
	To prevent that I used multiple seperate views (I used 8 of them), so instead of having the separate view moving with the cursor, the 8 view relay themselves to follow the cursor without leaving their positions nor revealing the unchanged image behind them.
'''

'''

	To see how it works, you can see the 8 views by setting debug=True

'''
debug=False

'''
	if you own an Apple Pencil, the smudge will be sensitive to the force by setting applePencil=True
'''
applePencil=False

'''
This function just convert a PIL image to an ui.Image that can be displayed in real time. It is only used because when  I convert the numpy array representing an image to an ui.Image, I intermediately convert it to a PIL image
''' 
def pil2ui(imgIn, format):
    with io.BytesIO() as bIO:
        imgIn.save(bIO, format)
        imgOut = ui.Image.from_data(bIO.getvalue())
    del bIO
    return imgOut


'''
The View of the circular cursor and the 8 relaying miniviews will all be siblings children of a main view. 

'''




'''
This is the most important class: Not only it displays the circular cursor, but it also controls every other views (the 8 miniviews and the big ImageView). Notice that these view aren't "added" to it, so they are not subviews of it (actually they will be its siblings within the GlobalView), but this class stills controls them. As it displays the cursor, it is naturally in front of every other sibling views and is the only view actually receiving the touch data of the finger/Apple Pencil. It displays a blue circular cursor on the screen at the finger's position. It also triggers the relaying between each views when the cursor gets too far, and the update of the big image behind.
'''
class CursorView (ui.View):
	#The CursorView will be instanced by the GlobalView which will pass the frame as a parameter.
	def __init__(self, frame):
		self.frame = frame
		self.flex = 'WH'
		#Access the dimensions of this view
		width=self.frame[2]
		height=self.frame[3]
		
		self.N=0
		self.t=time.perf_counter()
		
		#Define the numpy image array, fill with zeros (empty canva)
		self.imageArray=np.ones((height,width), dtype=np.float32)
		#Draw a black disk at the center of the image:
		for i in range(int(width)):
			for j in range(int(height)):
				self.imageArray[j,i]= int(255*(1.0-(math.sqrt((i-width/2)*(i-width/2)+(j-height/2)*(j-height/2))<0.1*height/2)))
		
		#Definition of the big image view
		self.image_view=ui.ImageView(frame=(0,0,width,height))
		#Initializing the big image by converting the numpy image array stored in the cursorView to an ui.Image
		self.image_view.image=pil2ui(ImageOps.grayscale(Image.fromarray(self.imageArray)),'PNG')
		#Remark: self.image_view isn't actually added to the CursorView, it is only a standard children of it. self.image_view will be added to the GlobalView, as will the CursorView, so they will actually be siblings.
		
		#Definition of the miniviews
		mnv=[]
		#Number of miniviews
		self.viewsNumber=8
		for i in range(self.viewsNumber):
			mnv.append(MiniView(parentCursorView= self))
		self.miniViews=mnv
		#Same remark as above, for self.miniViews
		
		#Initialize the cursor at the center of the canvas:
		self.cursor=scene.Vector2(self.frame[2]/2,self.frame[3]/2)
		
		#Parameters:
			
		#Force of the smudge, fixed if no Apple Pencil
		self.force=0.5
		#Radius of the smudge brush
		self.cursorRadius=min(width,height)/16
		#How much the miniviews will be bigger than the cursor
		self.miniViewsSideFactor=2
		
		
		#isSmudging will be true only after the cursor/brush has started moving:
		self.isSmudging=False
		#Useful to know when to relay the miniviews by comparing the current position of the cursor to the current miniview center
		self.currentMiniViewCenter=self.cursor
		#Used to have the smudge apply on a very refined and continuous path rather than the blocky and angular touch_moved path one might have because of a low touch_moved rate (for this reason, the continuous follower is updated in the while(true) loop at the end of the code) 
		self.cursorContinuousFollower=self.cursor
		#Will keep track of the index of the current miniView while they relay each other. Loops around {0,1,2,...,7}
		self.currentMiniViewIndex=0
		
		self.chrono=0
		
	def touch_began(self, touch):
		self.chrono=0
		
		self.N=0
		self.t=time.perf_counter()
		#The smudging really begins after the cursor has moved (otherwise, the last position of the cursor would be stamped on the new one)
		self.isSmudging=False
		#The circle of the cursor has to go to the finger even if it doesn't move
		self.cursor = touch.location
		#This is the only place the continuous follower is affected outside of the while(true) loop at the end of the code. Because we don't want the cursor to sweep from its last location to the new one here. We want an actual jump. So we need to force this discontinuous behavior.
		self.cursorContinuousFollower = touch.location
		#Resets the center of the current miniview
		self.currentMiniViewCenter=self.cursor
		#Reset the index
		self.currentMiniViewIndex=0
		#Redraw the current miniview at the new cursor's location
		self.miniViews[0].set_needs_display()
		#bring the miniview on top of everything
		self.miniViews[0].bring_to_front()
		#bring the cursor on top of it
		self.bring_to_front()
		self.set_needs_display()
		
		
	def touch_moved(self, touch):
		self.chrono+=1
		#Getting force sensitivity with the Apple Pencil:
		if applePencil:
			ui_touch = ObjCInstance(touch)
			self.force = ui_touch.force() /2.0
		self.cursor = touch.location
		
		#We compare the cursor's position (actually we use the continuous follower because he is the one truly smudging here, the cursor is just for displaying) to the center of the current miniview check if their distance is too large, implying that the cursor is going to step out of the current miniView, so a new miniview has to take the relay
		if abs(self.cursorContinuousFollower- self.currentMiniViewCenter)>1*(self.miniViewsSideFactor-1)*self.cursorRadius:
			#The new miniView center will not exactly be on the cursor (to avoid holes between miniviews when doing a long fast stroke)
			self.currentMiniViewCenter+=(self.cursorContinuousFollower-self.currentMiniViewCenter)*1
			
			#Here we check if we have looped over all the miniviews, and if so we need to update the big image:
			if self.currentMiniViewIndex==self.viewsNumber-1:
				#upadting the big image (JPEG is faster):
				self.image_view.image=pil2ui(ImageOps.grayscale(Image.fromarray(self.imageArray)), 'JPEG')
				#Puts the big image on top of the miniviews in case they didn't do the job right (latency, gap between miniviews, etc)
				self.image_view.bring_to_front()
				
			#Updating the index to get the next miniview:
			self.currentMiniViewIndex=(self.currentMiniViewIndex+1)%self.viewsNumber
			#Putting the new miniview on top of the others.
			self.miniViews[self.currentMiniViewIndex].bring_to_front()
			#Putting the cursor view on top of it
			self.bring_to_front()
			self.miniViews[self.currentMiniViewIndex].set_needs_display()
			#self.set_needs_display()
		else:					
			#We ask the current miniview to display itself (it just converts and draw the portion of the image array around its center position in one line in its draw method)
			if self.chrono%3==0:
				self.miniViews[self.currentMiniViewIndex].set_needs_display()
		self.set_needs_display()
		#Makes sure that we have waited after the cursor has start moving to start smudging (used at the end of the code) 
		self.isSmudging=True
		#display the cursor (blue circle)
		
		
		
		
	def touch_ended(self, touch):
		#print('fps:',self.N/(time.perf_counter()-self.t))
		self.isSmudging=False
		#Update of the big image after releasing the screen (so no need for real time here)
		self.image_view.image=pil2ui(ImageOps.grayscale(Image.fromarray(self.imageArray)), 'PNG')
		#Puts the big image on top of the miniviews in case they didn't do the job right (latency, gap between miniviews, etc)
		self.image_view.bring_to_front()
		#cursor view back on top
		self.bring_to_front()
		self.set_needs_display()
		
		
	def draw(self):
		r=self.cursorRadius
		#draw a red cursor/circle for the follower (debug mode):
		if debug:
			x,y=self.cursorContinuousFollower
			ui.set_color('red')
			ui.Path.oval(x-r, y-r, 2*r,2*r).stroke()
		#draw a blue cursor/circle for the actual cursor:
		x,y=self.cursor
		ui.set_color('blue')
		ui.Path.oval(x-r, y-r, 2*r,2*r).stroke()
		

#This is the class for the miniviews
class MiniView (ui.View):
	def __init__(self, parentCursorView):
		#Access the cursorView, mainly to access the image array inside it
		self.parentCursorView=parentCursorView
		self.width= parentCursorView.width
		self.height=parentCursorView.height
		
		
	def draw(self):
		#define the actual size of the miniview (the size of the region of the image array we are going to sample)
		r=self.parentCursorView.miniViewsSideFactor*self.parentCursorView.cursorRadius
		#define the center of this region
		x,y=self.parentCursorView.currentMiniViewCenter
		#sample the region of the array and display it (JEPG is faster)
		pil2ui(ImageOps.grayscale(Image.fromarray(self.parentCursorView.imageArray[y-r: y+r, x-r: x+r])),'JPEG').draw(x-r,y-r,2*r,2*r)
		
		self.parentCursorView.N+=1
		#For debugging, display the borders of the miniView:
		if debug:
			ui.Path.rect(x-r,y-r,2*r,2*r).stroke()


#The only point of the GlobalView is to have the other views as subviews.
class GlobalView (ui.View):
	def __init__(self, width=1024, height=1024):
		self.width=width
		self.height=height
		
		#Definition of the cursor view (thus creating a numpy image array, an imageView and 8 miniViews inside it)
		self.cursor_view=CursorView(frame=(0,0,width,height))
		cv=self.cursor_view
		
		#Adding all the views as subsviews:
		self.add_subview(cv.image_view)
		for i in range(cv.viewsNumber):
			#Add the miniviews in reverse order so that the top one is the miniViews[0]
			self.add_subview(cv.miniViews[7-i])
		self.add_subview(cv)
			
	
		
#Actual beginning of the code's instructions:
w, h = ui.get_screen_size()
#Creating a GlobalView (hence, a cursor view, hence an image view and 8 miniviews inside of it, by definition)
mv= GlobalView(w,h)
mv.present('fullscreen')
#Getting the cursor view inside the global view:
cv=mv.cursor_view
#Dereferencing the radius of the brush:
r=cv.cursorRadius


#Creating the mask that will be used for smudging (to smooth out the borders of the smudge instead of having hard edges caused by hard square smudge regions)
#The mask is initialize as an array of zeros
mask=np.zeros((2*r,2*r),dtype=np.float32)
#Center of the mask:
c= scene.Vector2(r,r)
for i in range(int(2*r)):
			for j in range(int(2*r)):
				#Vector representing the (i,j) position on the mask
				v=scene.Vector2(i,j)
				#d is the distance of (i,j) to the center of the mask, relative to the cursor's radius r, and clamped to not be one (to avoid a division by zero in the next line)
				d=min(abs(v-c)/r,0.9999)
				#Famous bump function in mathematics, the mask is thus filled with values depending on the distance of (i,j) to the center in a decreasing way, from 1 to 0, and with a zero derivative on the edges and at the center (so that we have a nice bump mask!)
				mask[j,i]= math.exp(1-1/(1-d*d))
				
				
#Initialization of the stamp (clean at the beginning)
stamp = np.zeros((2*r,2*r),dtype=np.float32)
#A clean is actually not ideal as explained below.
stampInit=False
#This is a parameter controlling how quickly the continuous floolower follows the cursor
attraction=0.1

#Infinite smudge loop:
while (mv.on_screen):
	if cv.isSmudging:
		#Here we compute the next position of the continuous follower and also store it in x,y to use it for the smudge computation below
		x,y= cv.cursorContinuousFollower*(1-attraction) +cv.cursor*attraction
		#To avoid issues when the cursor crosses the border of the screen, we clamp it to the screen
		x=int(max(min(x, w-2*r),2*r))
		y=int(max(min(y, h-2*r),2*r))
		#Updating the new positon of the continuous follower
		cv.cursorContinuousFollower=scene.Vector2(x,y)
		
		#Handles the initialization of the stamp:
		if stampInit==False:
			#That is a bit tricky: When we start smudging again, the stamp shouldn't be clean, because if it is, it will "erase" the drawing. Ideally, it should be identical to the region it is set on: 
			
			stamp=1*cv.imageArray[y-r:y+r, x-r:x+r]
			stampInit=True
			
		#The actual smudge computation!!! mixing the stamp with the region around the continuous follower with a 0.1 factor
		cv.imageArray[y-r:y+r, x-r:x+r]+=(stamp-cv.imageArray[y-r:y+r, x-r:x+r ])*mask*cv.force*0.1
		#Updation the stamp to get dirty by the new region:
		stamp+=0.1*(cv.imageArray[y-r:y+r, x-r:x+r ]-stamp)

	else:	
		#Deinitialize the stamp for the next time:
		stampInit=False