greed2411

import torch
import torchvision
from torchvision import datasets, transforms, models
import torch.nn as nn
from torch.autograd import Variable
from PIL import Image
import numpy as np

# >>> torch.__version__
# '0.2.0_4'

greed2411

import base64, uuid

def cookie_secret():
    return base64.b64encode(uuid.uuid4().bytes + uuid.uuid4().bytes)

if __name__ == "__main__":
    print(base64.b64encode(uuid.uuid4().bytes + uuid.uuid4().bytes))

greed2411

import os
import img2pdf

# takes all the images in the current working directory and puts as one centered image per page
with open("output.pdf", "wb") as f:
    f.write(img2pdf.convert([i for i in os.listdir(os.getcwd()) if i.endswith(".jpg")]))

greed2411

# onehot2labels()

def onehot2labels(y):
  labels = []
  for row_element in y:
    labels.append(np.argmax(row_element))
  return np.array(labels)

greed2411

import zipfile
zip_ref = zipfile.ZipFile(path_to_zip_file, 'r')
zip_ref.extractall(directory_to_extract_to)
zip_ref.close()

greed2411

greed2411

from vibora import Vibora, Response

app = Vibora()

@app.route('/')
async def index():
    return Response(b'', headers={'content-type': 'html'})

@app.route("/user/<int:id>", methods=['GET'])
async def user_info(id):

greed2411

def dice_loss(input, target):
    smooth = 1.

    iflat = input.view(-1)
    tflat = target.view(-1)
    intersection = (iflat * tflat).sum()
    
    return 1 - ((2. * intersection + smooth) /
              (iflat.sum() + tflat.sum() + smooth))

greed2411

from starlette.applications import Starlette
from starlette.responses import PlainTextResponse
import uvicorn

app = Starlette()


@app.route('/')
async def homepage(request):
    return PlainTextResponse('')

greed2411

A note on implementing community detection using Apache Spark + GraphX

Girvan Newman Algorithm

This is one of the earliest methods of community detection. This method is simple to understand and can be easily distributed across clusters for faster processing. Key assumption is that the graph is undirected and unweighted. But it is not hard to extend to directed graphs + weighted edges.

The algorithm is fairly straightforward. It defines a new measure called edge betweenness centrality based on which a divisive hierarchical clustering algorithm is designed to find communities. The stopping criteria for this uses a popular metric called modularity which quantifies how cohesive the communities are during the clustering process.

Side note: A bit of search reveled no implementation of this algorithm in a distributed way (mainly because its slow and better algorithms are available?). So, this note would pave way to use this naive algorithm inspite of its high time complexity.