Skip to content

Instantly share code, notes, and snippets.

@saketkc

saketkc/Pix-Plot-Demo.ipynb

Created July 5, 2018 14:42
Show Gist options
  • Save saketkc/708d8f03b8d42090fd3dd4a8ff8feba5 to your computer and use it in GitHub Desktop.
Save saketkc/708d8f03b8d42090fd3dd4a8ff8feba5 to your computer and use it in GitHub Desktop.
Download ZIP
Demo of Pix-plot with progress bars
Display the source blob
Display the rendered blob
Raw
Pix-Plot-Demo.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Before ..."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" * writing PixPlot outputs with 2 clusters for 20000 images to folder output\n",
" * skipping image validation\n",
" * creating image thumbs\n",
"0it [00:00, ?it/s]\n",
" * verifying inception model availability\n",
" * creating tf graph\n",
"2018-07-04 22:00:33.044406: W tensorflow/core/framework/op_def_util.cc:346] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization().\n",
" * creating image vectors\n",
"2018-07-04 22:00:33.366946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1378] Found device 0 with properties: \n",
"name: Tesla K10.G2.8GB major: 3 minor: 0 memoryClockRate(GHz): 0.745\n",
"pciBusID: 0000:04:00.0\n",
"totalMemory: 3.44GiB freeMemory: 4.00MiB\n",
"2018-07-04 22:00:33.450338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1378] Found device 1 with properties: \n",
"name: Tesla K10.G2.8GB major: 3 minor: 0 memoryClockRate(GHz): 0.745\n",
"pciBusID: 0000:05:00.0\n",
"totalMemory: 3.44GiB freeMemory: 68.00MiB\n",
"2018-07-04 22:00:33.450590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1457] Adding visible gpu devices: 0, 1\n",
"2018-07-04 22:00:33.450643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:938] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-07-04 22:00:33.450668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:944] 0 1 \n",
"2018-07-04 22:00:33.450691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:957] 0: N Y \n",
"2018-07-04 22:00:33.450703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:957] 1: Y N \n",
"100%|███████████████████████████████████| 20000/20000 [00:02<00:00, 9563.10it/s]\n",
" * loading image vectors\n",
"100%|███████████████████████████████████| 20000/20000 [00:12<00:00, 1568.53it/s]\n",
" * writing main JSON plot data file\n",
" * calculating 2 clusters\n",
"100%|███████████████████████████████████████████| 2/2 [00:00<00:00, 7108.99it/s]\n",
" * calculating 2D image positions\n",
" * building 2D projection\n",
" * writing JSON file\n",
"100%|███████████████████████████████████| 20000/20000 [00:06<00:00, 3093.08it/s]\n",
" * creating atlas files\n",
" * creating atlas 1 at size 32\n",
" * creating atlas 2 at size 32\n",
" * creating atlas 3 at size 32\n",
" * creating atlas 4 at size 32\n",
" * creating atlas 5 at size 32\n",
" * creating atlas 1 at size 64\n",
" * creating atlas 2 at size 64\n",
" * creating atlas 3 at size 64\n",
" * creating atlas 4 at size 64\n",
" * creating atlas 5 at size 64\n",
" * creating atlas 6 at size 64\n",
" * creating atlas 7 at size 64\n",
" * creating atlas 8 at size 64\n",
" * creating atlas 9 at size 64\n",
" * creating atlas 10 at size 64\n",
" * creating atlas 11 at size 64\n",
" * creating atlas 12 at size 64\n",
" * creating atlas 13 at size 64\n",
" * creating atlas 14 at size 64\n",
" * creating atlas 15 at size 64\n",
" * creating atlas 16 at size 64\n",
" * creating atlas 17 at size 64\n",
" * creating atlas 18 at size 64\n",
" * creating atlas 19 at size 64\n",
" * creating atlas 20 at size 64\n",
"Processed output for 20000 images\n",
"\u001b[0m"
]
}
],
"source": [
"!python ../utils/process_images.py --image_files \\\n",
"\"/Z/personal-folders/interns/saket/github/pywsi/normal_patches_valid/level_0/*.png\" --novalidate_images \\\n",
"--clusters 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# After .."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" * writing PixPlot outputs with 2 clusters for 20000 images to folder output\n",
" * skipping image validation\n",
" * creating image thumbs\n",
"0it [00:00, ?it/s]\n",
" * verifying inception model availability\n",
" * creating tf graph\n",
"2018-07-05 10:28:11.291891: W tensorflow/core/framework/op_def_util.cc:346] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization().\n",
" * creating image vectors\n",
"2018-07-05 10:28:11.654942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1378] Found device 0 with properties: \n",
"name: Tesla K10.G2.8GB major: 3 minor: 0 memoryClockRate(GHz): 0.745\n",
"pciBusID: 0000:04:00.0\n",
"totalMemory: 3.44GiB freeMemory: 4.00MiB\n",
"2018-07-05 10:28:11.753314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1378] Found device 1 with properties: \n",
"name: Tesla K10.G2.8GB major: 3 minor: 0 memoryClockRate(GHz): 0.745\n",
"pciBusID: 0000:05:00.0\n",
"totalMemory: 3.44GiB freeMemory: 68.00MiB\n",
"2018-07-05 10:28:11.753524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1457] Adding visible gpu devices: 0, 1\n",
"2018-07-05 10:28:11.753577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:938] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-07-05 10:28:11.753594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:944] 0 1 \n",
"2018-07-05 10:28:11.753607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:957] 0: N Y \n",
"2018-07-05 10:28:11.753617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:957] 1: Y N \n",
"100%|███████████████████████████████████| 20000/20000 [00:02<00:00, 8255.55it/s]\n",
" * loading image vectors\n",
"100%|███████████████████████████████████| 20000/20000 [00:13<00:00, 1484.93it/s]\n",
" * writing main JSON plot data file\n",
" * calculating 2 clusters\n",
"100%|███████████████████████████████████████████| 2/2 [00:00<00:00, 8363.52it/s]\n",
" * calculating 2D image positions\n",
" * building 2D projection\n",
"Using current time as random seed...\n",
"Will use momentum during exaggeration phase\n",
"Using no_dims = 2, max_iter = 1000, perplexity = 25.000000, theta = 0.500000, K = -1, Sigma = -30.000000, knn_algo = 1, early_exag_coeff = 12.000000, data[0] = 0.677611Computing input similarities...\n",
"Using perplexity, so normalize input data (to prevent numerical problems)\n",
"Using perplexity, not the manually set kernel width. K (number of nearest neighbors) and sigma (bandwidth) parameters are going to be ignored.\n",
"Using ANNOY for knn search, with parameters: n_trees 50 and search_k-1\n",
"Going to allocate N: 20000, K: 75, N*K = 1500000\n",
" Building Annoy tree...\n",
"Done building Annoy tree. Begin nearest neighbor search... \n",
"Calculating dynamic kernels using perplexity \n",
"parallel (32 threads):\n",
" - point 10000 of 20000, most recent beta calculated is 9.430298 \n",
" - point 0 of 20000, most recent beta calculated is 7.984253 \n",
"Initializing the solution\n",
"Exagerating Ps by 12.000000\n",
"Input similarities computed (sparsity = 0.005990)!\n",
"Learning embedding...\n",
"Using FIt-SNE approximation.\n",
"Iteration 50 (50 iterations in 2.057416 seconds)\n",
"Iteration 100 (50 iterations in 2.017811 seconds)\n",
"Iteration 150 (50 iterations in 2.047739 seconds)\n",
"Unexaggerating Ps by 12.000000\n",
"Iteration 200 (50 iterations in 2.108024 seconds)\n",
"Iteration 250 (50 iterations in 2.035964 seconds)\n",
"Iteration 300 (50 iterations in 2.099845 seconds)\n",
"Iteration 350 (50 iterations in 2.054103 seconds)\n",
"Iteration 400 (50 iterations in 2.081333 seconds)\n",
"Iteration 450 (50 iterations in 2.063150 seconds)\n",
"Iteration 500 (50 iterations in 2.089485 seconds)\n",
"Iteration 550 (50 iterations in 2.259495 seconds)\n",
"Iteration 600 (50 iterations in 2.706676 seconds)\n",
"Iteration 650 (50 iterations in 3.378853 seconds)\n",
"Iteration 700 (50 iterations in 3.713648 seconds)\n",
"Iteration 750 (50 iterations in 4.671574 seconds)\n",
"Iteration 800 (50 iterations in 3.849855 seconds)\n",
"Iteration 850 (50 iterations in 5.678153 seconds)\n",
"Iteration 900 (50 iterations in 4.936059 seconds)\n",
"Iteration 950 (50 iterations in 6.517190 seconds)\n",
"Iteration 999 (50 iterations in 7.452297 seconds)\n",
" * writing JSON file\n",
"100%|███████████████████████████████████| 20000/20000 [00:06<00:00, 2992.04it/s]\n",
" * creating atlas files\n",
"100%|█████████████████████████████████████████████| 2/2 [01:46<00:00, 53.40s/it]\n",
"Processed output for 20000 images\n",
"\u001b[0m"
]
}
],
"source": [
"!python ../utils/process_images.py --image_files \\\n",
"\"/Z/personal-folders/interns/saket/github/pywsi/normal_patches_valid/level_0/*.png\" --novalidate_images \\\n",
"--clusters 2 --layout fitsne"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python [default]",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment