Skip to content

Instantly share code, notes, and snippets.

@lucasw

lucasw/tensorflow.md

Last active June 3, 2019 00:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lucasw/d619cd6704adee28f91bb24dc90e7e16 to your computer and use it in GitHub Desktop.
Save lucasw/d619cd6704adee28f91bb24dc90e7e16 to your computer and use it in GitHub Desktop.
Download ZIP
tensorflow
Raw
tensorflow.md

Tensorflow 2 + Ubuntu 18.04

pip3 install tensorflow-gpu==2.0.0-alpha0

or

pip3 install tensorflow==2.0.0-alpha0

if no nvidia cudo gpu.

jupyter notebook

import tensorflow as tf
print(tf.__version__)

Image annotation

This one is very simple and fast:

https://github.com/Cartucho/OpenLabeling

Would like to have a mode where square is assumed (or some fixed aspect ratio) and the only thing to do is click in the center then scale the square, default would be last scaling.

It would be a good imgui project to make own custom one.

google drive / google photos annotation?

Standalone opengl3 + imgui based annotation https://github.com/lucasw/imgui_test - not a lot of features yet.

@lucasw
Copy link
Author

lucasw commented May 12, 2019

TODO cuda install instructions.

@lucasw
Copy link
Author

lucasw commented May 23, 2019 •
edited

AMD install?

[Radeon RX 550/550X] (rev c7)

-> Polaris 12 https://rocm.github.io/hardware.html

https://rocm.github.io/install.html

wget -qO - http://repo.radeon.com/rocm/apt/debian/rocm.gpg.key | sudo apt-key add -
echo 'deb [arch=amd64] http://repo.radeon.com/rocm/apt/debian/ xenial main' | sudo tee /etc/apt/sources.list.d/rocm.list
sudo apt update
sudo apt install rocm-dkms

Add user to video group.

$ /opt/rocm/bin/rocm-smi 


========================ROCm System Management Interface========================
================================================================================
GPU  Temp  AvgPwr  SCLK  MCLK  Fan  Perf  PwrCap  SCLK OD  MCLK OD  GPU%  
================================================================================
==============================End of ROCm SMI Log ==============================

$ /opt/rocm/opencl/bin/x86_64/clinfo 
Number of platforms:				 1
  Platform Profile:				 FULL_PROFILE
  Platform Version:				 OpenCL 1.1 Mesa 18.2.8
  Platform Name:				 Clover
  Platform Vendor:				 Mesa
  Platform Extensions:				 cl_khr_icd


  Platform Name:				 Clover
ERROR: clGetDeviceIDs(-1)

Try amdgpu-pro-19.10-785425-ubuntu-18.04.tar.xz from amd website for opencl support.

Unpacking libomxil-bellagio-bin (0.9.3-4) ...
Errors were encountered while processing:
 /tmp/apt-dpkg-install-BXQcpT/01-amdgpu-dkms_19.10-785425_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Need to undo it

https://community.amd.com/thread/229198 ?

sudo dpkg -P rocm-dkms
sudo dpkg -P rock-dkms
sudo apt install amdgpu-dkms
sudo apt install opencl-amdgpu-pro-icd

upgrade to linux 4.18 from 4.15

sudo apt install linux-image-4.18.0-20-generic linux-headers-4.18.0-20-generic

Ran the amdgpu-pro-install with the downloaded tar.xv file and it messed up the graphics (possibly due to also upgrading to 4.18 at the same time), so uninstalled and now clinfo is working again:

$ clinfo
Number of platforms                               1
  Platform Name                                   Clover
  Platform Vendor                                 Mesa
  Platform Version                                OpenCL 1.1 Mesa 18.2.8
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd
  Platform Extensions function suffix             MESA

  Platform Name                                   Clover
Number of devices                                 1
  Device Name                                     Radeon RX 550 Series (POLARIS12, DRM 3.26.0, 4.18.0-20-generic, LLVM 7.0.0)
  Device Vendor                                   AMD
  Device Vendor ID                                0x1002
  Device Version                                  OpenCL 1.1 Mesa 18.2.8
  Driver Version                                  18.2.8
  Device OpenCL C Version                         OpenCL C 1.1 
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Max compute units                               8
  Max clock frequency                             1183MHz
  Max work item dimensions                        3
  Max work item sizes                             256x256x256
  Max work group size                             256
  Preferred work group size multiple              64
  Preferred / native vector sizes                 
    char                                                16 / 16      
    short                                                8 / 8       
    int                                                  4 / 4       
    long                                                 2 / 2       
    half                                                 8 / 8        (cl_khr_fp16)
    float                                                4 / 4       
    double                                               2 / 2        (cl_khr_fp64)
  Half-precision Floating-point support           (cl_khr_fp16)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Address bits                                    64, Little-Endian
  Global memory size                              3221225472 (3GiB)
  Error Correction support                        No
  Max memory allocation                           2413737984 (2.248GiB)
  Unified memory for Host and Device              No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       32768 bits (4096 bytes)
  Global Memory cache type                        None
  Image support                                   No
  Local memory type                               Local
  Local memory size                               32768 (32KiB)
  Max number of constant args                     16
  Max constant buffer size                        2147483647 (2GiB)
  Max size of kernel argument                     1024
  Queue properties                                
    Out-of-order execution                        No
    Profiling                                     Yes
  Profiling timer resolution                      0ns
  Execution capabilities                          
    Run OpenCL kernels                            Yes
    Run native kernels                            No
  Device Extensions                               cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp64 cl_khr_fp16

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  No platform
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   No platform
  clCreateContext(NULL, ...) [default]            No platform
  clCreateContext(NULL, ...) [other]              Success [MESA]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 Clover
    Device Name                                   Radeon RX 550 Series (POLARIS12, DRM 3.26.0, 4.18.0-20-generic, LLVM 7.0.0)
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 Clover
    Device Name                                   Radeon RX 550 Series (POLARIS12, DRM 3.26.0, 4.18.0-20-generic, LLVM 7.0.0)
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 Clover
    Device Name                                   Radeon RX 550 Series (POLARIS12, DRM 3.26.0, 4.18.0-20-generic, LLVM 7.0.0)

But gnome metacity doesn't work?

Try rocm-dkms again:

sudo apt install rocm-dkms

Have to set secure boot password (but I've done this several times now).

It installs a special kernel/kernel module?

Restart, boot hangs on 'Started GNOME Display Manager'

sudo apt remove rock-dkms

restores kernel.

What next?

@lucasw
Copy link
Author

lucasw commented Jun 2, 2019 •
edited

Colab

Cheap way to get gpu access (for n-many hours?).

This suggests using github to import data
https://medium.com/@yuraist/how-to-upload-your-own-dataset-into-google-colab-e228727c87e9

Don't want to have write access to all of gdrive, could make a new account just for this (maybe the new account could have read only access to another gdrive, and that would be visible in colab?

from google.colab import files
uploaded = files.upload()

Wbere do the files go? Do they persist or have to be re-uploaded every session?

It would be nice if they were to go into the My Drive/colab Notebooks directory, maybe could drag and drop files directly into there.

It looks like after the kernel goes down everything has to be uploaded again, which is annoying. Maybe shrink the files down again (they were 50%) to speed this up.

@lucasw
Copy link
Author

lucasw commented Jun 2, 2019 •
edited

image preprocessing

Need a script that will take an image and chop it up into sub images of a given aspect ratio, then resize to desired output resolution/.
Overlap is okay, especially at different resolutions.

May need to manually browse output images and delete those that aren't desired (don't contain the desired content).

Better to do in colab/jupyter, or manually on local disk then upload? Probably the latter.

for i in *jpg; do convert $i -resize 25% ../small/$i; done

@lucasw
Copy link
Author

lucasw commented Jun 3, 2019

Try out these instructions: https://www.tensorflow.org/alpha/tutorials/images/hub_with_keras

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment