Skip to content

Instantly share code, notes, and snippets.

Avatar
🎯
Focusing

mani neomatrix369

🎯
Focusing
View GitHub Profile
@neomatrix369
neomatrix369 / dl-resources.md
Created Jul 7, 2020 — forked from upkarlidder/dl-resources.md
Deep Learning Hands-On Series with Eric Schles
View dl-resources.md
@neomatrix369
neomatrix369 / bagging_folds_cv_training.py
Last active Jun 18, 2020
Example of how to improve confidence in CV scores: using bagging, folds and using unique seed per bagging round, same seed per fold
View bagging_folds_cv_training.py
bagging_count = 5
folds = 5
SEEDS = [1234, 4567, 8910, 1112, 1314] # arbitary seeds can be any non-repeating series of integers
for bagging_index in range(0, bagging_count): # 5 rounds
stf = StratifiedKFold(n_splits=folds, shuffle=True, random_state=SEEDS[bagging_index])
for fold, (training_index, validation_index) in skf.split(train_df, train_df.sentiment): # 5 rounds
<rest of the training code>
# seeds used for each bagging round: 1234, 4567, 8910, 1112, 1314
@neomatrix369
neomatrix369 / downgrade-pytorch-model-version-from-4-to-3.sh
Last active Jun 27, 2020
Update Pytorch model (weights) version from 4 to 3 (It a shell script you can use inside your Kaggle kernel (using the `%%bash` magic)
View downgrade-pytorch-model-version-from-4-to-3.sh
%%bash
apt-get install -y xarchiver || true
SAVED_MODEL_PATH=$(realpath "/kaggle/input/[/path/to/model/folder]")
NEW_MODEL_PATH=$(realpath "/kaggle/working")
for model_file in $(ls $SAVED_MODEL_PATH/*.pth)
do
just_filename=$(basename "${model_file%.*}")
version_filename_with_path="$(unzip -t $model_file | grep version | awk '{print $2}')"
if [[ ! -e "$NEW_MODEL_PATH/$just_filename.pth" ]]; then
echo "Copying $model_file to $NEW_MODEL_PATH"
View native-image-build-for-server-binary_deploy.logs
Build on Server(pid: 1211, port: 34161)
[server-binary_deploy:1211] classlist: 178,177.80 ms
[server-binary_deploy:1211] (cap): 2,157.93 ms
[server-binary_deploy:1211] setup: 3,426.53 ms
[server-binary_deploy:1211] analysis: 104,524.34 ms
Error: Unsupported features in 6 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call: java.lang.invoke.MethodHandle.bindTo(Object)
Trace:
at parsing java.lang.invoke.MethodHandleImpl.makePairwiseConvertByEditor(MethodHandleImpl.java:221)
@neomatrix369
neomatrix369 / gist:fe8aed466f620cb2e7424fc076e50d31
Created Jan 30, 2020
Exception when building native-image from a Java 8 jar (stack trace available)
View gist:fe8aed466f620cb2e7424fc076e50d31
Build on Server(pid: 1211, port: 34161)
[server-binary_deploy:1211] classlist: 183,467.42 ms
[server-binary_deploy:1211] (cap): 4,099.72 ms
[server-binary_deploy:1211] setup: 10,420.03 ms
[server-binary_deploy:1211] analysis: 200,589.41 ms
Error: Unsupported features in 6 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call: java.lang.invoke.MethodHandle.bindTo(Object)
Trace:
at parsing java.lang.invoke.MethodHandleImpl.makePairwiseConvertByEditor(MethodHandleImpl.java:221)
@neomatrix369
neomatrix369 / build-macosx.logs
Last active Oct 24, 2019
build-macosx.logs trying to build native-image using the MLPMNist example
View build-macosx.logs
Build on Server(pid: 32662, port: 55816)
[MLPMnist-1.0.0-bin:32662] classlist: 100,008.17 ms
[MLPMnist-1.0.0-bin:32662] (cap): 3,925.44 ms
[MLPMnist-1.0.0-bin:32662] setup: 5,790.25 ms
[MLPMnist-1.0.0-bin:32662] analysis: 44,955.03 ms
Printing call tree to /Users/swami/git-repos/hands-on-workshops/deeplearning/valohai/MLPMnist-dl4j-example-graalvm/native-image/reports/call_tree_MLPMnist-1.0.0-bin_20191024_152332.txt
Printing list of used classes to /Users/swami/git-repos/hands-on-workshops/deeplearning/valohai/MLPMnist-dl4j-example-graalvm/native-image/reports/used_classes_MLPMnist-1.0.0-bin_20191024_152337.txt
Printing list of used packages to /Users/swami/git-repos/hands-on-workshops/deeplearning/valohai/MLPMnist-dl4j-example-graalvm/native-image/reports/used_packages_MLPMnist-1.0.0-bin_20191024_152337.txt
Error: Error encountered while parsing com.oracle.svm.reflect.Pointer_deallocatorThread_330a2bfd42a35375c44762314d9730587206b5b5.get(java.lang.Object)
Parsing context:
@neomatrix369
neomatrix369 / dl4j-nlp-cuda-run-gpu.logs
Created Oct 21, 2019
Logs of the GPU version of dl4j-nlp failing due to OOME (dl4j-nlp-cuda)
View dl4j-nlp-cuda-run-gpu.logs
openjdk version "1.8.0_192"
OpenJDK Runtime Environment (build 1.8.0_192-20181024121959.buildslave.jdk8u-src-tar--b12)
GraalVM 1.0.0-rc9 (build 25.192-b12-jvmci-0.49, mixed mode)
Data (.tar.gz file) already exists at /tmp/dl4j_w2vSentiment/aclImdb_v1.tar.gz
Data (extracted) already exists at /tmp/dl4j_w2vSentiment/aclImdb
2019-10-21 21:42:44,309ate - 21:42:44,309 INFO ~ Loaded [JCublasBackend] backend
2019-10-21 21:42:46,256ate - 21:42:46,256 INFO ~ Number of threads used for NativeOps: 32
2019-10-21 21:42:47,140ate - 21:42:47,140 INFO ~ Number of threads used for BLAS: 0
2019-10-21 21:42:47,145ate - 21:42:47,145 INFO ~ Backend used: [CUDA]; OS: [Linux]
2019-10-21 21:42:47,145ate - 21:42:47,145 INFO ~ Cores: [8]; Memory: [3.4GB];
@neomatrix369
neomatrix369 / dl4j-nlp-cuda-run-cpu.logs
Created Oct 21, 2019
Logs of the CPU version of dl4j-nlp failing due to OOME (dl4j-nlp-cuda)
View dl4j-nlp-cuda-run-cpu.logs
19:34:12 openjdk version "1.8.0_232"
19:34:12 OpenJDK Runtime Environment (build 1.8.0_232-20191008104205.buildslave.jdk8u-src-tar--b07)
19:34:12 OpenJDK 64-Bit GraalVM CE 19.2.1 (build 25.232-b07-jvmci-19.2-b03, mixed mode)
19:34:12 Starting data download (80MB)...
19:34:23 Data (.tar.gz file) downloaded to /tmp/dl4j_w2vSentiment/aclImdb_v1.tar.gz
19:34:28 2019-10-20 18:34:28,210ate - 18:34:28,210 INFO ~ Loaded [CpuBackend] backend
19:34:29 2019-10-20 18:34:29,610ate - 18:34:29,610 INFO ~ Number of threads used for NativeOps: 4
19:34:30 2019-10-20 18:34:30,299ate - 18:34:30,299 INFO ~ Number of threads used for BLAS: 4
19:34:30 2019-10-20 18:34:30,305ate - 18:34:30,305 INFO ~ Backend used: [CPU]; OS: [Linux]
19:34:30 2019-10-20 18:34:30,305ate - 18:34:30,305 INFO ~ Cores: [8]; Memory: [3.2GB];
@neomatrix369
neomatrix369 / graalvm-static-method-illegal-access-error.stacktrace
Created Oct 2, 2019
HotSpotConstantPool<org.bytedeco.javacpp.indexer.BooleanIndexer> illegal access exception
View graalvm-static-method-illegal-access-error.stacktrace
Values in the frame
-------------------
ex = {InvocationTargetException@235013} "java.lang.reflect.InvocationTargetException"
root = {HotSpotConstantPool@235012} "HotSpotConstantPool<org.bytedeco.javacpp.indexer.BooleanIndexer>"
initialize = false
opcode = 184
cpi = 13
After exiting the above frame
-----------------------------
@neomatrix369
neomatrix369 / graalvm-Slf4jLogger-silent-exception.stacktrace-2-of-2
Last active Oct 2, 2019
HotSpotConstantPool<org.bytedeco.javacpp.tools.Slf4jLogger> silent exception 2 of 2
View graalvm-Slf4jLogger-silent-exception.stacktrace-2-of-2
Values in the frame
-------------------
root = {HotSpotConstantPool@82148} "HotSpotConstantPool<org.bytedeco.javacpp.Loader>"
initialize = false
opcode = 179
cpi = 23
((WrappedConstantPool) root).wrapped = Cannot cast 'jdk.vm.ci.hotspot.HotSpotConstantPool' to 'com.oracle.graal.pointsto.infrastructure.WrappedConstantPool'
Stack trace
-----------