Skip to content

Instantly share code, notes, and snippets.

@wobu
Last active January 31, 2017 10:18
Show Gist options
  • Save wobu/555188a14ec5363f66ba7d2998e7268f to your computer and use it in GitHub Desktop.
Save wobu/555188a14ec5363f66ba7d2998e7268f to your computer and use it in GitHub Desktop.
dl4j decomposed TSNE
java.lang.IllegalArgumentException: Length is >= Integer.MAX_VALUE: lengthLong() must be called instead
at org.nd4j.linalg.api.ndarray.BaseNDArray.length(BaseNDArray.java:4269)
at org.nd4j.linalg.api.ndarray.BaseNDArray.muli(BaseNDArray.java:1550)
at org.nd4j.linalg.api.ndarray.BaseNDArray.muli(BaseNDArray.java:3418)
at org.deeplearning4j.plot.Tsne.x2p(Tsne.java:260)
at org.deeplearning4j.plot.Tsne.calculate(Tsne.java:106)
at org.deeplearning4j.plot.BarnesHutTsne.fit(BarnesHutTsne.java:431)
at org.deeplearning4j.plot.BarnesHutTsne.fit(BarnesHutTsne.java:641)
java.lang.IllegalArgumentException: Length is >= Integer.MAX_VALUE: lengthLong() must be called instead
at org.nd4j.linalg.api.ndarray.BaseNDArray.length(BaseNDArray.java:4269)
at org.nd4j.linalg.api.ndarray.BaseNDArray.doColumnWise(BaseNDArray.java:2198)
at org.nd4j.linalg.api.ndarray.BaseNDArray.addiColumnVector(BaseNDArray.java:2658)
at org.deeplearning4j.plot.Tsne.x2p(Tsne.java:264)
at org.deeplearning4j.plot.Tsne.calculate(Tsne.java:106)
at org.deeplearning4j.plot.BarnesHutTsne.fit(BarnesHutTsne.java:431)
at org.deeplearning4j.plot.BarnesHutTsne.fit(BarnesHutTsne.java:641)
java.lang.IllegalArgumentException: Length is >= Integer.MAX_VALUE: lengthLong() must be called instead
at org.nd4j.linalg.api.ndarray.BaseNDArray.length(BaseNDArray.java:4269)
at org.nd4j.linalg.cpu.nativecpu.CpuTADManager.getTADOnlyShapeInfo(CpuTADManager.java:67)
at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.invoke(NativeOpExecutioner.java:407)
at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:463)
at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:61)
at org.nd4j.linalg.api.ndarray.BaseNDArray.doColumnWise(BaseNDArray.java:2200)
at org.nd4j.linalg.api.ndarray.BaseNDArray.addiColumnVector(BaseNDArray.java:2658)
at org.deeplearning4j.plot.Tsne.x2p(Tsne.java:264)
at org.deeplearning4j.plot.Tsne.calculate(Tsne.java:106)
at org.deeplearning4j.plot.BarnesHutTsne.fit(BarnesHutTsne.java:431)
at org.deeplearning4j.plot.BarnesHutTsne.fit(BarnesHutTsne.java:641)
2017-01-31 11:08:38,590 INFO o.r.Reflections - Reflections took 311 ms to scan 1 urls, producing 410 keys and 1787 values
2017-01-31 11:08:38,602 INFO o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
2017-01-31 11:08:38,841 INFO o.r.Reflections - Reflections took 230 ms to scan 1 urls, producing 410 keys and 1787 values
2017-01-31 11:08:38,841 INFO o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
2017-01-31 11:08:38,883 INFO o.n.n.NativeOpsHolder - Number of threads used for NativeOps: 6
2017-01-31 11:08:39,019 INFO o.r.Reflections - Reflections took 135 ms to scan 1 urls, producing 29 keys and 185 values
2017-01-31 11:08:39,105 INFO o.n.n.Nd4jBlas - Number of threads used for BLAS: 6
2017-01-31 11:08:39,107 INFO o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
2017-01-31 11:08:39,107 INFO o.n.l.a.o.e.DefaultOpExecutioner - Cores: [12]; Memory: [49,8GB];
2017-01-31 11:08:39,107 INFO o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
2017-01-31 11:08:39,127 INFO d.s.f.v.WordEmbeddingToTensorflowProjectorTSNESparkJob - Load W2V model....
2017-01-31 11:08:39,167 DEBUG o.d.m.e.l.WordVectorSerializer - Trying simplified model restoration...
2017-01-31 11:08:46,901 DEBUG o.d.p.BarnesHutTsne - theta == 0, using decomposed version, might be slow
2017-01-31 11:08:46,910 DEBUG o.d.p.Tsne - Y:Shape is = [50000, 3]
2017-01-31 11:08:48,289 INFO o.r.Reflections - Reflections took 168 ms to scan 1 urls, producing 397 keys and 1552 values
2017-01-31 11:08:48,339 DEBUG o.d.p.Tsne - sumX shape: [50000, 1]
2017-01-31 11:08:50,988 DEBUG o.d.p.Tsne - times shape: [50000, 50000]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment