Skip to content

Instantly share code, notes, and snippets.

@huynhjl
Created April 13, 2019 04:21
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save huynhjl/00a9ee6958f1b0143b701eb7b2563005 to your computer and use it in GitHub Desktop.
Save huynhjl/00a9ee6958f1b0143b701eb7b2563005 to your computer and use it in GitHub Desktop.
tensorflow-scala-0.4.1 resource leak
import org.platanios.tensorflow.api._
object MemLeak {
val rt = Runtime.getRuntime
def mb(bytes: Long): String = f"${bytes.toFloat / 1024 / 1024}%.1fMB"
def printMemory(): Unit = {
rt.gc()
val total = rt.totalMemory
val free = rt.freeMemory
println(s"total=${mb(total)} free=${mb(free)} used=${mb(total - free)}")
}
def test(): Unit = {
val graph = Graph()
val session: Session = Session(graph)
try {
for (_ <- 0 until 1024 * 10) tf.createWith(graph)(tf.placeholder[Float](Shape(-1, 2, 2)))
} finally {
session.close()
graph.close()
}
}
def testLoop(count: Int): Unit = {
for (i <- 0 until count) {
println(s"testLoop $i")
test()
printMemory()
}
}
def main(args: Array[String]): Unit = {
val count = if (args.length > 0) args(0).toInt else 5
testLoop(count)
}
}
@huynhjl
Copy link
Author

huynhjl commented Apr 13, 2019

Running this program prints:

testLoop 0
2019-04-13 04:25:51.726990: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-04-13 04:25:51.806036: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:993] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-04-13 04:25:51.806643: I tensorflow/compiler/xla/service/service.cc:149] XLA service 0x7f6df16ec530 executing computations on platform CUDA. Devices:
2019-04-13 04:25:51.806660: I tensorflow/compiler/xla/service/service.cc:157]   StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1
2019-04-13 04:25:51.806943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
totalMemory: 5.94GiB freeMemory: 5.46GiB
2019-04-13 04:25:51.806976: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-04-13 04:25:51.807516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-13 04:25:51.807527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-04-13 04:25:51.807533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-04-13 04:25:51.807669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5291 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
total=944.0MB free=899.1MB used=44.9MB
testLoop 1
2019-04-13 04:25:56.641847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-04-13 04:25:56.641892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-13 04:25:56.641903: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-04-13 04:25:56.641910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-04-13 04:25:56.642066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5291 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
total=995.5MB free=947.9MB used=47.6MB
testLoop 2
2019-04-13 04:26:00.662688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-04-13 04:26:00.662737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-13 04:26:00.662746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-04-13 04:26:00.662754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-04-13 04:26:00.662987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5291 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
total=1009.5MB free=959.2MB used=50.3MB
testLoop 3
2019-04-13 04:26:04.510282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-04-13 04:26:04.510334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-13 04:26:04.510342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-04-13 04:26:04.510348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-04-13 04:26:04.510506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5291 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
total=1017.0MB free=964.0MB used=53.0MB
testLoop 4
2019-04-13 04:26:08.385520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-04-13 04:26:08.385588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-13 04:26:08.385615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-04-13 04:26:08.385621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-04-13 04:26:08.385777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5291 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
total=1019.5MB free=963.9MB used=55.6MB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment