Skip to content

Instantly share code, notes, and snippets.

View magnusja's full-sized avatar
👨‍💻
..

magnus magnusja

👨‍💻
..
123 followers · 165 following
View GitHub Profile
@magnusja
magnusja / gist:bfd4c861152b1446624e3cae9bfc0305
Created November 26, 2020 10:37
ray memory 1.0.1 gc collect
root@training-document-classification-pvndc:/training#
C:\Users\xgrxjah>kubectl exec -i -t -n staging training-document-classification-pvndc -c training "--" sh -c "clear; (bash || ash || sh)"
root@training-document-classification-pvndc:/training# ray memory
2020-11-26 10:11:42,978 INFO scripts.py:1340 -- Connecting to Ray instance at 10.42.0.177:6379.
2020-11-26 10:11:42,979 INFO worker.py:650 -- Connecting to existing Ray cluster at address: 10.42.0.177:6379
-----------------------------------------------------------------------------------------------------
Object ID Reference Type Object Size Reference Creation Site
=====================================================================================================
; worker pid=223743
@magnusja
magnusja / gist:f0cec405ace330ce8a25047646cbd0e5
Created November 26, 2020 10:36
ray memory nightly nov 24 gc collect
root@training-document-classification-8cfdt:/training# ray memory
2020-11-25 11:06:20,210 INFO scripts.py:1342 -- Connecting to Ray instance at 10.42.0.174:6379.
2020-11-25 11:06:20,210 INFO worker.py:654 -- Connecting to existing Ray cluster at address: 10.42.0.174:6379
-----------------------------------------------------------------------------------------------------
Object ID Reference Type Object Size Reference Creation Site
=====================================================================================================
; worker pid=464
ffffffffffffffffffffffff0100000001000000 PINNED_IN_MEMORY 18808561 (deserialize actor creation task arg) abc.DynamicTrainable.__init__
ffffffffffffffffffffffff0100000002000000 PINNED_IN_MEMORY 5988172 (deserialize actor creation task arg) abc.DynamicTrainable.__init__
ffffffffffffffffffffffff0100000003000000 LOCAL_REFERENCE 15 (deserialize actor creation task arg) abc.DynamicTrainable.__in
@magnusja
magnusja / gist:6ce14e08a93864fed015bc4931877c63
Created November 23, 2020 06:38
ray memory 0.8.6
root@training-document-classification-ls7b5:/training# ray memory
2020-11-23 06:27:45,222 INFO scripts.py:1042 -- Connecting to Ray instance at 10.42.0.147:6379.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1123 06:27:45.224284 114449 114449 global_state_accessor.cc:25] Redis server address = 10.42.0.147:6379, is test flag = 0
I1123 06:27:45.225081 114449 114449 redis_client.cc:141] RedisClient connected.
I1123 06:27:45.225144 114449 114449 redis_gcs_client.cc:88] RedisGcsClient Connected.
I1123 06:27:45.226150 114449 114449 service_based_gcs_client.cc:75] ServiceBasedGcsClient Connected.
-----------------------------------------------------------------------------------------------------
Object ID Reference Type Object Size Reference Creation Site
=====================================================================================================
@magnusja
magnusja / gist:0bd1849b42d7e6cd940505e3c5b779f2
Created November 21, 2020 22:28
ray 1.0.1 memory leak
root@training-document-classification-8vdnj:/training# ray memory
2020-11-21 12:54:39,332 INFO scripts.py:1317 -- Connecting to Ray instance at 10.42.0.143:6379.
2020-11-21 12:54:39,332 INFO worker.py:633 -- Connecting to existing Ray cluster at address: 10.42.0.143:6379
-----------------------------------------------------------------------------------------------------
Object ID Reference Type Object Size Reference Creation Site
=====================================================================================================
; worker pid=200348
ffffffffffffffffffffffff0100000002000000 PINNED_IN_MEMORY 5988276 (deserialize actor creation task arg) abc.DynamicTrainable.__init__
ffffffffffffffffffffffff0100000003000000 LOCAL_REFERENCE 15 (deserialize actor creation task arg) abc.DynamicTrainable.__init__
ffffffffffffffffffffffff0100000001000000 PINNED_IN_MEMORY 18808997 (deserialize actor creation task arg) abc.DynamicTrainable._
@magnusja
magnusja / gist:b9e381ada542a471dd90b95dce3f02d1
Created November 20, 2020 15:12
ray memory 1.0.1
root@training-document-classification-wdgc2:/training# ray memory
2020-11-20 15:09:56,419 INFO scripts.py:1340 -- Connecting to Ray instance at 10.42.0.130:6379.
2020-11-20 15:09:56,420 INFO worker.py:650 -- Connecting to existing Ray cluster at address: 10.42.0.130:6379
-----------------------------------------------------------------------------------------------------
Object ID Reference Type Object Size Reference Creation Site
=====================================================================================================
; worker pid=144014
ffffffffffffffffffffffff0100000002000000 PINNED_IN_MEMORY 5988152 (deserialize actor creation task arg) abc.DynamicTrainable.__init__
ffffffffffffffffffffffff0100000003000000 LOCAL_REFERENCE 15 (deserialize actor creation task arg) abc.DynamicTrainable.__init__
ffffffffffffffffffffffff0100000001000000 PINNED_IN_MEMORY 18808561 (deserialize actor creation task arg) abc.DynamicTrainable._
@magnusja
magnusja / gist:f4f4eb73678103c3438139543c15685e
Created November 20, 2020 10:22
ray memory
root@training-document-classification-4tvkc:/training# ray memory
2020-11-19 19:51:53,430 INFO scripts.py:1317 -- Connecting to Ray instance at 10.42.0.108:6379.
2020-11-19 19:51:53,431 INFO worker.py:633 -- Connecting to existing Ray cluster at address: 10.42.0.108:6379
-----------------------------------------------------------------------------------------------------
Object ID Reference Type Object Size Reference Creation Site
=====================================================================================================
; worker pid=136679
ffffffffffffffffffffffff0100000001000000 PINNED_IN_MEMORY 18808561 (deserialize actor creation task arg) abc.DynamicTrainable.__init__
ffffffffffffffffffffffff0100000002000000 PINNED_IN_MEMORY 5988152 (deserialize actor creation task arg) abc.DynamicTrainable.__init__
ffffffffffffffffffffffff0100000003000000 LOCAL_REFERENCE 15 (deserialize actor creation task arg) abc.DynamicTrainable._
@magnusja
magnusja / gist:4a3c11b310d5e3a70e7186593310b5d5
Created November 8, 2020 15:51
[tool.poetry]
name = "ocr"
version = "0.1.0"
description = ""
authors = ["Magnus <magnus@jahnen.me>"]
[tool.poetry.dependencies]
python = "3.8.6"
pyocr = "^0.7.2"
Pillow = "^8.0.1"
@magnusja
magnusja / gist:366076ae12aa456c7f5b0dea4693e249
Created June 26, 2019 16:46
[tool.poetry]
name = "document_classification"
version = "0.1.0"
description = "Generic (scanned) document classification"
authors = ["magnus <magnus@mgns.tech >"]
[tool.poetry.dependencies]
python = "^3.7"
jupyterlab = "^0.35.4"
click = "^7.0"
@magnusja
magnusja / keybase.md
Created January 2, 2019 14:14

Keybase proof

I hereby claim:

  • I am magnusja on github.
  • I am magnusja (https://keybase.io/magnusja) on keybase.
  • I have a public key ASB98UhZ7Xgqqp8_V2fNkKifhvQ8aVh39rGH5xtjhc0T1wo

To claim this, I am signing this object:

@magnusja
magnusja / rank_metrics.py
Created July 17, 2017 23:46 — forked from bwhite/rank_metrics.py
Ranking Metrics
"""Information Retrieval metrics
Useful Resources:
http://www.cs.utexas.edu/~mooney/ir-course/slides/Evaluation.ppt
http://www.nii.ac.jp/TechReports/05-014E.pdf
http://www.stanford.edu/class/cs276/handouts/EvaluationNew-handout-6-per.pdf
http://hal.archives-ouvertes.fr/docs/00/72/67/60/PDF/07-busa-fekete.pdf
Learning to Rank for Information Retrieval (Tie-Yan Liu)
"""
import numpy as np