alexcpn

The answer to the question here https://stackoverflow.com/questions/563198/how-do-you-detect-where-two-line-segments-intersect by Gareth Rees is the best explanation.

We need to expand this to fit the co-ordinates

From https://stackoverflow.com/a/565282/429476

$$ \begin{align} t= (q-p) \times \vec s / \vec r \times \vec s\

alexcpn

from transformers import T5Tokenizer, T5ForConditionalGeneration
import numpy as np
import torch

class FlaxDataCollatorForT5MLM:
    """
    From https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_t5_mlm_flax.py
    """
    def __init__(self,tokenizer,noise_density,mean_noise_span_length) -> None:
        self.tokenizer = tokenizer

alexcpn

from transformers import T5Tokenizer
import numpy as np

class FlaxDataCollatorForT5MLM:
    """
    From https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_t5_mlm_flax.py
    """
    def __init__(self,tokenizer,noise_density,mean_noise_span_length) -> None:
        self.tokenizer = tokenizer
        self.noise_density = noise_density

alexcpn

loki:
  auth_enabled: false
  commonConfig:
    path_prefix: /var/loki
    replication_factor: 1
  compactor:
    apply_retention_interval: 1h
    compaction_interval: 5m
    retention_delete_worker_count: 500
    retention_enabled: true

alexcpn

'''
 Adapted and extended from 
 https://github.com/huggingface/transformers/issues/1950#issuecomment-558679189

'''
import pandas as pd
from transformers import BertTokenizer, BertModel
from sklearn.metrics.pairwise import cosine_similarity
import torch

alexcpn

    #---------------------------------------------------------------------------------------------
    # Populate the Confusion Matrix
    #---------------------------------------------------------------------------------------------

    for key,val in wrong_per_class.items(): # Key is category and val is a list of wrong classes
        summed_wrong_classes =Counter(val).most_common()
        print(f"**To Predict {categories[key]}")
        for ele in summed_wrong_classes:
            print(f" --Predicted {categories[ele[0]]} count={ele[1]}")
            confusion_matrix[key][ele[0]]=ele[1]

alexcpn

How to Get the Data back for a StorageClass with reclaimPolicy as Retain

Testing Using Rook-Ceph Storage Class with External Ceph

(Should work with any storage class)

Step 1: Create a Storage Class with reclaimPolicy as Retain

alexcpn

Mar 31 09:31:48 1freeipa-f476c9ffb-8gtc4 ns-slapd[2762]: [31/Mar/2022:09:31:48.659318010 +0000] DSRetroclPlugin - delete_changerecord: could not delete change record 11855 (rc: 32)
Mar 31 09:31:48 2freeipa-f476c9ffb-8gtc4 ns-slapd[2762]: [31/Mar/2022:09:31:48.751333697 +0000] DSRetroclPlugin - delete_changerecord: could not delete change record 11856 (rc: 32)
                3DDDDfreeipa-f476c9ffb-8gtc4 ns-slapd[2762]: [31/Mar/2022:09:31:49.888424524 +0000] DSRetroclPlugin - delete_changerecord: could not delete change record 11864 (rc: 32)
Mar 31 09:31:48 4freeipa-f476c9ffb-8gtc4 ns-slapd[2762]: [31/Mar/2022:09:31:48.783381048 +0000] DSRetroclPlugin - delete_changerecord: could not delete change record 11857 (rc: 32)
Mar 31 09:31:48 5freeipa-f476c9ffb-8gtc4 ns-slapd[2762]: [31/Mar/2022:09:31:48.826483871 +0000] DSRetroclPlugin - delete_changerecord: could not delete change record 11858 (rc: 32)
Mar 31 09:31:49 freeipa-f476c9ffb-8gtc4 ns-slapd[2762]: [31/Mar/2022:09:31:49.156622971 +0000] DSRetroclPlugin - de

alexcpn

    # Using TFidfVectorizer 
    # https://melaniewalsh.github.io/Intro-Cultural-Analytics/05-Text-Analysis/03-TF-IDF-Scikit-Learn.html
    tfidf_vectorizer = TfidfVectorizer(token_pattern=u'(?ui)\\b\\w*[a-z]+\\w*\\b',stop_words='english')  #token_pattern=u'(?ui)\\b\\w*[a-z]+\\w*\\b'
    
    df = read_syslog(sys.argv[1]) 
    tfidf_vector = tfidf_vectorizer.fit_transform(df['y_org'])
    print(tfidf_vectorizer.get_feature_names_out())
    tfidf_df = pd.DataFrame(tfidf_vector.toarray(), index=df['ds_org'], columns=tfidf_vectorizer.get_feature_names())
    # Create a new row with sum of all the terms of the existing rows
    tfidf_df.loc['00_Document Frequency'] = (tfidf_df > 0).sum()

alexcpn

```
^Croot@balamurugan-VirtualBox:~# kubectl logs rook-ceph-osd-8-ccb58986d-5bm22 -n rook-ceph -f
debug 2022-04-13T08:55:05.330+0000 7f580cf7af40 0 set uid:gid to 167:167 (ceph:ceph)
debug 2022-04-13T08:55:05.330+0000 7f580cf7af40 0 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable), process ceph-osd, pid 1
debug 2022-04-13T08:55:05.330+0000 7f580cf7af40 0 pidfile_write: ignore empty --pid-file
debug 2022-04-13T08:55:05.331+0000 7f580cf7af40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel
debug 2022-04-13T08:55:05.331+0000 7f580cf7af40 1 bdev(0x56533512c000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph-8/block
debug 2022-04-13T08:55:05.331+0000 7f580cf7af40 -1 bdev(0x56533512c000 /var/lib/ceph/osd/ceph-8/block) _aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr
debug 2022-04-13T08:55:05.332+0000 7f580cf7af40 0 starting osd.8 osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
debug 2022-04-13T08:55:05.3