Skip to content

Instantly share code, notes, and snippets.

View ncdingari's full-sized avatar

Narahara Chari Dingari ncdingari

View GitHub Profile
@ncdingari
ncdingari / loading_wikipedia.py
Created June 19, 2020 16:58 — forked from thomwolf/loading_wikipedia.py
Load full English Wikipedia dataset in HuggingFace nlp library
import os; import psutil; import timeit
from nlp import load_dataset
mem_before = psutil.Process(os.getpid()).memory_info().rss >> 20
wiki = load_dataset("wikipedia", "20200501.en", split='train')
mem_after = psutil.Process(os.getpid()).memory_info().rss >> 20
print(f"RAM memory used: {(mem_after - mem_before)} MB")
s = """batch_size = 1000
for i in range(0, len(wiki), batch_size):
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
# Set JAVA_HOME, set max. memory, and load rJava library
Sys.setenv(JAVA_HOME='/path/to/java_home')
options(java.parameters="-Xmx2g")
library(rJava)
# Output Java version
.jinit()
print(.jcall("java/lang/System", "S", "getProperty", "java.version"))
# Load RJDBC library
@ncdingari
ncdingari / server.R
Created March 31, 2014 02:23 — forked from withr/server.R
library(shiny)
library(datasets)
Logged = FALSE;
PASSWORD <- data.frame(Brukernavn = "withr", Passord = "25d55ad283aa400af464c76d713c07ad")
# Define server logic required to summarize and view the selected dataset
shinyServer(function(input, output) {
source("www/Login.R", local = TRUE)
observe({
if (USER$Logged == TRUE) {