Skip to content

Instantly share code, notes, and snippets.

@me7
Last active August 29, 2015 14:18
Show Gist options
  • Save me7/616ba0532525ce1e1803 to your computer and use it in GitHub Desktop.
Save me7/616ba0532525ce1e1803 to your computer and use it in GitHub Desktop.
Process text file. One line at a time. Python is 45% shorter (19 vs 34 line) but go run 25% faster (1.23 vs 1.70 second on 500,000 line file). But I prefer go because easier distribute (build exe file)
package main
import (
"fmt"
"bufio"
"os"
"regexp"
"strings"
"time"
)
func main(){
start := time.Now()
f, _ := os.Open("d:\\2015\\Standish STR12A\\log\\1000373.LOG")
defer f.Close()
sc := bufio.NewScanner(f)
sc.Split(bufio.ScanLines)
re, _ := regexp.Compile("BOARD\\d\\dQ\\d=[-\\d]")
prefix := ""
for sc.Scan(){
s := string(sc.Text())
switch{
case len(s) > 30 && string(s[0]) == "@":
prefix = strings.Replace(s[1:], " ", ",", -1)
case re.MatchString(s):
s = strings.Replace(s, "=", ",", -1)
s = strings.Replace(s, "(", ",", -1)
s = strings.Replace(s, ")", ",", -1)
s = strings.Replace(s, "M", "", -1)
fmt.Println(prefix+","+s)
}
}
fmt.Printf("time usage = %.12f", time.Since(start).Seconds())
}
import re
import string
import time
start = time.time()
f = open("1000373.LOG", "r")
prefix = ""
for line in f:
if len(line) > 30 and line.startswith("@"):
prefix = string.replace(line, " ", ",")[1:].strip()
elif re.match("BOARD\\d\\dQ\\d=[-\\d]", line):
line = string.replace(line, "=", ",")
line = string.replace(line, "(", ",")
line = string.replace(line, ")", ",")
line = string.replace(line, "M", "")
print prefix + "," + line,
print "time usage = ", time.time() - start
@me7
Copy link
Author

me7 commented Apr 3, 2015

Note: Both python and go not check any error.
Python time usage = 1.70300006866 second
Go time usage = 1.234375000000 second
Text file have 489477 line. File size = 16,115,227 byte

@me7
Copy link
Author

me7 commented Apr 3, 2015

Change text file to 3.348 Million line. File size 104 MB. Go run about 30% faster
Go 8.078125 second vs Python 11.5939998627 Second
Run on windows XP SP3
CPU Intel Core2 Quad Q9300 @2.5GHz. 4 GB ram

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment