Skip to content

Instantly share code, notes, and snippets.

@stain
Last active November 1, 2017 15:41
Show Gist options
  • Save stain/8ce77dc74d2a86b4d772ba5f4403ad8a to your computer and use it in GitHub Desktop.
Save stain/8ce77dc74d2a86b4d772ba5f4403ad8a to your computer and use it in GitHub Desktop.
Mentions of words like "http" or Web" in 250 PDFs submitted to WWW2018 Web Content Analysis, Semantics, and Knowledge track
for a in $(find . -name '*pdf') ; do
pdf2txt $a > $a.txt;
done
find . -name '*txt' | xargs grep -c http | cut -d : -f 2 > http
find . -name '*txt' | xargs grep -c doi.org | cut -d : -f 2 > doi.org
find . -name '*txt' | xargs grep -ci web | cut -d : -f 2 > web
find . -name '*txt' | xargs grep -ci www | cut -d : -f 2 > www
find . -name '*txt' | xargs grep -ci mining | | cut -d : -f 2 > mining
find . -name '*txt' | xargs wc -l | grep -v total$ | awk -e '{print $1}' > lines
find . -name '*txt' | xargs wc -w | grep -v total$ | awk -e '{print $1}' > words
echo 'http,doi.org,web,www,mining,words,lines' > http-counts.csv
paste -d , http doi.org web www mining words lines | sort -t , -g -r >> http-counts.csv
http doi.org web www mining words lines
138 7 14 23 1 7627 2944
56 33 37 31 2 10686 1680
52 25 9 22 4 9579 1366
44 29 19 20 6 10222 3181
42 11 20 16 0 8268 1394
40 12 26 6 1 10354 4338
39 0 184 23 5 10901 3022
37 13 3 18 6 7567 2801
37 10 23 18 18 9387 2903
34 15 14 17 0 4346 826
33 7 20 13 0 10853 3187
33 16 32 21 2 9506 1233
33 10 12 20 1 8672 1579
31 1 28 3 2 8606 1335
30 7 6 3 1 10101 3313
29 16 5 14 0 7337 1270
29 11 12 17 0 5263 863
28 5 17 15 2 5438 800
28 2 18 29 0 5578 2268
28 19 4 2 8 11235 3392
28 0 6 4 1 6390 2116
27 4 53 6 0 11034 4235
27 23 21 13 0 10496 1488
27 21 108 3 10 7769 1220
26 4 33 6 5 8071 1184
26 24 14 36 4 10676 3571
26 17 71 4 2 8773 1317
25 15 8 15 0 9630 1782
25 13 27 19 4 8116 1198
24 17 7 15 1 8718 1137
24 10 16 20 5 8814 1337
23 14 78 22 0 10250 3919
22 8 47 6 0 7994 1243
22 16 2 16 2 8985 1245
22 14 12 4 3 7761 1337
22 10 30 1 0 7348 1089
21 15 9 3 17 6825 1020
20 7 17 16 0 9286 1526
20 5 13 2 0 10520 2886
20 12 17 16 1 8193 2673
19 2 77 10 0 9383 4010
19 0 35 3 0 11106 1386
17 4 3 17 4 8301 1007
17 12 11 13 0 9043 1525
16 8 10 13 1 9556 2743
16 5 22 15 3 9060 3194
15 3 2 15 2 8550 1447
15 12 75 12 4 10776 2925
15 10 11 1 5 9929 1604
14 6 6 0 0 7656 1363
14 13 11 5 5 6118 834
14 10 9 11 0 11332 2982
14 0 16 20 0 10409 3246
13 6 8 4 5 9357 1532
13 1 47 7 8 8398 1671
13 0 4 13 0 9014 1369
12 8 28 1 0 9778 1442
12 4 4 17 24 10253 3294
12 3 2 2 13 8961 1448
12 3 16 3 1 8002 1543
12 3 12 14 1 9654 3140
12 2 7 7 4 9917 3145
12 2 69 15 1 10673 1491
12 2 21 5 2 9026 3191
11 6 37 16 6 10904 1505
11 5 17 4 4 8121 1238
11 5 13 16 2 7849 2112
11 5 1 1 5 7409 1300
11 4 84 2 3 9521 1317
11 4 24 5 2 6661 1106
11 3 3 1 2 10496 3794
11 3 14 15 4 10908 3349
11 2 2 15 0 9396 4079
11 2 20 14 0 10898 2955
11 2 11 4 25 6169 1201
11 1 27 6 9 8650 1190
11 10 34 0 3 4509 935
10 8 81 12 3 5973 1976
10 7 0 3 5 5503 913
10 5 2 12 2 7012 1389
10 5 15 12 1 9080 2526
10 3 5 11 0 6492 1064
10 3 3 2 0 10330 3191
10 3 12 0 7 6837 1150
10 2 5 15 2 9863 2959
10 2 5 10 9 8066 1538
10 2 14 3 0 6058 1450
10 0 7 0 1 10463 1632
9 2 7 13 10 7704 1270
9 2 107 5 1 11390 3095
9 2 10 5 1 7320 2827
9 0 24 4 2 9012 1828
9 0 17 3 1 7044 1131
8 7 12 11 10 8351 2533
8 5 14 1 2 8028 1365
8 5 13 4 5 6606 1855
8 3 5 10 5 8687 1239
8 3 4 12 0 8314 2743
8 3 12 0 7 11250 3068
8 3 1 15 5 11943 3411
8 2 8 14 3 8155 1204
8 2 3 1 17 8354 1272
8 2 2 4 7 9292 3970
8 2 17 6 0 7425 3412
8 2 13 1 2 9361 3494
8 2 10 0 1 7012 1252
8 0 9 7 3 7833 1334
8 0 18 8 1 9673 2882
7 6 58 8 0 8683 1436
7 3 2 12 0 9672 3200
7 3 15 2 4 7693 3853
7 2 57 1 0 6514 1047
7 2 49 20 3 9695 1391
7 2 25 2 0 8652 1557
7 2 1 6 4 9854 3991
7 2 16 1 8 6855 930
7 2 1 4 7 10032 4040
7 2 13 2 1 8057 2517
7 2 13 2 1 6763 990
7 2 11 11 8 8734 1416
7 1 7 4 3 9764 1680
7 1 46 3 4 9915 4153
7 1 19 2 2 10261 2282
7 0 93 7 3 10204 1419
6 4 16 11 7 8035 1477
6 3 9 2 0 6998 1008
6 3 14 14 3 10653 3157
6 2 9 13 1 8827 1421
6 2 6 12 4 8231 1379
6 2 4 5 0 9325 3045
6 2 4 2 3 7617 1612
6 2 4 1 6 7211 1038
6 2 41 11 0 9488 3012
6 2 4 10 1 9576 3099
6 2 31 3 1 10088 1329
6 2 24 18 2 11208 3037
6 2 2 13 0 11751 3405
6 2 15 12 3 9046 3006
6 2 12 0 3 7718 1390
6 2 1 13 4 8926 2034
6 2 1 1 1 8165 1555
6 1 30 13 1 10580 2999
6 1 17 11 17 11448 3143
6 0 4 2 2 8549 1237
5 5 5 9 4 7104 2651
5 2 8 11 0 7961 1497
5 2 5 16 1 9221 4323
5 2 3 3 0 9356 3271
5 2 3 1 22 9086 1729
5 2 2 1 0 8634 3427
5 2 13 16 3 9105 1198
5 2 1 12 1 9686 2899
5 2 11 13 6 8175 1506
5 0 4 13 5 8807 1508
5 0 31 13 17 11051 1706
5 0 29 7 12 9449 2372
4 3 3 10 4 7224 2807
4 3 1 0 2 5731 1489
4 2 9 12 0 9414 3136
4 2 4 12 32 8549 1300
4 2 4 11 4 9880 2914
4 2 3 3 17 8909 1278
4 2 2 5 1 8662 2900
4 2 2 1 3 7337 1497
4 2 2 13 2 8754 1197
4 2 15 12 6 12773 3416
4 2 14 11 0 9019 1676
4 2 1 1 7 10032 3612
4 2 1 15 1 8844 1652
4 2 11 10 2 6808 1037
4 1 8 13 0 10072 1564
4 1 7 0 7 6749 1129
4 1 11 7 4 9736 1253
4 0 5 19 3 9680 2987
4 0 4 1 4 8086 1461
4 0 0 16 2 10684 2516
3 2 8 14 21 10385 2919
3 2 8 1 0 8523 1568
3 2 7 1 1 9556 1563
3 2 6 11 3 8137 1880
3 2 33 3 2 10630 3663
3 2 3 11 2 10428 3408
3 2 16 8 2 3691 1183
3 2 15 16 7 10994 3294
3 1 8 14 0 8979 1393
3 1 51 14 11 9033 1407
3 1 4 12 0 9545 3900
3 1 2 3 2 11112 1887
3 1 11 3 1 8386 1261
3 0 8 3 0 9972 1405
2 2 8 11 2 11207 3198
2 2 6 0 4 7552 1314
2 2 46 11 2 8680 2459
2 2 3 10 2 7947 3059
2 2 3 0 0 412 75
2 2 16 13 17 10576 3286
2 2 15 18 0 10701 1349
2 2 13 3 6 11592 3365
2 2 11 12 3 8414 1301
2 2 11 11 8 7928 1090
2 1 7 10 8 9610 1385
2 1 14 1 1 8654 2916
2 1 13 3 2 10434 1676
2 0 25 13 4 10551 1654
2 0 22 14 3 8394 1635
2 0 2 0 2 9675 1765
1 4 5 13 2 10187 1880
1 1 2 5 1 9661 2021
1 0 2 1 1 7476 1344
1 0 1 0 2 9193 1826
0 9 13 16 4 8800 1195
0 8 10 3 7 7692 2744
0 7 9 3 1 6450 2297
0 7 37 8 1 11235 2967
0 5 14 0 1 9751 2795
0 4 22 5 8 8027 1354
0 1 5 2 15 10051 1725
0 13 12 23 5 9646 1458
0 0 8 16 5 9192 2462
0 0 8 16 1 9425 1483
0 0 8 11 2 7561 1521
0 0 7 4 0 8992 2281
0 0 7 18 1 9731 2720
0 0 7 11 14 10045 1538
0 0 68 17 5 8382 1463
0 0 6 3 0 9348 1409
0 0 6 10 21 9515 2762
0 0 5 2 2 6127 1087
0 0 5 1 2 7615 1164
0 0 43 0 2 8803 1492
0 0 4 14 8 10707 2995
0 0 4 1 4 7790 1293
0 0 4 0 7 8136 1390
0 0 33 1 0 7609 2247
0 0 3 2 0 9509 1595
0 0 29 4 1 10072 1841
0 0 2 4 1 9369 1573
0 0 2 0 8 4364 1032
0 0 2 0 4 4906 1148
0 0 17 1 21 9556 2732
0 0 15 0 2 8192 2229
0 0 13 1 8 8701 1447
0 0 121 5 3 6171 1707
0 0 1 12 6 8866 2532
0 0 10 15 7 10341 1585
0 0 1 0 0 720 134
0 0 1 0 0 221 27
0 0 0 3 1 9713 1581
0 0 0 0 2 9508 1541
0 0 0 0 0 5331 1014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment