Vimprovement Online 1 - summary

vimprovement_1_summary.txt

Vimprovement 1: Using external programs to process text inside vim
(Based on an event https://www.meetup.com/Berlin-vim-users/events/272044751 of Berlin vim users meetup) 

Vim is well suited to interact with UNIX environment.
One of the examples of that is filtering lines with external commands without even having to leave vim.
UNIX has plenty programs to process text, and knowing them can make you more effective in working with text.


1. How to run external programs as filters inside vim

There are 3 main techniques:

  1. With visual selection: Shift-v, select lines, then :!some_command
  2. With line range: :5,10!some_command will process lines 5-10.
  3. All lines in current buffer: :%!some_command

Try out all of the above with `sort` command. You can use the text below:

number,last_name,first_name,age
15,Fraser,Kevin,34
4,Clarkson,Isaac,52
38,Dickens,Jason,63
37,Chapman,Gordon,40
5,Berry,Karen,43
27,Dowd,Joe,27
19,Ball,Joanne,48
32,Buckland,Lillian,27




2. Sorting

Sorting is a pretty common task, useful when working with data and code.
It's worth learning a couple of common options for UNIX `sort` program.

  1. Simple sort: you can use external sort without any options: `:!sort`, or vim's built-in command: `:sort`.  All the points below use the external sort.
  2. When sorting by numbers, you almost certainly want to use `-n` option.  Without it, `sort` will treat numbers as characters (so it will sort "1 10 2"); with `-n`, it will understand arithmetic numbers (and it will sort "1 2 10").
  3. Reverse sort: `sort -r`
  4. Sort by nth column: `sort -n -t, -k4`.  Explanation of each option:
    -n   - sort by numbers arithmetically
    -t,  - treat comma (",") as column separator
    -k4  - sort by 4th field of each line

Try all of the above on the text below:

3,Clarkson,Isaac,40
1,Duncan,John,30
10,Burgess,Lily,20
2,Clark,Ian,50




3. Add line numbers

Use `nl` program to add line numbers.

  1. Using plain `nl` will add some whitespace padding on both sides of added number.
  2. `nl -s, -w1` will use comma instead of whitespace as delimiter, and use width of 1 character.

Try both on the text below:

Allan,Irene,68
Duncan,John,24
Davies,James,42
Clarkson,Isaac,52
Berry,Karen,43
Black,Katherine,63




4. Pretty-print JSON

Use `python -m json.tool` to format selected JSON.
You can also use `jq` program - if you have it installed - with: `jq '.'`.  The result should be the same as when using python's `json.tool`.

Try it on the JSON snippet below:

{"widget":{"debug":"on","window":{"title":"Sample Konfabulator Widget","name":"main_window","width":500,"height":500},"image":{"src":"Images/Sun.png","name":"sun1","hOffset":250,"vOffset":250,"alignment":"center"},"text":{"data":"Click Here","size":36,"style":"bold","name":"text1","hOffset":250,"vOffset":100,"alignment":"center","onMouseUp":"sun1.opacity = (sun1.opacity / 100) * 90;"}}}   




5. Reverse lines

Use `tail -r` (or `tac`, if you have it installed) to reverse ordering of lines.

Try it on the text below:

5,Berry,Karen,43
4,Clarkson,Isaac,52
3,Davies,James,42
2,Duncan,John,24
1,Allan,Irene,68




6. Using awk

Awk is a powerful language for processing text.  A small snippet of awk can achieve as much as a quite long program written in a general-purpose language.
Here are just 2 examples of using awk:

  1. Get only first name and last name from text sample below: `awk -F, '{print $3 " " $2}'.
     `-F,` makes awk recognize commas as field separators, and `$3` and `$2` are column numbers.
  2. Filter by a selected column: `awk -F, '$4 < 45 {print $3 " " $4}'` will filter rows below by age smaller than 45, and print only first name and age columns.

Try both on the sample below:

number,last_name,first_name,age
40,Anderson,Jane,24
39,Cameron,Madeleine,69
38,Dickens,Jason,63
37,Chapman,Gordon,40
36,Alsop,Jan,24
35,Campbell,Frank,60
34,Forsyth,Keith,40




7. Combining filters

You can combine multiple programs by piping them together, just like in command line.

Try running `uniq` on the lines below, and notice that it did not filter all duplicates, because they were not subsequent.
Then undo, and run `sort | uniq`, and notice the difference.

number,last_name,first_name,age
3,Davies,James,42
6,Black,Katherine,63
2,Duncan,John,24
6,Black,Katherine,63
1,Allan,Irene,68
6,Black,Katherine,63
5,Berry,Karen,43
1,Allan,Irene,68
2,Duncan,John,24
5,Berry,Karen,43
1,Allan,Irene,68
1,Allan,Irene,68
6,Black,Katherine,63
4,Clarkson,Isaac,52
2,Duncan,John,24
4,Clarkson,Isaac,52
5,Berry,Karen,43
3,Davies,James,42
5,Berry,Karen,43
2,Duncan,John,24
4,Clarkson,Isaac,52
3,Davies,James,42
3,Davies,James,42
4,Clarkson,Isaac,52




8. Format columns neatly

`column -t` formats columnar input with neat whitespacing.
You can use `-s` option to specify column delimiter: `column -s, -t`

Try it out on the text below:

number,last_name,first_name,age
1,Allan,Irene,68
2,Duncan,John,24
3,Davies,James,42
4,Clarkson,Isaac,52
5,Berry,Karen,43
6,Black,Katherine,63
7,Bell,Julia,28




9. Using "alien" programs

You are not restricted to standard UNIX programs - you can use any program that can filter standard input and print something on standard output.
For example, check out https://pypi.org/project/python-json2yaml/.

Follow the instructions on the website to install json2yaml, and run it on the sample below:

{
    "widget": {
        "debug": "on",
        "image": {
            "alignment": "center",
            "hOffset": 250,
            "name": "sun1",
            "src": "Images/Sun.png",
            "vOffset": 250
        },
        "text": {
            "alignment": "center",
            "data": "Click Here",
            "hOffset": 250,
            "name": "text1",
            "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;",
            "size": 36,
            "style": "bold",
            "vOffset": 100
        },
        "window": {
            "height": 500,
            "name": "main_window",
            "title": "Sample Konfabulator Widget",
            "width": 500
        }
    }
}




10. Writing your own

You can not only use standard UNIX programs or programs downloaded from the Internet; you can also write your own filters, using practically any programming language!

Here is an example written in JavaScript; you will need node.js installed to run it.



#!/usr/bin/env node

// dejson - remove quote marks (") from object keys (as long as they don't contain whitespace).

var readline = require('readline')

var rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
  terminal: false
})

rl.on('line', line => {
  const match = line.match(/^(\s*)"([^ "]+)"(:.*)/)
  if (match) {
    console.log(match[1] + match[2] + match[3])
  } else {
    console.log(line)
  }
})



Copy it to a file named `dejson`, then run `chmod u+x dejson` to make it executable.
Then, run it on the JSON snippet below with `!./dejson`:

{
    "widget": {
        "debug": "on",
        "image": {
            "alignment": "center",
            "hOffset": 250,
            "name": "sun1",
            "src": "Images/Sun.png",
            "vOffset": 250
        },
        "text stuff": {
            "alignment": "center",
            "data": "Click Here",
            "hOffset": 250,
            "name": "text1",
            "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;",
            "size": 36,
            "style": "bold",
            "vOffset": 100
        },
        "window": {
            "height": 500,
            "name": "main_window",
            "title": "Sample Konfabulator Widget",
            "width": 500
        }
    }
}



Note that this little program can be also used outside of vim!  Because it adheres to the usual UNIX interface of accepting standard input and printing standard output, it can be used in variety of situations.





11. Mapping visual selection commands

You can use `vmap` to add mappings for running external commands on visual selection.
Here are some examples based on the previous exercises:

:vmap ,j !python -m json.tool<cr>
:vmap ,y !json2yaml<cr>
:vmap ,o !./myown<cr>

Try them out on the sample below:

{
"widget": {
"debug": "on",
"image": {
"alignment": "center",
"hOffset": 250,
"name": "sun1",
"src": "Images/Sun.png",
"vOffset": 250
},
"text stuff": {
"alignment": "center",
"data": "Click Here",
"hOffset": 250,
"name": "text1",
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;",
"size": 36,
"style": "bold",
"vOffset": 100
},
"window": {
"height": 500,
"name": "main_window",
"title": "Sample Konfabulator Widget",
"width": 500
}
}
}


You can add some of them to your .vimrc if you like.  If you do, just drop the leading `:` from the examples below.




THE END
Written by Milosz Danczak (milosz.danczak@sidebits.tech)