Skip to content

Instantly share code, notes, and snippets.

@rain-1
Created February 27, 2022 18:47
Show Gist options
  • Save rain-1/799a4af4a9919c14c956272fe971b5d1 to your computer and use it in GitHub Desktop.
Save rain-1/799a4af4a9919c14c956272fe971b5d1 to your computer and use it in GitHub Desktop.

How to make a small tweak to free software

The target audience for this is people who are beginners at software engineering and using linux. A lot of the information here may be obvious or already known to you. The language involved is C but you do not need to know any C to read this tutorial. I used mg to write this blog post. I used vs code to edit the source code.

This post is also available on gopher://tilde.team:70/0/~river/tweak-free-software

If you use a piece of free software and it's 99% perfect but there's just this one thing it does that annoys the hell out of you.. you can in theory just fix it! Here's a look at what doing that is like. Hopefully it inspires you, or you pick up a could tricks on the way!

Step 0: Have a problem

This is the problem I'm having with the text editor I use called mg.

--**-Mg: no-newline-testfile.txt          (fundamental)--L1---------------------
No newline at end of file, add one? (y or n) 

It pesters me about adding a newline to the end of the file I'm working on. Either just add one or don't add one. I don't care! I just want to quit the program! Not as bad as vim though.

Step 1: Get the code, Build the code

The first thing to do is get the source code for the project and build it yourself, get a working version built from source. Do not assume that what you build from source will be the same as what you've installed with your linux package manager!

From googling there's 3 obvious repositories for mg:

Now we pull down the latest source code. In many cases you will want to check out a specific release or development branch, sometimes the master branch is not the right thing at all. In this situation it's fine though.

[river@river Hacking]$ git clone https://github.com/hboetes/mg.git
Cloning into 'mg'...
remote: Enumerating objects: 633, done.
remote: Counting objects: 100% (56/56), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 633 (delta 44), reused 45 (delta 41), pack-reused 577
Receiving objects: 100% (633/633), 339.19 KiB | 2.92 MiB/s, done.
Resolving deltas: 100% (386/386), done.

Find out what build system is used and how to build the software.

In this case we have a Makefile and just running make built the thing!

[river@river mg]$ make
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c autoexec.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c basic.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c bell.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c buffer.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c cinfo.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c dir.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c display.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c echo.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c extend.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c file.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c fileio.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c funmap.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c interpreter.c
interpreter.c: In function ‘expandvals’:
interpreter.c:625:18: warning: variable ‘excbuf’ set but not used [-Wunused-but-set-variable]
  625 |         char     excbuf[BUFSIZE], argbuf[BUFSIZE];
      |                  ^~~~~~
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c help.c
help.c: In function ‘showall’:
help.c:144:59: warning: ‘%s’ directive output may be truncated writing up to 79 bytes into a region of size 16 [-Wformat-truncation=]
  144 |                 (void)snprintf(keybuf, sizeof(keybuf), "%s%s ", prefix, buf);
      |                                                           ^~            ~~~
help.c:144:23: note: ‘snprintf’ output 2 or more bytes (assuming 81) into a destination of size 16
  144 |                 (void)snprintf(keybuf, sizeof(keybuf), "%s%s ", prefix, buf);
      |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c kbd.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c keymap.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c line.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c macro.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c main.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c match.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c modes.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c paragraph.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c re_search.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c region.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c search.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c spawn.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c tty.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c ttyio.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c ttykbd.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c undo.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c util.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c version.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c window.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c word.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c yank.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c cmode.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c cscope.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c dired.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c grep.c
cc -O2 -pipe -g -Wall -Wno-strict-aliasing -Wno-deprecated-declarations -DREGEX -D_GNU_SOURCE -isystem /usr/include/bsd -DLIBBSD_OVERLAY  -DHAVE_PTY_H -c tags.c
cc  autoexec.o basic.o bell.o buffer.o cinfo.o dir.o display.o echo.o extend.o file.o fileio.o funmap.o interpreter.o help.o kbd.o keymap.o line.o macro.o main.o match.o modes.o paragraph.o re_search.o region.o search.o spawn.o tty.o ttyio.o ttykbd.o undo.o util.o version.o window.o word.o yank.o cmode.o cscope.o dired.o grep.o tags.o -o mg -lncursesw  -lbsd  -lutil
[river@river mg]$ ./mg -v
./mg: invalid option -- 'v'

This is good going! Normally this setup will be much more involved, take a lot of googling and looking into how to correctly invoke the build system tools. We got lucky today.

Step 2: Locate relevant portions of code.

[river@river mg]$ wc *.c
   116    363   2424 autoexec.c
   580   1965  12088 basic.c
    90    178   1201 bell.c
  1070   3768  23763 buffer.c
   158    914   4797 cinfo.c
   522   1801  10617 cmode.c
   611   1771  12632 cscope.c
   183    487   3496 dir.c
  1210   3712  26251 dired.c
  1081   4556  26841 display.c
  1020   3530  21527 echo.c
   942   3346  22160 extend.c
   778   2991  19141 file.c
   763   2638  16594 fileio.c
   336   1075   9996 funmap.c
   362   1081   7686 grep.c
   235    773   5228 help.c
   977   3389  22232 interpreter.c
   462   1471   9569 kbd.c
   577   1676   9211 keymap.c
   624   2315  14699 line.c
   408   1132   9031 log.c
   112    306   1970 macro.c
   347   1064   7351 main.c
   190    750   4376 match.c
   174    558   3555 modes.c
   499   1727  10953 paragraph.c
   687   2272  15082 region.c
   669   2283  15238 re_search.c
   855   2697  18504 search.c
    51    148   1003 spawn.c
   562   1751  11996 tags.c
   459   1719  10569 tty.c
   233    752   4726 ttyio.c
    80    234   1942 ttykbd.c
   606   1736  12196 undo.c
   529   1887  11405 util.c
    28     84    525 version.c
   440   1648  10139 window.c
   514   1568  10449 word.c
   267   1141   6636 yank.c
 20407  69257 449799 total

There's 20k lines of C code here. It is pretty much never possible to read and understand the entire source code of a project you are working on. That's not how software development works

So what I'm going to do to find a relevant piece of code is simply search for the error message:

[river@river mg]$ grep -R --include '*.c' 'No newline at end of file'
file.c:		eobnl = eyorn("No newline at end of file, add one");

Now in my code editor I can see this

	if (llength(lback(lpend)) != 0) {
		eobnl = eyorn("No newline at end of file, add one");
		if (eobnl != TRUE && eobnl != FALSE)
			return (eobnl); /* abort */
	}

So just looking at the function names alone. I can infer that the if condition is doing something alone the lines of counting how far away from the end the last newline is. If this is zero the file ends in a newline. I could just delete this block of code if I wanted to. Then it wont pester me.

Lets test that out.

Reproduce the defect

If I am going to make changes I need to test them. There needs to be a sequence of instructions that reproduces the bug/feature/problem.

Here's what we can do:

[river@river mg]$ echo -n 'test' > /tmp/no-newline-testfile.txt && ./mg /tmp/no-newline-testfile.txt

press any letter to modify the file.

press control-x control-s to save it.

and we see:

'No newline at end of file, add one? (y or n)'

Now we can try deleting the block of code, build and test it out. and indeed the prompt is gone.

Step 3: Come up with an appropriate solution

Deleting the block of code would be fine for my case alone. But a better solution would be to provide a setting that can go in the mg config file ~/.mg which can pick between the current behavior (which should be the default) and just saving without adding a newline or prompting about it. While we are there, it would make sense to also add an option to just always add the newline without prompting.

In the mg manual you can see an example ~/.mg config/startup file. It looks like this:

           global-set-key ")" self-insert-command
           global-set-key "\^x\^f" find-file
           global-set-key "\e[Z" backward-char
           set-default-mode fill
           set-fill-column 72
           auto-execute *.c c-mode

So we want to add our own command that's similar to set-default-mode, let's call it set-newline-at-eof-mode and we will have 3 options:

  • prompt
  • always
  • never

To add this feature to a completely new codebase that I have no experience with. The easiest thing to do is study how set-default-mode is implemented and copy it!

In funmap.c there is a table entry:

	{set_default_mode, "set-default-mode", 1},

The comment at the top of this table explains the meaning of the number 1:

/*
 * 3rd column in the functnames structure indicates how many
parameters the
 * function takes in 'normal' usage. This column is only used to identify
 * function profiles when lines of a buffer are being evaluated via excline().
 *
 *  0 = a toggle, non-modifiable insert/delete, region modifier, etc
 *  1 = value can be string or number value (like: file/buf name,
search string)
 *  2 = multiple type value required, see auto-execute, or
global-set-key, etc
 * -1 = error: interactive commmand, unsuitable for interpreter
 *
 * Some functions when used interactively may ask for a 'y' or 'n' (or another
 * character) to continue, in excline, a 'y' is assumed. Functions
like this
 * have '0' in the 3rd column below.
 */

so we will add our own table entry

	{set_newline_at_eof_mode, "set-newline-at-eof-mode", 1},

and we will need to implement our own function set_newline_at_eof_mode which just sets a variable to one of 3 values. We can do that following what the set_default_mode function does.

So here's the patch that I've come up with:

Now we can build and test it manually and see if it works correctly. If so, set up our automatic tests.

Step 4: Regession testing the new feature

Every time you fix a bug or implement a feature in software you should also create automatic regression tests that verify the fix or functionality. This means that any code changes or updates that occur in future will either not break/revert your work - or the fact that they do will be automatically caught and flagged up!

To test command line programs on linux you can use GNU expect. This doesn't work in this situation because we have an ncurses based application. What I've found works for testing something like that is to use tmux, it implements terminal emulator and has a capture feature which dumps the pane as a text file. So you can just grep the text file to check that it looks like what you expect.

So here I've got one script for each mode of the newline setting. It verifies that the prompt comes up when we want it, and doesn't come up in the 'never' and 'always' modes. It also verifies that the file was saved with the newline added (or not, depending).

[river@river test]$ ./go.sh
1..3
ok 1 - prompt
ok 2 - always
ok 3 - never

and with that we are done. Thanks for coming with on this journey today!

@hydrocat
Copy link

A gentle and complete introduction. I liked reading this :)

@donsn
Copy link

donsn commented Feb 28, 2022

Interesting article.

Been a while I used C

@biox
Copy link

biox commented Feb 28, 2022

good stuff!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment