ncallister/dev-setup.md

## dev-setup.md

      
    Raw
  

              dev-setup.md
            
          
    Development Setup for Ubuntu

The following documents is basically notes on tools, repos, etc. to use for development in Ubuntu. It is devided into sections so that it can be used in pieces, it doesn't need to be followed like a script.
Notes

This is likely to end up out-of-date. Things to watch out for:

Package archives change, are abandoned or are superseded. Most of the packages archives will need to be checked before they are used.
Package names change over time, this is less likely than package archives and I have tried to use generic names for packages wherever possible.


A useful note on PPAs, wherever a debian source ends up looking like this:
deb http://ppa.launchpad.net/<team>/<archive>/ubuntu <distribution> <components ...>
You should be able to find the page for the ppa by substituting as follows:
https://launchpad.net/~<team>/+archive/ubuntu/<archive>
e.g.
deb http://ppa.launchpad.net/git-core/ppa/ubuntu xenial main --> https://launchpad.net/~git-core/+archive/ubuntu/ppa

For PPAs & debian sources, only the source is listed here. After the source has been added it is vital to refresh the package lists before proceeding to install the package:
sudo apt-get update
Kernel

Ubuntu will tend to stay on older, stable kernel releases. If you want to use a more up-to-date kernel release you can see advice here. Note that the trend seems to be heading toward automatically using newer kernel versions.

Ubuntu 18:

sudo apt install --install-recommends linux-generic-hwe-18.04 xserver-xorg-hwe-18.04
sudo apt purge linux-generic


Ubuntu 16:

sudo apt-get install --install-recommends linux-generic-hwe-16.04 xserver-xorg-hwe-16.04


Upgrade Ubuntu

To upgrade Ubuntu use update-manager for GUI or do-release-upgrade for terminal.
To keep external repositories use environment variable RELEASE_UPGRADER_ALLOW_THIRD_PARTY=1.
Full:
RELEASE_UPGRADER_ALLOW_THIRD_PARTY=1 do-release-upgrade
Link:

https://www.linuxuprising.com/2019/01/ubuntu-users-can-now-keep-ppas-and.html

Debian Package Manager Tools


aptitude - Command line apt utility

sudo apt-get install aptitude


synaptic - Gui apt utility

sudo apt-get install synaptic


debtree - Debian dependency tree generation

sudo apt install debtree
example: debtree --show-installed --no-recommends texlive-fonts-extra | dot -Tpng > /tmp/font-depends.png


Debsums - Debian checksum check tool, check packages for changes or corruption

sudo apt-get install debsums


reprepro - Create your own Debian repositories (fairly esoteric)

sudo apt-get install reprepro


Flatpak

Flatpak is a container based packaging system. Some of the apps below have options to install via flatpak in which case the following needs to have been done. More information can be found here
sudo add-apt-repository ppa:alexlarsson/flatpak && sudo apt update && sudo apt install flatpak
sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
AppImage

AppImage is a container based packaging system that allows applications to be packed into a single file which can then be executed to run the program.
This system is improved with the System Integration tool AppImageLauncher. The .deb file can be downloaded from the releases page.
AppImage applications can be downloaded directly or you can search in AppImage Hub.
Shells


ZSH - Z-Shell

sudo apt install zsh
See instructions here:

For Ubuntu 20.04
For Ubuntu 18.04


System Tools


iotop - View IO usage by process

sudo apt-get install iotop
In Ubuntu 22.04 needs a flag enabled when running, see here


dconf-editor - Graphical system configuration editor

sudo apt install dconf-editor


nethogs - View network usage by process

sudo apt-get install nethogs


gksudo - GUI Sudo (useful for .desktop files, may already be installed) - Deprecated, use pkexec

sudo apt-get install gksudo


samba - Windows share compatibility

Client - ability to connect to windows shares

sudo apt-get install smbclient


Server - Create windows shares

sudo apt install samba


Setup Guide


GAdmin-Samba - GUI config tool for samba server

sudo apt install gadmin-samba


cifs - Common Internet File System

sudo apt-get install cifs-utils


homeshick - A dot-file repository manager

Note: When uploading dot-files be very careful of files that contain private or secure information such as private keys or passwords (even hashed).
Follow the installation instructions here


gufw - A GUI for configuring the linux firewall

sudo apt-get install gufw


gtkhash - A file manager plugin for creating file hashes

Nautilus: sudo apt-get install nautilus-gtkhash
Nemo: sudo apt install nemo-gtkhash


pasystray - Pulse Audio control utilities, may already be installed

sudo apt-get install --install-suggests pasystray
It is useful to add pasystray to the applications that start on boot


PulseEffects - Pulse Audio special effects (normalizer, balancer, condenser, etc.)

Follow instructions here
For Ubuntu 18 you may need to add PPA: sudo add-apt-repository ppa:mikhailnov/pulseeffects
Alternatively it is avalable on flathub


NoiseTorch - Pulse Audio noise filtering
PAGraphControl - graphical audio routing for pulse audio

It helps to put papeaks on the PATH

papeaks requires rust to build: sudo apt install rustc


Gnome Disk Utility (a.k.a. disk image mounter) - Tools to easily mount disk images

sudo apt-get install gnome-disk-utility


OpenConnect Gnome - Cisco compatible VPN client

sudo apt install network-manager-openconnect-gnome
For GlobalProtect support try:

sudo add-apt-repository ppa:dwmw2/openconnect
sudo add-apt-repository ppa:directhex/ocgp - (use for version 8)


For SSO login try openconnect-sso - I haven't tested this yet
If it doesn't remember username:

sudo vi /etc/NetworkManager/system-connections/<connection name>
Under [vpn-secrets] add

form:main:username=<username>
save_passwords=yes
save & exit


Immediately:

sudo systemctl restart NetworkManager


Note that OpenConnect 9 correctly identifies the source operating system. If this is a problem (e.g. you need to present as win on linux) then either pin version 8 or use the --os=win on command line.
If connection attempts result in unexpected 404 responses add --useragent=AnyConnect


7zip - Compression tool. Note that Ubuntu can use the 7z format without this but this tool give specific command line access

sudo apt install p7zip-full


Unrar - Decompress rar files (fixes the "Parsing filters is unsupported" error message)

sudo apt install unrar


Alien - Convert package formats (may be already installed by default)

sudo apt install alien


Teamviewer

See here


Remmina - Remote desktop

Included in installation, normally fine
Note: If using pidgin-sipe then sipe-collab will update as appropriate
For latest version instructions here

sudo apt-add-repository ppa:remmina-ppa-team/remmina-next


Nemo - Alternative file manager with more features than nautilus

sudo apt install nemo
To set nemo as the default file manager:
xdg-mime default nemo.desktop inode/directory application/x-gnome-saved-search
gsettings set org.gnome.desktop.background show-desktop-icons false
gsettings set org.nemo.desktop show-desktop-icons true


To set nautilus back to the default:
xdg-mime default nautilus*.desktop inode/directory application/x-gnome-saved-search
gsettings set org.nemo.desktop show-desktop-icons false
gsettings set org.gnome.desktop.background show-desktop-icons true


More info here


Package Diff - Diff the internals of packages

sudo apt install pkgdiff


Memtester - Test RAM integrity

sudo apt install memtester


Apt Cacher NG - Cache downloaded debian packages

sudo apt install apt-cacher-ng
To avoid some problems, set the following in /etc/apt-cacher-ng/acng.conf:
UserAgent: Debian APT-HTTP/1.3 (1.6.10)


exFAT file system support - Prioprietary MS filesystem

sudo apt install exfat-fuse exfat-utils


gthumb - Image viewer with awesome metadata properties viewer / editor
MSGConvert - Convert email files from microsoft msg format to open eml format

sudo apt install libemail-outlook-message-perl
To run: msgconvert *.msg or msgconvert --mbox emails.mbox *.msg


tnef - Tool to unpack MIME application/ms-tnef attachments (winmail.dat)

sudo apt install tnef


Citrix receiver - if you really need it


Download the appropriate deb from here


Then fix the certs:
cd /opt/Citrix/ICAClient/keystore/
sudo mv cacerts cacerts-old
sudo ln -s /etc/ssl/certs cacerts


mkusb - Tool for creating live USB with options like persistence

See instructions here
sudo add-apt-repository ppa:mkusb/ppa
sudo apt install mkusb usb-pack-efi


qBitTorrent - Bit torrent client

sudo apt install qbittorrent


Stacer - System cleaner, monitor, configuration

Instructions on page
sudo add-apt-repository ppa:oguzhaninan/stacer
sudo apt-get install stacer


GKrellM - System monitor

sudo apt install gkrellm


Caffeine - Screen auto-lock control & rules

sudo apt install caffeine
Use the gnome extension, detailed in gnome extensions


Wine - Run windows applications in linux (not an emulator)

Some good instructions here


Ring (AKA Jami) - Softphone, SIP compatible

sudo apt install ring


Ekiga - Softphone SIP, H323 compatible

Sometimes buggy but better than ring if it works
sudo apt install ekiga


Linphone - SIP only soft-phone

sudo apt install linphone
With asterisk sip.conf use:
disallow=all                   ; First disallow all codecs
allow=speex                     ; Allow codecs in order of preference


X11VNC - Remote desktop server

Follow instructions here, needs 2 servers: one for login and one for session.
If constantly disconnecting (version 0.9.13-3), download version 0.9.13-6 from here and install manually.

x11vnc_0.9.13-6_amd64.deb
x11vnc-data_0.9.13-6_all.deb


sshuttle - Pseudo vpn using SSH forwarding

sudo apt install sshuttle
Documentation


sshfs - Mount remote directories over ssh

sudo apt install sshfs
Mount: sshfs user@host:/remote_directory /local_directory
Unmount:  fusermount -u /local_directory
https://askubuntu.com/a/925848


ClamAV & ClamTK - Antivirus for linux (trying it out for now)

sudo apt install clamtk


SnowFlake - SSH terminal & file transfer & more.

Currently install using deb file from releases page.


XSane / Sane - Scanner applications

sudo apt install xsane
sudo apt install sane


xdotool - script mouse movement, clicks, key presses etc.

sudo apt install xdotool
Windows equivalent: AutoHotkey
Can also be used to play / pause media: xdotool key XF86AudioPlay

See here


ocrmypdf - OCR my PDF, adds text layer to images of text so that they can be searched

sudo apt install ocrmypdf
ocrmypdf input.pdf output.pdf


Add Hours and Minutes - A simple time calculator

snap install add-hours-and-minutes


Emoji pickers:

gnome-characters: snap install gnome-characters
emote: snap install emote


MenuLibre - Dekstop file & menu editor

sudo apt install menulibre


Unison - File synchronization, like rsync

sudo apt install unison-all
for GUI: sudo apt install unison-all-gtk


Veracrypt - Updated version of truecrypt, encrypted drives

Consider using the unofficial PPA: here
sudo add-apt-repository ppa:unit193/encryption


duf - Disk usage and free (pretty version of du)

Ubuntu 22: sudo apt install duf
Earlier versions download deb from github


fsearch - File searching tool

sudo add-apt-repository ppa:christian-boxdoerfer/fsearch-stable && sudo apt install fsearch


fingerpaint - Use touchpad as a drawing pad
at - Run command at a certain time or after a delay

Normally already installed, otherwise sudo apt install at


evtx - Windows event log file (.evtx) to json tool
solaar - Logitech device manager for linux.

sudo apt install solaar


Drivers


Some extra Realtek chipset drivers: https://github.com/brektrou/rtl8821CU

See also some notes here: https://askubuntu.com/a/1163018


Video Editors


kdenlive

sudo add-apt-repository ppa:kdenlive/kdenlive-stable

PPA supports ubuntu 20.04 or later


sudo apt install kdenlive
Appimage also available here


openshot

sudo apt install openshot
Or follow instructions here:

sudo add-apt-repository ppa:openshot.developers/ppa
sudo apt-get update
sudo apt-get install openshot-qt


WebCam Stuff


Video4Linux - Video system for linux

sudo apt install v4l-utils
List video devices: v4l2-ctl --list-devices
List capture formats: v4l2-ctl --list-formats-ext -d /dev/video0


Cheese - Really basic webcam recorder, can't switch formats
guvcview - Basic but effective webcam recorder, lots of options

sudo apt install guvcview


Instructions on video loopback to change options: here

Btrfs

Btrfs or B-Tree File System is a newer file system with some extra featues such as snap-shots and RAID management.
If using BTRFS for the root filesystem it is a good idea to set up the following as subvolumes to exclude them from snapshots:

/tmp
/var/log
/var/tmp
/var/cache
/var/lib/snapd/
/var/lib/mlocate
~/.npm
~/.cache

Tools:

snapper - btrfs snapshot manager

sudo apt-get install snapper
or for newer versions follow instructions here (from here)


snapper-gui - A GUI for snapper

No deb, no auto updates
sudo apt-get install python3 libgtksourceview-3.0-1 python3 python3-dbus python3-setuptools git && cd /tmp && git clone --depth=1 https://github.com/ricardomv/snapper-gui.git && cd snapper-gui.git && sudo python3 setup.py install


btrfs-du: See the space used by snapshots

git clone git@github.com:nachoparker/btrfs-du
Guide


BTRFS Maintenance Scrips

Links:

BTRFS Wiki
Using BTRFS with multiple devices
BTRFS Cheat-Sheet
Using RAID with BTRFS
Debian BTRFS Wiki
Fixing BTRFS filesystem full problems

BTRFS Troubleshooting

If using a rotary disk seriously consider using the autodefrag mount option. It can increase data size with snapshots (see here) however it improves performance with scan times.
Deduplication info can be found in the wiki
It is a good idea to regularly scrub a BTRFS filesystem. A cron job may be a good idea. Regular balances in raid mode are not necessary.
To see info: btrfs filesystem show
To see raid type: btrfs filesystem df
To manually roll back with snapper:

Boot from live USB
sudo mkdir /mnt/btrfs
sudo mount -t btrfs /dev/[partition] /mnt/btrfs
Look through the snapshots to find the one you want
sudo mv /mnt/btrfs/@ /mnt/btrfs/broken/1/@
sudo mv /mnt/btrfs/@home /mnt/btrfs/broken/1/@home
sudo btrfs subvolume snapshot /mnt/btrfs/broken/1/@/.snapshots/[chosen]/snapshot /mnt/btrfs/@
sudo btrfs subvolume snapshot /mnt/btrfs/broken/1/@home/.snapshots/[chosen]/snapshot /mnt/btrfs/@home
sudo rmdir /mnt/btrfs/@/.snapshots
sudo mv /mnt/btrfs/broken/1/@/.snapshots /mnt/btrfs/@/.snapshots
sudo rmdir /mnt/btrfs/@home/.snapshots
sudo mv /mnt/btrfs/broken/1/@home/.snapshots /mnt/btrfs/@home/.snapshots
sudo btrfs property set -ts /mnt/btrfs/broken/1/@ ro true
sudo btrfs property set -ts /mnt/btrfs/broken/1/@home ro true
NOTE: Do not delete the old broken roots or it will orphan all the snapshots which makes it more difficult to perform iterative backup / copy later. If, at a later point, all snapshots of the old roots have been deleted then they can be deleted.

Consider using the latest btrfs-progs for debugging. Requires some work to build but can be built stand-alone.
If the filesystem cannot be mounted and / or is reporting as corrupted:

Try mounting with mount -t btrfs -o ro,usebackuproot
Try recovering data with btrfs restore

You may need to find the main subvolumes and use the --root <rootid> option
Consider using --xattr --metadata --symlinks to get the file metadata as well


btrfs-clone can be used to extract or copy a filesystem.
btrfs check is a good tool to see if there are problems but btrfs check --repair should only be used as a last resort. It has been known to do more harm than good.
Gnome


gnome-tweak-tool - Additional settings GUI for gnome

sudo apt-get install gnome-tweak-tool


Gnome Extensions

Gnome extensions can be managed via the (pretty cool) website: https://extensions.gnome.org/
Note: As of Ubuntu 22.04 you will need to install gnome-shell-extension-manager to manage extensions: sudo apt install gnome-shell-extension-manager
That will alow you to view both available and installed exensions. It is also the go-to place to update / upgrade installed extensions.

Applications Menu - Restore the old gnome menu

Use this one - Frippery applications menu


Dash to Dock - A dock extension to show favourites all the time rather than only when viewing current windows. I prefer this over dash to panel.
Dash to Panel - Same role as dash to dock, some differences. Play with them both and see which you prefer.
TopIcons Plus - Moves notification icons to the status bar
System Monitor - System stats on the top bar

Depends on the following:
sudo apt install gir1.2-gtop-2.0 gir1.2-networkmanager-1.0  gir1.2-clutter-1.0
For fix on disk NaN see here
When it won't install because its gnome compatibility is out of date there is a fork that often works here


No Topleft Hot Corner - Turn off hot corner
Clock Override - Format the clock (I like to use %Y-%m-%d (%a) %I:%M %p)

Note: This no longer works in Ubuntu 22 (until it is updated). An alternative is Date Menu Formatter. A sample format: EEE. y-MM-dd hh:mm aaa


Caffeine - Control the automatic screen lock
New Mail Indicator - Icon showing new mail
Places Status Indicator - Menu of filesystem shortcuts
Removable Drive Menu - Menu to eject drives
Status Area Horizontal Spacing - Customise the spacing on the status system tray

LXDE (Lubuntu) Extensions


indicator-multiload - A multi-graph system stats monitor, works in other window managers too

sudo apt install indicator-multiload
Won't show until after restart, may need to run as indicator-multiload the first time. It has an auto-start option but can also be put into the system's auto start applications list.


Gnome Themes


Adapta - A nice theme based on google material design

sudo add-apt-repository ppa:tista/adapta
sudo apt install adapta-gtk-theme
It is then helpful to use the gnome tweak tool to set the system fonts to variants of Roboto and Noto Sans. I'm using:

Applications: Adapta-Eta
Cursor: DMZ-White
Icons: Ubuntu-mono-dark
Window Title: Roboto Medium - 11
Interface: Roboto Regular - 11
Document: Roboto Regular - 11
Monospace: Noto Mono Regular - 10
Hinting: Slight
Antialiasing: Subpixel


Software Development

Version Control Systems


git - Version Control System

Git PPA - sudo add-apt-repository ppa:git-core/ppa
sudo apt-get install git
Set graph alias:

git config --global --add alias.graph "log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%ci) %C(bold blue)<%an (%G?% GK)>%Creset' --abbrev-commit --all --date-order"


To store passwords install Git credential manager

After installation run make sure to configure.
Some basic instructions:

Download & install git credential manager deb file
sudo apt install pass
git config --global --replace-all credential.credentialstore gpg
If needed create GPG key: gpg --full-generate-key
pass init <gpg-key-id>
If being constantly asked to unlock the GPG key is getting annoying edit ~/.gnupg/gpg-agent.conf and set:

default-cache-ttl 3600
max-cache-ttl 3600


subversion - Another VCS

sudo apt-get install subversion


Compilers / Runtime Engines


NodeJS - Server side JavaScript engine

Follow instructions here to get the latest version and updates.
Follow instructions here, here or elsewhere to switch the NPM global to the home directory so that sudo is not used.


Oracle Java Development Kit (JDK) - As opposed to OpenJDK

Download from the Oracle JSE downloads page
Extract the .tar.gz archive, copy to /usr/lib/jvm


Oracle Java (OLD METHOD) - For record.

WebUpd8Team's Oracle Java PPA - sudo add-apt-repository ppa:webupd8team/java
sudo apt-get install oracle-java8-installer - Check for latest JDK major version before installing
Note: There's a new PPA for java 10. Details here


OpenJDK

Java 8 with JWS:

sudo apt install openjdk-8-jdk icedtea-plugin icedtea-netx icedtea-netx-common


Java 11:

sudo apt install openjdk-11-jdk openjdk-11-source


To set default java:

sudo update-alternatives --config java
sudo vi /etc/profile.d/jdk.sh -> :%s/old-java/new-java/g
create symlink at /usr/lib/jvm/java-default for use in JAVA_HOME


Iced Tea (Updated) - the hacky, but effective way

Download the latest linux version from here (you can also use portable, but all paths end in .sh)
Put it somewhere it can stay (not a temp or Downloads dir)
sudo update-alternatives --install /usr/bin/javaws javaws $ICEDTEA_DIR/bin/javaws 200
sudo update-alternatives --install /usr/bin/itweb-settings itweb-settings $ICEDTEA_DIR/bin/itweb-settings 200
sudo update-alternatives --install /usr/bin/policyeditor policyeditor $ICEDTEA_DIR/bin/policyeditor 200
If icedtea-netx is installed via apt then also run sudo update-alternatives --config on each of the above


Java Mission Control - Awesome debugging tool

Download from Oracle JSE downloads
Unpack .tar.gz, copy to /usr/lib/jvm
sudo ln -s /usr/lib/jvm/jmc-7+14_linux-x64_bin /usr/lib/jvm/jmc

Update the link as appropriate


sudo update-alternatives --install /usr/bin/jmc jmc /usr/lib/jvm/jmc-7+14_linux-x64_bin/bin/jmc 1

Update the link as appropriate


Desktop file: vi ~/.local/share/applications/jmc.desktop
[Desktop Entry]
Name=Java Mission Control
Exec=jmc
Type=Application
Terminal=false
Icon=/usr/lib/jvm/jmc/bin/icon.xpm
Categories=Development;


.NET SDK

sudo snap install --classic dotnet-sdk
Install via package manager instructions are here

If the Ubuntu repo packages get mixed up with the microsoft one see here.


Note: The dotnet core codebase can be explored here
To create a custom SSL certificate:

Create the certificate, make sure to include OID 1.3.6.1.4.1.311.84.1.1=DER:02 on the advanced extensions
Export to PKCS12 file (.aspnet/https/name.p12 and note the password)
dotnet dev-certs https --clean --import ~/.aspnet/https/name.p12 -p password
For reference installed certs are stored here: ~/.dotnet/corefx/cryptography/x509stores/my


To trust the dev certificate follow instructions here (or here)
If you need to access a repository that needs authentication use the artifacts-credprovider


Mono - Open Source .NET Framework

Follow instructions here


PowerShell - Microsoft kind of equivalent to bash?

Follow instructions here


Security tools


sonarqube

Static analysis tool for lots of languages


Security Code Scan

Security static analysis tool for .NET and C#


IDEs / Editors


Atom - Text / code editor

WebUpd8Team's Atom PPA - sudo add-apt-repository ppa:webupd8team/atom
sudo apt-get install atom


Brackets - Adobe's Website (HTML, JavaScript, CSS etc.) editor

WebUpd8Team's Brackets PPA - sudo add-apt-repository ppa:webupd8team/brackets
sudo apt-get install brackets


NetBeans - IDE (primarily Java), Requires installed JDK

Via snap: sudo snap install netbeans

To edit config file with snap: see here (Note, replace the 10.0 with the current version of Netbeans)
mkdir -p $HOME/snap/netbeans/common/data/10.0/etc/
cp /snap/netbeans/current/netbeans/etc/netbeans.conf $HOME/snap/netbeans/common/data/10.0/etc/
gedit $HOME/snap/netbeans/common/data/10.0/etc/netbeans.conf


I've not yet found a PPA for this, for now manually download from here
Because NetBeans runs as a java process Gnome won't recognise it as its own application by default. To fix this edit the .desktop file (either in place /usr/share/applications/netbeans-8.2.desktop or copy to ~/.local/share/applications/netbeans.desktop) and add the line StartupWMClass=NetBeans IDE 8.2. If that doesn't work, check the window class using xprop. - This has been fixed with the snap installation


vim - The swiss army knife of linux text editors, careful you don't cut yourself

sudo apt-get install vim


geany - An open souce text editor / IDE (plugins optional but recommended)

sudo apt-get install geany geany-plugins
Additional colour schemes can be added using the geany-themes project.


Remarkable - Markdown editor

Download deb from here


Visual Studio Code - .NET IDE

sudo snap install --classic code

Alternatively download the deb from here


More instructions here
Configure dotnet core work-around from here on .omnisharp/omnisharp.json
Use Settings Sync plugin


Kate - KDE text editor, really cool

snap install --classic kate
Benefits from LSP servers - Some will work out of the box, others need to be configured:

Bash - snap install --classic bash-language-server
Markdown - Follow instructions here. Use the pre-built binary, don't use the snap -- permission problems.
JS/TS - npm install -g typescript-language-server typescript


Tools


Maven - Build Manager Tool (primarily for Java)

sudo apt-get install maven


diffuse - Diff viewer

sudo apt-get install diffuse


kdiff3 - 3 way diff viewer (good for merges)

sudo apt-get install kdiff3


Nautilus Compare - Use comparison tools via nautilus (installs meld)

sudo apt install nautilus-compare


Seahorse Nautilus - PGP / GPG sign, encrypt etc. via nautilus

sudo apt install seahorse-nautilus


WireShark - Packet capture and analysis

sudo apt-get install wireshark
During installation it will ask about permissions make sure to select the option to run wireshark as a non-root user
After installation add the relevant users to the wireshark group: sudo usermod -a -G wireshark username


jd-gui - A GUI for the JD Java Decompiler

Download from here (no decent ppa right now).\
Placed in personal debian repo


nmap & zenmap - Port scanning, network exploration and penetration testing (zenmap is the GUI front end)

pi-rho security PPA - sudo add-apt-repository ppa:pi-rho/security
apt-get install nmap zenmap


plantuml - Awesome UML language

Can't really be installed, see here: http://plantuml.com/
Will require graphviz: sudo apt-get install graphviz
To update use scripts/plantuml-update.sh


FileZilla - File transfer client (FTP, SFTP, etc)

apt-get install filezilla
or
flatpak install flathub org.filezillaproject.Filezilla


Etcher - Image burning utility

Follow instructions here


jq - Command line JSON parser & formatter

apt install jq


qy - Command line YAML parser & formatter

sudo snap install yq
or

sudo add-apt-repository ppa:rmescandon/yq
sudo apt install yq


XMLStarlet - Command line XML XPath parser & editor

apt install xmlstarlet
Documentation


html-xml-utils - Command line utilities for manipulating HTML, css & xml.

sudo apt install html-xml-utils


DBeaver - "Universal" database GUI

sudo add-apt-repository ppa:serge-rider/dbeaver-ce && sudo apt update && sudo apt install dbeaver-ce
or
snap install dbeaver-ce
Note: If scripts report syntax errors on semi-colons go to Window > Preferences > Editors > SQL Editor > SQL Processing, Change the "Statements delimiter" to GO and tick "Ignore native delimiter"


Eclipse Memory Analyzer (MAT) - A tool for analyzing HPROF files

See instructions from here


syft - Tool to build a "Software Bill of Materials"
grype - Vulnerability scanner for software projects

Download the latest deb from the releases page


Virtualisation


VirtualBox - Virtual machine server

Follow instructions to add the debian source from here
After installation add the relevant users to the vboxusers group (see user manual):  sudo usermod -a -G vboxusers username
For USB access install the extension pack which can be downloaded from here


Other Software


tipp10 - Typing tutor

sudo apt install tipp10


Provisioning


ansible - Server deployment / management / provisioning tool

Ansible PPA - sudo add-apt-repository ppa:ansible/ansible
sudo apt-get install ansible


Docker

Containerised applicaiton deployment

Follow instructions here

If you get an error message from the daemon like this:
dockerd[13737]: Error starting daemon: Error initializing network controller: list bridge addresses failed: no available network
Use this (replace address with one that is free in your network routing config, doesn't clash with other networks):
#!/bin/bash

#
# create docker0 bridge
# restart docker systemd service
# confirm new outgoing NAT masquerade is set up
#
# reference
#     https://docs.docker.com/engine/userguide/networking/default_network/build-bridges/
#


sudo brctl addbr docker0
sudo ip addr add 192.168.42.1/24 dev docker0
sudo ip link set dev docker0 up
ip addr show docker0
sudo systemctl restart docker
sudo iptables -t nat -L -n

exit(0)
From here, from here
Docker Tools


Docker Convenience Scripts - Clone volumes

Docker Desktop

If Windows 10 build less than 2311 (the last bit from (Get-ComputerInfo).WindowsUBR) use docker version 4.28.0, not later.
See here
Office Tools


libreoffice - Office suite, should already be installed but can be updated using the snap or PPA below

sudo snap install libreoffice
LibreOffice PPA - sudo add-apt-repository ppa:libreoffice/ppa
The Code Highlighter plugin is useful


Google Chrome - Browser, or you can go with chromium which is supposed to be similar

Download deb from here, it will install a debian source for updates.


Pandoc - Markdown compiler / renderer

sudo apt-get install pandoc
Or for the latest version go to the downloads page
Note that you will need a version of LaTeX to compile to PDF, texlive works sufficiently, just be careful that the doc packages can be huge:
sudo apt-get install texlive texlive-latex-base texlive-fonts-extra texlive-latex-recommended texlive-latex-extra texlive-generic-recommended


KolourPaint - A basic paint program

Note: This depends on a pile of KDE libraries that might otherwise be unecessary. I have not yet found a decent alternative paint program.
sudo apt-get install kolourpaint4


Krita - A not so basic paint program

sudo snap install krita


GIMP - The GNU Image Manipulation Program

sudo apt install gimp gimp-plugin-registry
Alternatives

sudo snap install gimp
flatpak install org.gimp.GIMP
These don't have a plugin aggregation, recommended Resynthesizer Plugin


Blender - 2D / 3D animation

sudo apt install blender


TupiTube - Animation for beginners & kids

Use installer from website for latest version or:
sudo apt install tupi


davmail - Gateway to allow SMTP, IMAP, etc. access to MS Exchange server

Unfortunately I haven't found any deb or snap source yet. Download from the webiste
Make sure to install suggests for the menu icon to work


Thunderbird - Email client

sudo apt-get install thunderbird
Recommended add-ons:

DKIM Verifier -
DMARC support for Thunderbird
Enigmail - PGP / GPG support
Threadvis - Email thread visualisation
Duplicate Contact Manager
Lightning - Calendar
Provider for Google Calendar
S3.Google Translator
StartupMaster - Ask for password at start-up rather than every time
Expression Search / Google Mail UI - Serch tools
Markdown Here - Write emails in markdown
Maximize Message Pane - Clean up UI for small screens
LookOut - Decode some of the proprietary Microsoft outlook stuff
ReplyWithHeader - Change the reply header to the more detailed one used by Outlook


If emojis are appearing huge run sudo apt install fonts-symbola (see details here)
If lightning calendar stops working with version 60.2 try installing it from apt: sudo apt install xul-ext-lightning
If some directories don't sync automatically and you have to keep opening the folder to get it to sync, try this

Config Editor: mail.server.default.check_all_folders_for_new -> true


BirdTray - Tray notification and control for thunderbird

Ubuntu > 20: sudo apt install birdtray
Ubuntu <= 18: sudo add-apt-repository ppa:linuxuprising/apps && sudo apt install birdtray
Flatpak: flatpak install flathub org.gyunaev.Birdtray


Delta.Chat - Email client, end to end encrypted, acts like a chat program
Dropbox - File sharing

Follow the instructions here
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys FC918B335044912E

Or after installing the repo wait for it to complain about the key to get the key id


sudo add-apt-repository "deb http://linux.dropbox.com/ubuntu $(lsb_release -sc) main"
sudo apt-get update && sudo apt-get install dropbox
Note: Dropbox now requires EXT4 so if using another FS (e.g. btrfs) first do the following (if Dropbox is already installed then this gets more complicated, be careful):

truncate -s 128G ~/dropbox.img
mkfs.ext4 ~/dropbox.img
mkdir ~/Dropbox
echo "$HOME/dropbox.img $HOME/Dropbox ext4 loop 0 2" | sudo tee -a /etc/fstab
sudo mount $HOME/Dropbox
sudo chown "$(whoami):$(whoami)" "$HOME/Dropbox"


Mega - File storage & sync

Download linux client deb from here


Grive2 - Google drive command line sync

Instructions here
sudo add-apt-repository ppa:nilarimogard/webupd8
sudo apt install grive


OneDrive for Linux - OneDrive integration client for linux

Follow instructions here


OneDriver - Fuse driver for Microsoft OneDrive
rclone - Remote cloning for cloud storage (lots of them including google & oneDrive)
Skype for Business (Lync) - Use Pidgin with the pidgin-sipe plugin (using sipe-collab)

sudo apt-add-repository ppa:sipe-collab/stable && sudo apt update
sudo apt install pidgin pidgin-sipe
You will also need to install the additional GStreamer plugins for codecs. To find the right ones search apt for "gst-plugin" and find the one labelled "bad".

sudo apt install gir1.2-gst-plugins-bad-1.0


Follow instructions here
Pidgin Sipe FAQ
Skype for Business User Agents
Note: If joining a meeting returns "can't find URI" mesage add "?UseCdn=false" to the end of the URL. Alternatively try the organiser email and meeting ID. (This issue should be resolved in the unstable PPA).
For serious debugging run pidgin from the command line using: PURPLE_UNSAFE_DEBUG=1 pidgin --debug
If the notifications of people logging on are annoying go to the plugin settings and configure the "libnotify" plugin to turn off log in notifications.
In Ubuntu 20.04 pidgin crashes when turning on system tray always to work around this edit ~/.purple/prefs.xml

Set: /pref[@name='/']/pref[@name='pidgin']/pref[@name='docklet']/pref[@name='show'] attribute value to "always"
Set: /pref[@name='/']/pref[@name='pidgin']/pref[@name='docklet']/pref[@name='gtk']/pref[@name='embedded'] attribute value to "1"


MS Teams - communication suite

Follow instructions here for the Microsoft version

Or: snap install teams


Open source client here

snap install teams-for-linux


Multimedia


vlc - Media player

sudo snap install vlc
sudo apt-get install vlc


MuseScore - Music composition

snap install musescore
Check snap info musescore for interfaces that need to be manually connected


TLS / Security


xca - A GUI Certificate Authority (awesome for inspecting certificates etc.)

sudo apt-get install xca
To create a certificate revocation list (CRL)

Use xca to generate one for your CA certificate
Export the CRL to a .pem file
Create a new github repository and put the pem file in it
Configure the githup pages to host the file
Create a template to use when generating new certificates


See also here


testssl - An awesome SSL configuration testing tool

To use, git clone the git project, git checkout an appropriate branch head and run the testssl.sh script against the website of your choice.


LetsEncrypt - Free automated TLS certification

Using CertBot
Note that with apache it now needs to use the webroot plugin. Notes here and here


Adding CA certificates to system

Warning: Only do this if you really trust the certificate. Do not EVER do this at the direction of an email without verbal confirmation from someone you trust.

If necessary create a directory for the category in /usr/share/ca-certificates

(e.g. /usr/share/ca-certificates/mycompany)


Sudo copy the CA certificate(s) to the directory

(e.g. sudo cp ./company-ca.crt /usr/share/ca-certificates/mycompany/


For each certificate add a line to /etc/ca-certificates.conf with the directory name & certificate

(e.g. mycompany/company-ca.crt)


Run: sudo update-ca-certificates

Games


LinCity-NG - SimCity style game

sudo apt install lincity-ng && mkdir ~/.lincity-ng
If you get an error on first opening, try opening from terminal
If you get an error opening on terminal:

rm -r ~/.lincity-ng && mkdir ~/.lincity-ng


Micropolis - Original SimCity

sudo snap install micropolisj


MGBA - Gameboy Advanced Emulator

snap install mgba


DeSmuME - NDS Emulator

sudo apt install desmume


Tetzle - Tetris style jigsaw puzzle, really good

sudo apt install tetzle


Simon Tatham's Portable Puzzle Collection

sudo apt install sgt-puzzles


Tanglet - Boggle game

sudo apt install tanglet

old version


flatpak install flathub org.gottcode.Tanglet


Misc / Troubleshooting


Gnome with Ubuntu 16.04 and nvidia drivers currently has a problem that may cause either tearing as windows are dragged or artifacts around the mouse cursor. To resolve this add the line below to the /etc/environment file. This, apparently, tells vsync to refresh all. I'm not 100% sure how it works.

CLUTTER_PAINT=disable-clipped-redraws:disable-culling


Gnome keyring / ssh-agent clash

See here: https://bugs.launchpad.net/ubuntu/+source/gnome-keyring/+bug/1586835
Here is the workaround:
1. disable start of ssh-agent from openssh by commenting out the use-ssh-agent line from file "/etc/X11/Xsession.options".
2. start ssh-agent from gnome-keyring daemon and set SSH_AUTH_SOCKET variable by adding the following to ${HOME}/.profile:
# From man gnome-keyring-daemon:
# Connect to an already running daemon and initialize it.
# This is often used to complete initialization of a daemon that was
# started by PAM using the --login argument.
# Output is SSH_AUTH_SOCKET=path_to_the_socket
if [ -z "$SSH_AUTH_SOCK" ]; then
    export `/usr/bin/gnome-keyring-daemon --start`
fi

Another post with some good debugging info: https://bugzilla.redhat.com/show_bug.cgi?id=1568895
Installing MS Fonts

See here or here

sudo apt install font-manager ttf-mscorefonts-installer
Accept licenses
sudo fc-cache -f -v

Any additional downloaded fonts can then be installed using font manager (e.g. Calibri).
Mounting a MS network share

Make sure cifs is installed:  apt install cifs-utils
Use the following command (with parameter substitution):
sudo mount -t cifs -o "uid=$(id -u),gid=$(id -g),username=$(id -un),file_mode=0640,dir_mode=0770" "//${SERVER}/${SHARE}" "${MOUNTDIR}"
CTRL + ALT + F terminals

As of GDM3 & Wayland terminal 1 has the login screen and terminal 2 has the session. Use terminals 3 - 7 for text, then 2 to get back to the session. If this doesn't work see below.
For some reason the old text terminals are sometimes not enabled by default. You can follow instructions here:
Modify /etc/systemd/logind.conf, uncomment NAutoVTs=6 and restart.
Also, check /etc/default/console-setup, the line: ACTIVE_CONSOLES="/dev/tty[1-6]"
Magic SysRq Key

The Magic SysRq Key is very useful, allowing reboot or oomkiller invocation via a key combination.
By default Ubuntu allows the reboot combination but not the oomkiller. If desired oomkiller can be enabled by editing /etc/sysctl.d/10-magic-sysrq.conf and adding the desired number, in my case it ended up as 240.
For reference the key combinations are listed in the wikipedia document however the following are good to know:

Alt + SysRq + b - Reboot
Alt + SysRq + f - Force oomkiller to kill something (useful when system locks up)

18.04 Bionic - Icons not showing in system tray

The best solution is to install the topicons-plus gnome plugin, otherwise see here

The reason of issue is lack of the libgtk2-appindicator-perl package in main Ubuntu 18.04 repository. A workaround is to install libappindicator-dev:
sudo apt install libappindicator-dev
and Gtk2::AppIndicator perl extension:
sudo cpan -i Gtk2::AppIndicator
Restart your applications, now all of missing icons have to appear in system tray.

Touchpad tap to click

gsettings set org.gnome.desktop.peripherals.touchpad tap-to-click true
or same in debconf
Linux / Windows shared clock

The problem here is that Linux expects the hardware clock to be set to UTC while windows sets the hardware clock to local time.
The easiest solution is to set LInux to use local time:
timedatectl set-local-rtc 1 --adjust-system-clock
Links:

https://www.howtogeek.com/323390/how-to-fix-windows-and-linux-showing-different-times-when-dual-booting/
http://ubuntuhandbook.org/index.php/2016/05/time-differences-ubuntu-1604-windows-10/
https://askubuntu.com/questions/169376/clock-time-is-off-on-dual-boot

Can't mount NTFS r/w with Windows problem

link
For reference, a sample fstab line for an NTFS drive:
UUID="<drive UUID>"  /drive   ntfs   defaults,permissions,umask=007,gid=46   0     0

Where the drive UUID can be determined using sudo blkid and the group ID 46 should to the plugdev group which users may join to gain access to the drive.
snapd invlid cross-device link

This seems to be something to do with the btrfs subvolumes? Maybe?
Workaround here: https://forum.snapcraft.io/t/snap-refresh-invalid-cross-device-link/12911
snap download <snap>
snap ack <snap>_<version>.assert
snap install <snap>_<version>.snap
Windows Domain interactions

To find the domain controller for a network, from within the network:
dig -t ANY _ldap._tcp.dc._msdcs.<domain>
For example:
dig -t ANY _ldap._tcp.dc._msdcs.example.com
USB Tethering Disconnects Immediately

Ubuntu runs a service named ModemManager by default which tries to use the USB device as a modem rather than a tether. This will disable the tethering.
For a once off solution:
sudo systemctl stop ModemManager.service
For a permanent solution:
sudo systemctl disable ModemManager.service
How to programatically type the clipboard

When an application doesn't have a paste function or can't access the clipboard use the following (from here):
Install: sudo apt install xdotool xclip
Run: sh -c 'sleep 2; xdotool type --delay 100 "$(xclip -o -selection clipboard)"'
This can also be bound to a key combination.
Firefox Graphics Acceleration

Set the following settings:
layers.acceleration.force-enabled   true
layers.omtp.enabled                 true
layout.display-list.retain          true
layout.display-list.retain.chrome   true

Next search for vaapi and enable all matching keys.
Another, semi-related optimisation which improves firefox performance is to turn on web-rendering. This will later be default so may become irrelevant:
gfx.webrender.all                   true

You may also need to set webgl.force-enabled to true and go to settings and  Untick the “recommended performance settings” box and tick the “use hardware acceleration if possible”
Using the VA-API

I haven't gotten this working yet but worth a try:

https://ubuntuhandbook.org/index.php/2021/08/enable-hardware-video-acceleration-va-api-for-firefox-in-ubuntu-20-04-18-04-higher/
https://github.com/elFarto/nvidia-vaapi-driver

Graphics tearing

For Gnome

Gnome uses mutter as composite manager.
Try this:

Alt-F2
Type "lg" and press Enter
Type "Meta.disable_unredirect_for_screen(global.screen)" and press Enter
Try to reproduce

Links:

https://bugzilla.mozilla.org/show_bug.cgi?id=1134077#c4
https://bugzilla.gnome.org/show_bug.cgi?id=741376#c15

With NVidia Drivers

Tearing can be fixed with driver settings, see here: https://www.youtube.com/watch?v=oYWer86A20s
Note that that video is a bit out of date, the option to force full composition pipeline can now be found in the NVidia X Server settings in the display configuration by pressing the "Advanced" button.
Also note, if the windows are flickering / flipping when recording or screen sharing then turn of GL Flipping in the OpenGL settings. See here.
For Lubuntu (LXDE)

Install compton:
sudo apt install compton
Add configuration (sample config file below) as either ~/.config/compton.conf or /etc/xdg/compton.conf
backend = "glx";
paint-on-overlay = true;
glx-no-stencil = true;
vsync = "opengl-swc";
unredir-if-possible = true;

# Shadow
shadow = true; # Enabled client-side shadows on windows.
no-dock-shadow = true; # Avoid drawing shadows on dock/panel windows.
no-dnd-shadow = true; # Don't draw shadows on DND windows.
clear-shadow = true; # Zero the part of the shadow's mask behind the window (experimental).
shadow-radius = 7; # The blur radius for shadows. (default 12)
shadow-offset-x = -7; # The left offset for shadows. (default -15)
shadow-offset-y = -7; # The top offset for shadows. (default -15)
shadow-exclude = [
 "n:e:Notification",
 "n:e:Docky",
 "g:e:Synapse",
 "g:e:Conky",
 "n:w:*Firefox*",
 "n:w:*Chromium*",
 "n:w:*dockbarx*",
 "class_g ?= 'Cairo-dock'",
 "class_g ?= 'Xfce4-notifyd'",
 "class_g ?= 'Xfce4-power-manager'",
 "class_g ?= 'Notify-osd'",
 "_GTK_FRAME_EXTENTS@:c"
];

# Opacity
detect-client-opacity = true;

# Window type settings
wintypes:
{
 tooltip = { shadow = false; };
};

Run compton from command line to test it. If it is working then add compton to the autostart applications in Preferences > Default Applications for LXSession then Autostart tab.
Relevant links:

https://askubuntu.com/questions/1047181/why-does-my-lubuntu-18-04-display-tear-when-theres-a-lot-of-movement-on-a-windo
https://www.reddit.com/r/linux4noobs/comments/6dpty1/screen_tearing_on_lubuntu_1704/

Screen Recording


RecordMyDesktop - A good screen recorder

sudo apt install gtk-recordmydesktop


Simple Screen Recorder

sudo add-apt-repository ppa:maarten-baert/simplescreenrecorder
sudo apt install simplescreenrecorder


Peek - Gif recording

sudo add-apt-repository ppa:peek-developers/stable && sudo apt install peek


Audio Recording

Note that to record multiple sound streams with pulse audio you will need PA Volume Control (pavucontrol).
Run the following commands:
pactl load-module module-null-sink sink_name=combined
pactl load-module module-loopback sink=combined
pactl load-module module-loopback sink=combined

You can repeat the loopback command more times if you wish to combine more streams.
Next, open volume control and go to the "recording" tab. There will be a number of "Loopback to Null Output from" recording devics.
Set each of these to monitor a stream you wish to record.
When you start the recording, set the input of the recording to "Monitor of null output".
The above config can be removed by either rebooting or restarting pulse audio with: systemctl --user stop pulseaudio.service
More good info can be found here
Name Resolution in SystemD

Bionic and later uses SystemD's ResolveD.
For details use:

man resolved.conf
systemd-resolve --status

Also, if you really want to set a nameserver to be used it seems you can add it to /etc/resolv.conf and it will work. I'm not 100% sure on that one but it is working for now.
To set a DNS server for a particular domain, the easiest way is to set the search domain on the connection in NetworkManager.
Either:
nmcli connection modify <CONNECTION NAME> ipv4.dns-search +<DOMAIN>
or edit the appropriate connection in /etc/NetworkManager/system-connections and under [ipv4] set dns-search to a semicolon delimited list of domains.
Using DNS over TLS

To enable DNS over tls:
sudo mkdir /etc/systemd/resolved.conf.d
sudo cp /etc/systemd/resolved.conf /etc/systemd/resolved.conf.d/
sudo vi /etc/systemd/resolved.conf.d/resolved.conf
Uncomment the DNSOverTLS option and set it to true to always use DNS over TLS or opportunistic to use it where it is supported but not where it isn't (less secure).
Finally edit network settings (e.g. via network manager) and include DNS servers 1.1.1.1 and 1.0.0.1.
Top icons spacing in Bionic

See here.
Generating SSH keys

There is a new SSH key type which is smaller, more secure and better than the default RSA. To use it generate your key like this:
ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519 -C "john@example.com"
Links:

https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54
https://www.ssh.com/ssh/keygen/

SSH Certificates

https://smallstep.com/blog/use-ssh-certificates/
Reload GPG Agent Config

gpg-connect-agent reloadagent /bye
See gpg-connect-agent help /bye for a complete list of commands
Playing DVDs

To play DVDs install the restricted codecs package and the content scramble system (check legal).
More info here
sudo apt install ubuntu-restricted-addons libdvd-pkg && sudo dpkg-reconfigure libdvd-pkg

Note that the restricted addons are variant specific so for Lubuntu that would be:
sudo apt install lubuntu-restricted-addons libdvd-pkg && sudo dpkg-reconfigure libdvd-pkg

LXDE & NVidia - Tiny fonts

When installing the NVidia proprietary drivers in LXDE sometimes the system font gets really, really tiny.
From here:

I'll assume you have your proprietary Nvidia driver installed. First thing you must do is to generate
default xorg.conf like this:
sudo nvidia-xconfig --no-use-edid-dpi

This will generate default xorg.conf and disable use of DPI info acquired from your monitor EDID. Now
you are free to specify your desired DPI by placing following inside section "Monitor" of your
/etc/X11/xorg.conf:
Option "DPI" "96 x 96"

You can change DPI value from 96 to your desired value. Changes should be noticeable after reboot.

This site can help you work out the correct DPI: https://www.pxcalc.com/
If the font sizes are still crazy small after setting the DPI then follow the advice here: https://code.luasoftware.com/tutorials/linux/enable-hidpi-scaling-on-lubuntu/
Also for Lubuntu increase the font size in both of:

Preferences > Openbox Configuration Manager > Appearance
Preferences > Customize Look And Feel > Widget

Switching back to Nouveau graphics

When switching away from NVidia to Nouveau some problems may occur.

Install Nouvou, easiest way is through the drivers GUI
Make sure ALL of the NVidia drivers are purged
Make sure nothing in /etc/modprobe.d is blacklisting the nouveau module
Make sure the /etc/X11/xorg.conf file isn't listing the NVidia drivers

This file can be renamed to a backup to clear it


Check that the nouveau module is loaded, try modprobe to manually load it

There is no need to list nouveau explicitly in the /etc/modules file, if everything is clear it will load.
References:

Nouveau - ArchWiki

Constant system errors

Sometimes whoopsie isn't able to report and clear system crash files and so keeps saying there has been an error then doing nothing.
The crash files are in /var/crash and can be deleted manually.
Packet sniffing from command line

Tools to man:

tcpdump
pcap-filter
ping
arping

Some examples:
tcpdump -vvvX -i eth0 host 192.168.1.184
arping -D -I <iface> -c 3 <address> || echo "Conflict found";

HDMI Overscan

This is that problem you get on a lot of TVs where it cuts off the edges inexplicably.
You can use xrandr like this:
xrandr --output HDMI-1 --set underscan on
xrandr --output HDMI-1 --set "underscan hborder" 54 --set "underscan vborder" 51

Adjust the two border values until a perfect fit.
If you get an error message about invalid values then underscan doesn't work so you will need to do the same with transform.
Example:
xrandr --output HDMI-1 --panning 1920x1080 --transform 1.05,0,-47,0,1.05,-30,0,0,1


Change the panning to the desired resolution
Adjust the transform

The first value scales x (larger shrinks)
The third value moves x (lower (more negative) moves right)
The fifth value scales y
The sixth value moves y (lower moves down)


To save the settings for that monitor you can either add those two lines to the ~.xprofile file or install autorandr.
autorandr --save annoyingtv

TODO: How to load settings
Links:

https://unix.stackexchange.com/questions/255444/how-do-i-correct-for-overscan-when-using-the-nouveau-display-driver
https://askubuntu.com/questions/754231/how-do-i-save-my-new-resolution-setting-with-xrandr

With NVIDIA setings

Fix overscan with nvidia settings by changing the display output. For example, to fix an overscan of 50 x 30 on each side set the "ViewPortOut" to 1820x1020+50+30.
Don't forget to save to the X11 config file as well.
Reset USB port

Find the appropriate port using a combination of lsusb and ll /sus/bus/usb/devices
Find the matching device in one of:

/sys/bus/pci/drivers/xhci_hcd - USB3
/sys/bus/pci/drivers/uhci_hcd - USB2

Then unbind and re-bind the device from the driver using:
echo "<device>" | sudo tee /sys/bus/pci/drivers/<driver>/unbind
echo "<device>" | sudo tee /sys/bus/pci/drivers/<driver>/bind
For example:
echo "0000:00:14.0" | sudo tee /sys/bus/pci/drivers/xhci_hcd/unbind
echo "0000:00:14.0" | sudo tee /sys/bus/pci/drivers/xhci_hcd/bind
Extend Date on GPG Key

https://www.g-loaded.eu/2010/11/01/change-expiration-date-gpg-key/
Gnome Top Bar Spacing

https://unix.stackexchange.com/questions/80542/change-reduce-gnome-3-horizontal-icon-padding-in-notification-area-debian-whe
Canon MP250 Drivers Ubuntu 18.04

Try the following if the pauses and delays when using gutenprint are just too much.
Note this will not work for > 18.04
Download the driver deb files from here
Manually install old dependencies as described here :

http://old-releases.ubuntu.com/ubuntu/pool/universe/t/tiff3/

Get libtiff4_3.9.6-9ubuntu1.2_amd64.deb


https://packages.ubuntu.com/xenial/amd64/libpng12-0/download

Get latest libpng12-0 package


From the downloaded drivers install cnijfilter-common and cnijfilter-mp250series in that order.
For > 18.04 follow instructions here
sudo add-apt-repository ppa:linuxuprising/libpng12
sudo apt update
sudo apt install libpng12-0

Battery Diagnostics

upower -e
Use this to get the battery device name / path then
upower -i <pattery path>
https://www.cyberciti.biz/faq/linux-check-battery-status/
Loopback Disk Devices

Note that this applies to disk image files, not ISO files.
Use losetup to add virtual devices to /dev representing the disks.
Use partx to list the partitions within the device and add them to the kernel so that they can be mounted.
See: https://dustymabe.com/2012/12/15/mounting-a-partition-within-a-disk-image/
Controlling Journal Size

The journal is the overall system log for newer versions of Ubuntu.
To control how large it is allowed to grow see guide here.
As a short:

sudo vi /etc/systemd/journald.conf
Uncomment SystemMaxUse and set it to SystemMaxUse=1G

Network Utilities


ss - TCP connection control including kill
nping - Ping tool which can use TCP, UDP or SNMP
arping - ARP Ping tool
fping - Ping tool which supports scanning subnets

Example: fping -c 1 -g 10.0.1.0/24


Using an Android phone as a webcam

This is based on DroidCam by Dev47Apps.
Android app:

Free
Paid

Install client on Linux, follow instructions here.
Connecting instructions here
Github here
Set WiFi Country Regulations

To see the current country settings:
iw reg get
To set the country:
sudo iw reg set <CC>
Where CC is the country code, e.g. AU.
Set it permanently by putting it in the config file: /etc/default/crda
References:

https://feeding.cloud.geek.nz/posts/setting-wifi-regulatory-domain-linux-openwrt/

Atheros WiFi Country codes

Atheros drivers use country code numbers.
Not yet sure how to apply these, or whether to ignore.
https://wireless.wiki.kernel.org/en/users/Drivers/ath
Changing the disk scheduler

The disk scheduler changes the queueing and control of requests to access the disk.
To see the current scheduler for a disk:
cat /sys/block/sdb/queue/scheduler
Note that sdb in this case is the disk.
More schedulers can be added by loading the relevant kernel modules, for example:
sudo modprobe bfq
To change the scheduler for now echo the desired scheduler to to the block queue scheduler.
To change at boot set the elevator kernel parameter by editing /etc/grub/default and adding it into the grub cmdline:
GRUB_CMDLINE_LINUX="elevator=bfq"
Then run sudo update-grub2. Note that you may need to also add the module into the /etc/modules to be loaded on boot.
TODO: Apparently grub doesn't support bfq on boot, investigate here.
References:

Kernel / Reference / IO Schedulers
Improving linux system performance with IO Schedulers

Example: Laptop

To switch sdb to use bfq:
Add file /etc/modules-load.d/bfq.conf:
bfq

Add file /etc/udev/rules.d/60-scheduler.rules:
ACTION=="add|change", KERNEL=="sdb", ATTR{queue/scheduler}="bfq"

One that doesn't rely on the devices name:
ACTION=="add|change", KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="Seagate_Expansion_NA8RFQF1-0:0",  ATTR{queue/scheduler}="bfq"

or a more generic version:
# Set known rotational disks to bfq
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"

# Set all USB disks to bfq
ACTION=="add|change", KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_BUS}=="usb",  ATTR{queue/scheduler}="bfq"

GDM No Password Login

See here

Create group nopasswdlogin
Add the desired user to that group
Sudo edit /etc/pam.d/gdm-password

Add the following to the first line:


auth sufficient pam_succeed_if.so user ingroup nopasswdlogin
GDM Screen Resolution

This allows the monitors to be configured for the GDM3 greeter in Ubuntu 20.04. I haven't tried it in any other versions.

Set up the monitors as desired in your user using the normal screen settings
sudo cp ~/.config/monitors.xml /var/lib/gdm3/.config/
sudo chown gdm:gdm /var/lib/gdm3/.config/monitors.xml

From there you can manually customise the monitors.xml file if you want to change anything further.
Taken from here
Note: If gnome's screen resolution differs from the set X11 screen resolution that can lead to some repeated resolution swapping on login / logout which can cause problems or even lose your screen for the session.
You can also customise other aspects of the GDM3 greeter using the dconf-editor but be careful:
xhost +SI:localuser:gdm;
sudo -u gdm dconf-editor
GDM Greeter Black Screen

If you get a black screen, screens off, when logging in or logging out try this
add the lines
nvidia
nvidia-modeset
nvidia-drm

to
/etc/initramfs-tools/modules
and run
sudo update-initramfs -u
Edit /etc/gdm3/custom.conf
Ucomment the line: WaylandEnable=false
Then run: sudo dpkg-reconfigure gdm
TeamViewer & Ubuntu 22

As of Ubuntu 22 the default display manager (TODO: Work out correct terminology) has been changed from X11 to Wayland.
Unfortunately Wayland requires manual acknowedgement before allowing any remote desktop sessions.
While I applaud the security measure, this makes unattended remote desktop impossible.
To work around this switch back to X11 by editing /etc/gdm3/custom.conf and uncommenting WaylandEnable=false.
Then run: sudo dpkg-reconfigure gdm
Ubuntu 22 systemd-oomd service

The systemd-oomd service in Ubuntu 22 can be quite aggressive in making sure the system doesn't slow down. Sometimes it is preferable to put up with some slow-down in order to get things done.
The following gives some pointers for blunting its claws without disabling it altogether:
Links:

https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#Memory%20Pressure%20Control
https://www.reddit.com/r/archlinux/comments/mk2lg6/how_to_properly_configure_systemdoomd/

The key properties are the memory pressure limit (% of time that the system is slowed, default 60%) and memory pressure duration (how long the system can be slower than the limit, default 20s).

To see the current settings and status use: oomctl dump
To see all current default settings use: systemd-analyze cat-config systemd/oomd.conf

To update the default settings:
sudo mkdir /etc/systemd/oomd.conf.d
sudo vi /etc/systemd/oomd.conf.d/oomd.conf
An example configuration that allows a lot of slow down:
[OOM]
DefaultMemoryPressureLimit=90%
DefaultMemoryPressureDurationSec=90s

The defaults are typically overridden by the settings in the user service (in the user slice). These are simply ways of allowing resource control to be defined for different areas or users.

To see the current user's slice configuration use: systemctl cat "user-$(id -u).slice"
To see the default user service configuration use: systemctl cat user@.service
To see the current user's service configuration use: systemctl cat "user@$(id -u).service"

To create an override for the user service configuration:
sudo mkdir /etc/systemd/system/user@.service.d
sudo vi /etc/systemd/system/user@.service.d/oomd.conf
An example contents:
[Service]
ManagedOOMMemoryPressureLimit=90%

To apply all changes:
sudo systemctl daemon-reload
sudo systemctl restart systemd-oomd.service
Configuring a headless desktop

sudo apt install -y xserver-xorg-video-dummy
sudo vi /etc/default/grub
# add "nomodeset" to GRUB_CMDLINE_LINUX_DEFAULT, save and quit
sudo vi /usr/share/X11/xorg.conf.d/xorg.conf.conf
# Add contents below
sudo update-grub && sudo reboot
xorg.conf:
Section "Monitor"
  Identifier "Monitor0"
  HorizSync 28.0-80.0
  VertRefresh 48.0-75.0
  # https://arachnoid.com/modelines/
  # 1920x1080 @ 60.00 Hz (GTF) hsync: 67.08 kHz; pclk: 172.80 MHz
  Modeline "1920x1080_60.00" 172.80 1920 2040 2248 2576 1080 1081 1084 1118 -HSync +Vsync
EndSection

Section "Device"
  Identifier "Card0"
  Driver "dummy"
  VideoRam 256000
EndSection

Section "Screen"
  DefaultDepth 24
  Identifier "Screen0"
  Device "Card0"
  Monitor "Monitor0"
  SubSection "Display"
    Depth 24
    Modes "1920x1080_60.00"
  EndSubSection
EndSection

Links:

https://askubuntu.com/a/1210419
https://techoverflow.net/2019/02/23/how-to-run-x-server-using-xserver-xorg-video-dummy-driver-on-ubuntu/
https://topitanswers.com/post/video-dummy-the-fake-display-with-without-monitor-connected-at-the-same-time
https://gist.github.com/mangoliou/27c6c5867a95932f21ae59ad7152aa33
https://www.x.org/releases/X11R7.7/doc/man/man5/xorg.conf.5.xhtml
xdpyinfo

Headless with nvidia

NVIDIA has an option in xrandr.org in the Screen section named ConnectedMonitor which forces it to consider those devices connected.
Example:
Section "Device"
    Identifier  "nvidiagpu"
    Driver      "nvidia"
EndSection

Section "Screen"
    Identifier  "nvidiascreen"
    Device      "nvidiagpu"
    Option      "ConnectedMonitor" "LVDS-0,DP-1"
EndSection

Links:

https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Headless_(no_monitor)_resolution
https://unix.stackexchange.com/a/640262

Below is a full example of headless with NVIDIA, using graphics card for 2 monitors but no monitors plugged in. Note that a HDMI dummy plug is required for the graphics card to start up at all.
# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig:  version 390.157  (dvs-builder@U64-E08-27-1)  Wed Oct 12 09:14:08 UTC 2022

# nvidia-settings: X configuration file generated by nvidia-settings
# nvidia-settings:  version 470.57.01

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
    Option         "Xinerama" "0"
EndSection

Section "Files"
EndSection

Section "Module"
    Load           "dbe"
    Load           "extmod"
    Load           "type1"
    Load           "freetype"
    Load           "glx"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"

    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Lenovo Group Limited LEN LT2323pwA"
    HorizSync       30.0 - 83.0
    VertRefresh     50.0 - 75.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "NVS 310"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "ConnectedMonitor" "DP-2,DP-3"
    Option         "ModeValidation" "AllowNon60hzmodesDFPModes, NoEDIDDFPMaxSizeCheck, NoVertRefreshCheck, NoHorizSyncCheck, NoDFPNativeResolutionCheck, NoMaxSizeCheck, NoMaxPClkCheck, AllowNonEdidModes, NoEdidMaxPClkCheck"
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "DFP-3"
    Option         "metamodes" "DP-3: 1920x1080 +0+0 {ForceCompositionPipeline=On, ForceFullCompositionPipeline=On}, DP-2: 1920x1080 +1920+0 {ForceCompositionPipeline=On, ForceFullCompositionPipeline=On}"
    Option         "SLI" "Off"
    Option         "MultiGPU" "Off"
    Option         "BaseMosaic" "off"
    Option         "CustomEDID" "DP-2:/etc/X11/DP-3-edid.bin; DP-3:/etc/X11/DP-3-edid.bin"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Boot Repair

Tool to fix booting: https://help.ubuntu.com/community/Boot-Repair
Also Grub Customiser:
sudo add-apt-repository ppa:danielrichter2007/grub-customizer
sudo apt install grub-customizer
Android Emulator

sudo apt install qemu-kvm
sudo adduser "$(whoami)" kvm
snap install android-studio

Run, setup studio. Use SDK tools to ensure emulator is installed
Open emulator device manager and create a new one, make sure it has play store (look for the icon)

Using VirtualBox


https://www.howtogeek.com/164570/HOW-TO-INSTALL-ANDROID-IN-VIRTUALBOX/
https://www.android-x86.org/
https://www.android-x86.org/documentation/virtualbox.html

Need to get the arm bridge. May need debug:
On android 9, download houdini.sfs from here
http://dl.android-x86.org/houdini.php?v=9_y
Rename it houdini9_y.sfs
And place in /sdcard/arm folder
Enable native bridge from settings also

See here: https://groups.google.com/forum/embed/#!topic/android-x86/mlU4ajAdr7A
Gnome Freeze & Log Spam

Symptoms: Gnome shell locking up or not working properly & one or either of the following spamming in syslog or journal:
Mar 21 16:56:59 starlight-glimmer gnome-shell[99361]: Attempting to run a JS callback during garbage collection. This is most likely caused by destroying a Clutter actor or GTK widget with ::destroy signal connected, or using the destroy(), dispose(), or remove() vfuncs. Because it would crash the application, it has been blocked.
Mar 21 16:56:59 starlight-glimmer gnome-shell[99361]: The offending callback was SourceFunc().

Mar 21 16:55:00 starlight-glimmer gnome-shell[99361]: == Stack trace for context 0x5ddf92267170 ==
Mar 21 16:55:00 starlight-glimmer gnome-shell[99361]: == Stack trace for context 0x5ddf92267170 ==

The problem is detailed here (hopefully fixed in ubuntu 23 and later).
The only solution is to restart gnome-shell. This can be done with: kill -HUP $(pidof gnome-shell)
Or you can create a cron job to do it for you: Run sudo crontab -e and add the following line:
*/1 * * * * [[ $(journalctl -rn 10 | grep -vi "cron" | grep 'The offending callback was SourceFunc().\|== Stack trace for context') ]] && kill -HUP $(pidof gnome-shell)

Also to minimise the damage it can do:

Edit /etc/systemd/journald.conf and set SystemMaxUse=1G
Make sure logrotate is installed, edit /etc/logrotate.d/rsyslog and add maxsize 1G inside the block.

Penetration Testing

Use Kali Linux, a dedicated distribution which provides a lot of good tools.
Windows Stuff

Yeah... I know, Windows shudder.
Windows Utilities


Full Event Log View - See all event logs
WinDirStat - See disk usage tree, similar to baobab

WSL Links

X Server:

https://sourceforge.net/projects/vcxsrv/

Guides

https://www.zdnet.com/article/how-to-run-run-the-native-ubuntu-desktop-on-windows-10/
https://ubuntu.com/blog/wslconf-sessions-part-2-devops
https://docs.microsoft.com/en-us/windows/wsl/wsl2-index
https://www.reddit.com/r/bashonubuntuonwindows/comments/9lpc0o/ubuntu_1804_dbus_fix_instructions_with/

Standard Tools


OpenSSH Client

Enable in programs (extra features)
Open services and enable the OpenSSH agent


Generate openssh key (same as linux)
Install Git

winget install --source winget --id Git.Git
Get git to use openssh agent

git config --global core.sshCommand C:/Windows/System32/OpenSSH/ssh.exe


Install GPG

winget install --source winget --id GnuPG.Gpg4win
Get git to use GPG

git config --global gpg.program (Get-Command gpg).Source


Visual Studio

Setup stuff for VS:

Extensions:

Fine Code Coverage
SonarLint


Options

Projects and Solutions > General > Track Active Item in Solution Explorer -> Selected