test.rst

Looking at Logs

Logs are extremely useful in piecing together what went wrong when things go wrong. They contain a forensic record of what individual pieces of software were doing before things went bad, and can help us understand the problem so we can fix it.

TLJH collects logs from JupyterHub, Configurable HTTP Proxy & from each individual user's notebook server. All the logs are accessible via journalctl.

JupyterHub Logs

JupyterHub is responsible for user authentication, & starting / stopping user notebook servers. When there is a general systemic issue with JupyterHub (rather than a specific issue with a particular user's notebook), looking at the JupyterHub logs is a great first step.

Warning

If you are providing a snippet from the logs to someone else to help debug a problem you might have, be careful to redact any private information (such as usernames) from the snippet first!

sudo journalctl -u jupyterhub

This command displays logs from JupyterHub itself. See journactl_tips for tips on navigating the logs.

Configurable HTTP Proxy Logs

Configurable HTTP Proxy redirects traffic to JupyterHub / user notebook servers as necessary & handles HTTPS. It usually is the least problematic of the components, but things do go wrong sometimes!

sudo journalctl -u configurable-http-proxy

This command displays logs from Configurable HTTP Proxy. See journactl_tips for tips on navigating the logs.

User Server Logs

Each user gets their own notebook server, and this server also produces logs. Looking at these can be useful when a user can launch their server but run into problems after that.

sudo journalctl -u jupyter-<name-of-user>

This command displays logs from the given user's notebook server. See journactl_tips for tips on navigating the logs.

---

journalctl has a lot of options to make your life as an administrator easier. Here are some very basic tips on effective journalctl usage.

1. When looking at full logs (via sudo journalctl -u <some-name>), the output usually does not fit into one screen. Hence, it is paginated with less. This allows you to scroll up / down, search for specific words, etc. Some common keyboard shortcuts are:

   • Arrow keys to move up / down / left / right
   • G to navigate to the end of the logs
   • g to navigate to the start of the logs
   • / followed by a string to search for & enter key to search the logs from current position on screen to the end of the logs. If there are multiple results, you can use n key to jump to the next search result. Use ? instead of / to search backwards from current position
   • q or Ctrl + C to exit

   There are plenty of other commands & options to explore if you wish.

2. Add -f to any journalctl command to view live logging output that updates as new log lines are written. This is extremely useful when actively debugging an issue.

   For example, to watch live logs of JupyterHub, you can run:

   sudo journalctl -u jupyterhub -f