Casper Installation Notes Ubuntu 22.04, Xilinx Vivado 2021.1, Matlab R2021a

casper_installation_notes.md

Warning: I've been told by the CASPER developers that our specific combination of versions of the software in this document are not supported and break the software, which is why we were not able to get simulation working in Simulink. Follow the casper setup walkthrough and use this document only to troubleshoot if you get similar errors.

Installation Notes

Build computer

Install Ububtu 22.04.1 on ssd

• Netword was connected for updates
• Create account

Install basic tools

• Install SSH server

  sudo apt update && sudo apt upgrade
  sudo apt install openssh-server
  sudo systemctl enable --now ssh
  

  • edit /etc/ssh/sshd_config:

    • port 2222
    • enable public key
    • disable password

  • on router

    • pin 10.10.10.33 to twiddlebug MAC address
    • port forward TCP 2689 to 10.10.10.33:2222

  • Add public key to ~/.ssh/authorized keys (create file)

  • chmod og-rw ~/.ssh/authorized keys (restrict r/w access to user only)

• Install GIT

  sudo apt install git git-lfs
  git lfs install # (valid forever for user tbuser)
  

• copy .inputrc and .screenrc from gamma to ~ to allow reverse searches with up/down arrow and provide a default screen setup

• Other tools

  • sudo apt install htop vim net-tools
  • net-tools needed for ifconfig

Now, you can either follow the casper tutorial until you hit an error then come back here, or read on.

Vivado

Install Vivado 2022.2

• In Browser, google Xilinx downloads, download web installer for 2022.2 for ubunto (.bin file) in ~/Downloads

• cd Downloads

• chmod +x Xilinx_Unified...bin # make executable sudo ./Xilinx_Unified...bin # start installing

  • (previously tried to execute without sudo, but the /tools/Xilinx folder was not writable and had to restart install)

  • ENter Xilinx username/password

    • Instal Vitis
    • Tick Vitis IP cache
    • Tick I agree 7x

  • download takes 1h, final (3rd) install step stalls half way. Ctrl-C on script. /tools/Xilinx/2022.2/settings64.sh is not present. Install was incomplete?

• Chech install notes on UG973.

  • download signature gpg --import ./xilinx-master-signing-key.asc gpg --list-keys gpg --edit-key 0x<enter key number> gpg> prompt: trust, 5 (ultimately), q (quit) gpg -v --verify Xilinx_Unified...sig Xilinx...bin

    • seems ok

    • restart script with sudo

    • Not happy because folder exists. Install update is done within vivado (not useful). Have to specify another folder for full reinstall.

    • Decide to ask to download the files first so we don't have to download them again on next steps. The full 83 GB downloads,,, 1h, in folder /tools/XIlinx/Downloads/2022.2

    • while this is downloading, looking for answers on web. Similar problem: https://support.xilinx.com/s/question/0D52E00006hpQNASA2/vivado-installation-got-stuck-says-generating-installed-devices-list?language=en_US

    • looking at logs in ~/.xilinx/xinstall. Nothing obvious. FOllowing post:

    https://support.xilinx.com/s/question/0D52E00006hpmTmSAI/vivado-20183-final-processing-hangs-at-generating-installed-device-list-on-ubuntu-1904?language=en_US

    ... installing

    sudo apt install libtinfo5
    sudo apt install libncurses5
    

    then restart the vivado install. It redownloads everything (1h), but then it completes successfully...

    We have not installed the JTAG drivers yet

• Display problem

  • JFC: while running Vivado 2022.2, labels are not displayed in schematics and device cells. When starting Vivado, we have the following message: "Failed to load module "canberra-gtk-module" we fix this by installing

    sudo apt install libcanberra-gtk-module libcanberra-gtk3-module

    restart vivado, the error is gone, and labels display properly.

Final Vivado 2022.2 installation solution

sudo apt install libtinfo5 libncurses5 libcanberra-gtk-module libcanberra-gtk3-module

download .bin file from Xilinx downloads

chmod a+x <downloaded .bin file>;
./<downloaded .bin file>

Installing Vivado 2021.1, is exactly the same as 2022.2 (see above)

• start with . /tools/Xilinx/Vivado/2021.1/settings64.sh vivado
• JFC: created an example design (CPU). Synthesis found license from previous install. Synthesis and implementation worked. Good to go.

Installing Vivado license

• start vivado
• go to help->license manager
• click obtain license -> connect now
• Web browser opens. Enter Matt's credentils
• Select account. Select current license for full Vivado. generate node-locked license
• run ifconfig to see network interfaces MAC addresses. Wired ethernet is 74:56:3c:64:5a:5d
• Add host: twiddlebug, MAC address: use the wired ethernet 74:56:3c:64:5a:5d
• ok, then license is shown and is selected. In the bottop, click tiny download arrow to download license (to browser dowload folder by default).
• in license manager, click load license, and selected downloaded license
• check with View License status
• done

Start Vivado

. /tools/Xilinx/Vivado/2022.2/settings64.sh &&
vivado

To pin vivado to the fast Pcores:

tasksel 0ff vivado # note: first digit is zero, not 'o'

Remote desktop access

• native remote desktop is enabled in settings->sharing

• This works well over a RDP client (used the Windows one), but has serious limitations

  • The user must be logged in locally (!). This can be alleviated by enabling auto-login.
  • Connection is rejected of screen is locked by screen saver! Solved by using a gnome shell extension
    • gse no longer works from the browser on 22.04! Install manually (https://linuxhint.com/install-gnome-shell-extensions-ubuntu-22-04/) sudo apt install gnome-shell-extension-manager launch the "Extension manager" graphical app go to browse, search for "remote", install "AllowedLockedRemoteDesktop"
  • The password changes every time (because the RDP password is in the keyring and the keyring is not unlocked when the user service starts). A solution is to create a non-encrypted keyring just for rdp.

• An alternative is to install xdrp. The limitations are:

  • works ONLY with no used logged in locally

• xfce

• vnc https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-22-04 https://www.howtoforge.com/how-to-install-vnc-server-ubuntu-22-04/

• x2go? references: https://bytexd.com/ubuntu-remote-desktop/

• install xrdp

  • Followinghttps://www.digitalocean.com/community/tutorials/how-to-enable-remote-desktop-protocol-using-xrdp-on-ubuntu-22-04

  	sudo apt install xfce4 xfce4-goodies
  	sudo apt install xrdp
  	sudo systemctl status xrdp
  

  • error messages because port is in use (by native Ubuntu sharing)

  We edit the config /etc/xrdp/xrdp.ini

  set port to port=3390

  sudo systemctl restart xrdp
  sudo systemctl status xrdp
  

  Now seems happy.

  The following is for enabling the xfce4 lightweight desktop manager. It works, but I find this is not necessary. If we don't create the .xsession file, xrdp shows us a gnome desktop manager, which is less confusing, and performance is ok. had to enable all 3 build-in gnome-extensions, but that may be because it was a new user.

    To enable lightweight desktop for xrdp (optional)
    	in the user home folder:
  
  ```
    	echo "xfce4-session" | tee .xsession
  ```
  
  
    	create .xsession for new user
  
    ```
    		su jfcliche
    		echo "xfce4-session" | tee .xsession
    ```
  

  • JFC: after some use, I find this works well, even from home. I can re-log in to existing desktop sessions from differnet computers, but the only downside is that I can't log in from the computer itself (if I do so it crashed the existing sessions and all applications are closed).

• VNC server

  JFC: installed vnc server. Got it to work somewhat, but now getting blank screen. The idea would be to find a way to access an already-locally-opened desktop session (such as tbuser)

Pinning pcores to apps

• JFC: We have 8 fast Pcores and 24 slow Ecores. Vivado seems to use core indiscrimitely. WOuld be nice if vivato stuck to Pcores.

• trying https://www.xmodulo.com/run-program-process-specific-cpu-cores-linux.html

• turns out we dont have to sudo apt-get install util-linux, it's already installed, and we already have tasksel.

• In fact, we can set affinity of specific processes in the build in gnome system monitor (right-click on task). But Vivado creates nes tasks with new PIDs all the time for synthesis, implementation etc.

• The solution is simple: specify the tasks wen launching vivado

• tasksel 0xffff vivado # 0ffff means allow all first 16 hyperthreading cores, or 8 physical cores

• Looking at system monitor, seems to work.

Other utils

• Although it's uncertain whether we need the two following utils, we installed them

sudo add-apt-repository ppa:rock-core/qt4
sudo apt install libqtcore4 libqtgui4

• i7z - show core frequencies sudo apt install i7z sudo i7z

• Stress test

stress-ng --matrix 0

• Other stress test (not installed nor tested) sudo apt install stress stress --cpu 8 --io 4 --vm 4 --vm-bytes 1024M --timeout 10s

Matlab

Installing Matlab 2021a

• Download matlab from website
• Extract to downloads folder, subfolder called mlab2021a
• sudo ./install
• Enter Stephen's McGill uname (email) and pswd
• agree T&C
• select licence licence 'MATLAB (individual)', (this should be valid through to 2027), this didn't work
  • select the one labelled 'stand alone'
• select destination folder /tools, I think this makes it accesible to all users, (default was /usr/...)
• selected products MATLAB, Simulink, and Signal Processing Toolbox from the dropdown list
• create symbolic links in /usr/local/bin

CasperFPGA

• Install python3.8
  • NB: python 3.10 may break code

    sudo apt update
    sudo apt install software-properties-common
    sudo add-apt-repository ppa:deadsnakes/ppa
    sudo apt install python3.8
  

  • now check with: python3.8 --version
• Make a venv
  • First, you need venv, install with sudo apt install python3.8-venv
    • You may also need to curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
    • And install with python3.8 get-pip.py
  • Then create a venv python3.8 -m venv ./cfpga_venv
• Activate venv
  • source ./cfpga_venv/bin/activate
• Packages we installed
  • sudo apt install build-essentials
  • sudo apt-get install python3.8-dev

Set up the Toolflow

• clone the branch m2021a of git clone -b m2021a https://github.com/casper-astro/mlib_devel
• then cd into mlib_devel
• make sure you are in the venv we set up earlier (install casperfpga)
• pip3.8 install -r requirements.txt
• copy the config file cp startsg.local.example startsg.local
• modify it, making sure all of these paths point to the correct places
  • MATLAB R2021a
  • Xilinx Vivado 2021.1
  • Xilinx Composer path
  • Path to venv, also, uncomment this line

Now, we encounter an issue launching Matlab with CASPER toolflow. If we launch Matlab with ./startsg, we encounter the following error in matlab command-line window:

Starting Model Composer
Caught "std::exception" Exception message is:
Error loading /tools/mlabR2021a/bin/glnxa64/builtins/sl_main/mwlibmwsimulink_builtinimpl.so. /lib/x86_64-linux-gnu/libhogweed.so.6: undefined symbol: __gmpn_cnd_sub_n: Success: Success
Vitis Model Composer: Block Reference User Guide Tutorial Examples
Provide feedback for Vitis Model Composer.
INFO: Could not find Vitis installation. For AI Engine development, Vitis must be installed.

Furthermore, we can't run Simulink properly (it will open, but will have no tool tabs, etc.).

We got this error because linux shared libraries are being shadowed by Xilinx's own versions. Crucially, Vivado and Vivado Model Composer's versions of libgmp.so.10 fails to define the variable __gmpn_cnd_sub_n which causes Ubuntu's libhogweed.so.6 to crash, which is needed for Simulink's menubar GUI to operate.

The solution is to replace all of Xilinx's libgmp.so.10 with Ubuntu's version. To find them search with

cd $XILINX_PATH
find . -name libgmp.so.10

There should be (at least) two of them, one of them in a Vivado subdirectory (for us it was /tools/Xilinx/.../Vivado), and another in a Model Composer subdirectory.

Then, instead of deleting Xilinx's versions just move them to a subdirectory called exclude (in case we want to restore them later for debugging) and then replace with the system's version: (you might want to check that you have the right version of libgmp)

cd <path to one of xilinx's version of libgmp>
mkdir exclude
mv libgmp.so.10 exclude/
cp /lib/x86_64-linux-gnu/libgmp.so.10 ./

For good measure enter sudo apt update.

Now pray to the CASPER gods, then cd into mlib_devel and execute ./startsg, if you didn't add a virtual environment path in your startsg_local, make sure your venv is activated when you execute startsg.

Open Simulink, open a model, and compile it.

AAAAAAAAHHHHHHHH it WORKED!

References

• A good resource on a similar error that helped us debug (blogpost): https://strath-sdr.github.io/tools/matlab/sysgen/vivado/linux/2021/01/28/sysgen-on-20-04.html

Appendix

We thought it prudent to include the errors we were getting for anyone struggling with similar issues.

We don't have $XILINX_VIVADO, but I'm assuming this refers to $XILINX_PATH. So, we first do

cd $XILINX_PATH/lib/lnx64.o/Ubuntu/

• however, if we instead launch matlab with

  matlab

we can initialize matlab. We get the printout

Starting Model Composer

with some errors for missing libraries we don't care about. We can then launch simulink and access the casper libraries.

When dropping in, for example, an fft block and setting its parameters, we get the following when we hit "apply":

Error in 'untitled/fft': Failed to evaluate mask initialization commands.
Caused by:
	Error in 'untitled/fft/fft_biplex': Failed to evaluate mask initialization commands.
	Caused by:
		Error in 'untitled/fft/fft_biplex/even_bussify': Failed to evaluate mask initialization commands.
		Caused by:
			There is no block named 'xbsIndex_r4/Concat'
			Caused by:
				Unable to load block diagram 'xbsIndex_r4'

The library xbsIndex_r4 is mentioned on a link on the casper website at

https://casper.astro.berkeley.edu/wiki/images/1/10/XBlock_doc.pdf

but it is unclear if this is a Xilinx built-in library or a casper library. The pdf is from Xilinx, so it's possible that it's a built-in coming from Vivado. In that case, maybe the pointer to the library is incorrect, or we simply don't have the library.

I also find that when I load example FFT systems (either those provided by casper or JF's CHIME FFT), there are lots of blocks instantiated in the model with the error

xbsIndex_r4 not found

written on these blocks. So it's clearly a crucial dependency for several types of blocks.

I'm also seeing the following at https://www.mathworks.com/matlabcentral/answers/13580-warning-unable-to-load-block-diagram-xbsindex_r4-in-matlab-2007a:

"The library model xbsIndex_r4.mdl belongs to the Xilinx System Generator blockset"

This answer was referring to matlab 2007a, so things have certainly changed since then. But maybe it's as simple as installing this library.

Here's the relevant "Introduction to Simulink" casper page:

https://casper-toolflow.readthedocs.io/projects/tutorials/en/latest/tutorials/roach/tut_intro.html

This seems to imply that we should have the Xilinx System Generator blockset by default when launching matlab with ./startsg - so it's probably better to get that working instead of using $ matlab to start matlab.

Resource (blog) for debugging Matlab + Vivado launch https://strath-sdr.github.io/tools/matlab/sysgen/vivado/linux/2021/01/28/sysgen-on-20-04.html

Debugging matlab error raised when run ./startsg

• It turns out that Xilinx Vivado 2021.1 is using qt4, which was discontinued as of Ubuntu 20.04 and cannot easily be found in apt package...
  • I installed qt4 dependencies by adding an archived repo to apt with sudo add-apt-repository ppa:ubuntuhandbook1/ppa
  • Then i installed the qt4, a legacy graphics backend thing, it didn't help...
• The error raised by MATLAB when we boot it with CASPER's startsg script is

Error loading /tools/mlabR2021a/bin/glnxa64/builtins/sl_main/mwlibmwsimulink_builtinimpl.so. /lib/x86_64-linux-gnu/libhogweed.so.6: undefined symbol: __gmpn_cnd_sub_n: Success: Success

• So I looked at the dependencies with ldd libhogweed.so.6 and found this file called libgmp.so.10, a system file, which has "gmp" in it's name, a smoking gun. We can see that /lib/x86_64-linux-gnu/libgmp.so.10 defines the symbol __gmpn_cnd_sub_n by running this command readelf -s libgmp.so.10 | grep __gmpn_cnd_sub_n and it returns "69: 0000000000062a50 310 FUNC GLOBAL DEFAULT 14 __gmpn_cnd_sub_n". This error is not raised when we boot MATLAB on it's own, I suspect CASPER is re-routing libgmp.so.10 to point to a Xilinx version of the same file. Sure enough, when we go to the Xilinx Vivado 2021.1 folder, we find the same dependency with the name with a reccursive search find . -name libgmp.so.10. With the same search command, on the Xilinx file (instead of the system version) readelf -s libgmp.so.10 | grep __gmpn_cnd_sub_n returns nothing. So it seems like Xilinx's version of this file fails to define this symbol.
• As an attempt at a quick fix, I moved all (of Xilinx's) libgmp files into a subdirectory I created called "exclude", and copied over the linux system's libgmp.so.10 into the Xilinx directory. But that failed.
  • Then I returned all libgmp files into Xilinx's directory (from the exclude/), but that failed with the same message.
• I just noticed another error on Matlab startup. When I run ./startsg, (on top of the error in Matlab's terminal mentioned above), an error of a similar nature is raised in the terminal output.

Configuring MATLAB runtime using Xilinx Supplied .matlab7rc.sh file copied to your home directory
MATLAB is selecting SOFTWARE OPENGL rendering.
Gtk-Message: 12:32:52.116: Failed to load module "canberra-gtk-module"
/tools/mlabR2021a/bin/glnxa64/cef_helper: symbol lookup error: /lib/x86_64-linux-gnu/libgnutls.so.30: undefined symbol: __gmpz_limbs_write
"
- What could this mean?
	- `ldd /lib/x86_64-linux-gnu/libgnutls.so.30` lists both libhogweed.so.6 and libgmp.so.10 as dependencies. 
		- It really seems to me that Xilinx's version of libgmp.so.10 is the culprit... or perhaps we need to change the system's libhogweed.so.6 to not use these __gmpz variables... 
	- I wonder if Xilinx has it's own libhogweed... `find . -name "libhogweed*"` in the Vivado2021.1 folder returns no matches. 

- Okay, new track, how about pointing to the newer version of Xilinx in startsg.local, version 2022.2... Same exact error. Including Simulink gui not working. 
- Lets try a different version of Matlab. I installed and tried mlab version R2022b, Matlab crashed on launch. 
- Moved system libhogweed.so.6 into /exclude. When I do this, the error message in Matlab says it can't open libhog weed, now I am unable to open Simulink. That makes sense. It seems libhogweed handles GUI stuff that matlab uses for it's gui. The only problem is that XilinxVivado has its own ideas about what should be done with libhogweed. 
- Just did a search in the next computer, which is running Ubuntu 20.04.5. I found `libhogweed.so`, `libhogweed.so.6`, `libhogweed.so.6.4` and `libgmp.so`, `libgmp.so.10`, `libgmp.so.10.4.1`, `libgmpxx.so`, `libgmpxx.so.4`, `libgmpxx.so.4.6.1`, all inside of a snap library `/snap/gnome42-2204/56/usr/lib/x86_64-linux-gnu/`. I found similar files in a similar subdirectory of `/snap`
	- I looked at those files and found that, unlike the xilinx versions, they do define the variable __gmpn_cnd_sub_n

Renewed efforts
- Put the system `libgmp.so.10` into Xilinx Model_composer 2021.1's `lib/lnx64.o/Ubuntu` this time. Excluding only the file that was in that repository. The thinking is that perhaps they all raise the same error. AAAAAAAAHHHHHHH it worked.  

</details>