Skip to content

Instantly share code, notes, and snippets.

@hansemro
Last active December 20, 2023 05:29
Show Gist options
  • Save hansemro/5f48f4098e59f9db2e34ae25cb0b6ecd to your computer and use it in GitHub Desktop.
Save hansemro/5f48f4098e59f9db2e34ae25cb0b6ecd to your computer and use it in GitHub Desktop.
KC705 LiteX-VexRiscv-DDR demo built with nextpnr-xilinx (router2, single-port LUTRAM support)
#!/usr/bin/env bash
# copy xilinx_kc705.py and this script to demo-projects/litex-ddr-kc705 and run this script to generate a new litex build.
#conda activate litex_env
python3 xilinx_kc705.py --sys-clk-freq 75000000.0 --cpu-type=vexriscv --build --no-compile-gateware
sed -i "s/^\(\(set_property CFGBVS\|set_property CONFIG_VOLTAGE\|set_property DCI_CASCADE\|set_clock_groups\|set_false_path\|set_max_delay\)\)/#\1/" build/xilinx_kc705/gateware/xilinx_kc705.xdc
cp -f build/xilinx_kc705/gateware/xilinx_kc705.v .
cp -f build/xilinx_kc705/gateware/*.init .
cp -f build/xilinx_kc705/gateware/xilinx_kc705.xdc .
/ / (_) /____ | |/_/
/ /__/ / __/ -_)> <
/____/_/\__/\__/_/|_|
Build your hardware, easily!
(c) Copyright 2012-2023 Enjoy-Digital
(c) Copyright 2007-2015 M-Labs
BIOS built on Jun 6 2023 00:52:44
BIOS CRC passed (316757a7)
LiteX git sha1: a1106b99
--=============== SoC ==================--
CPU: VexRiscv @ 125MHz
BUS: WISHBONE 32-bit @ 4GiB
CSR: 32-bit data
ROM: 128.0KiB
SRAM: 8.0KiB
L2: 8.0KiB
SDRAM: 1.0GiB 64-bit @ 1000MT/s (CL-7 CWL-6)
MAIN-RAM: 1.0GiB
--========== Initialization ============--
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Write leveling:
tCK equivalent taps: 24
Cmd/Clk scan (0-12)
|111111111111| best: 0
Setting Cmd/Clk delay to 0 taps.
Data scan:
m0: |10000000000000111111111111| delay: 14
m1: |00000000000001111111111110| delay: 13
m2: |11110000000000000111111111| delay: 17
m3: |11110000000000000111111111| delay: 17
m4: |11111111000000000000111111| delay: 00
m5: |11111111000000000000011111| delay: 00
m6: |11111111111000000000000011| delay: 00
m7: |11111111111000000000000011| delay: 00
Write latency calibration:
m0:0 m1:0 m2:0 m3:0 m4:6 m5:6 m6:6 m7:6
Write DQ-DQS training:
m0: |000000000111111111110000000000000| delays: 13+-04
m1: |000000011111111111100000000000000| delays: 12+-05
m2: |000000000001111111111100000000000| delays: 15+-04
m3: |000000000001111111111110000000000| delays: 16+-05
m4: |111111100000000000000000000000000| delays: 02+-02
m5: |111111000000000000000000000000000| delays: 02+-02
m6: |111111000000000000000000000000000| delays: 02+-02
m7: |111111000000000000000000000000000| delays: 02+-02
Read leveling:
m0, b00: |00000000000000000000000000000000| delays: -
m0, b01: |00000000000000000000000000000000| delays: -
m0, b02: |11111000000000000000000000000000| delays: 01+-01
m0, b03: |00000000000111111000000000000000| delays: 13+-02
m0, b04: |00000000000000000000000111111100| delays: 26+-03
m0, b05: |00000000000000000000000000000000| delays: -
m0, b06: |00000000000000000000000000000000| delays: -
m0, b07: |00000000000000000000000000000000| delays: -
best: m0, b04 delays: 25+-02
m1, b00: |00000000000000000000000000000000| delays: -
m1, b01: |00000000000000000000000000000000| delays: -
m1, b02: |11110000000000000000000000000000| delays: 01+-01
m1, b03: |00000000001111111000000000000000| delays: 13+-03
m1, b04: |00000000000000000000000111111000| delays: 25+-02
m1, b05: |00000000000000000000000000000000| delays: -
m1, b06: |00000000000000000000000000000000| delays: -
m1, b07: |00000000000000000000000000000000| delays: -
best: m1, b03 delays: 12+-02
m2, b00: |00000000000000000000000000000000| delays: -
m2, b01: |00000000000000000000000000000000| delays: -
m2, b02: |11110000000000000000000000000000| delays: 01+-01
m2, b03: |00000000111111110000000000000000| delays: 11+-03
m2, b04: |00000000000000000000011111110000| delays: 24+-03
m2, b05: |00000000000000000000000000000000| delays: -
m2, b06: |00000000000000000000000000000000| delays: -
m2, b07: |00000000000000000000000000000000| delays: -
best: m2, b03 delays: 11+-03
m3, b00: |00000000000000000000000000000000| delays: -
m3, b01: |00000000000000000000000000000000| delays: -
m3, b02: |11110000000000000000000000000000| delays: 01+-01
m3, b03: |00000000011111110000000000000000| delays: 12+-03
m3, b04: |00000000000000000000001111111000| delays: 25+-03
m3, b05: |00000000000000000000000000000000| delays: -
m3, b06: |00000000000000000000000000000000| delays: -
m3, b07: |00000000000000000000000000000000| delays: -
best: m3, b04 delays: 25+-03
m4, b00: |00000000000000000000000000000000| delays: -
m4, b01: |00000000000000000000000000000000| delays: -
m4, b02: |00000000000000000000000000000000| delays: -
m4, b03: |01111111110000000000000000000000| delays: 05+-04
m4, b04: |00000000000001111111111000000000| delays: 18+-04
m4, b05: |00000000000000000000000000111111| delays: 28+-02
m4, b06: |00000000000000000000000000000000| delays: -
m4, b07: |00000000000000000000000000000000| delays: -
best: m4, b04 delays: 18+-04
m5, b00: |00000000000000000000000000000000| delays: -
m5, b01: |00000000000000000000000000000000| delays: -
m5, b02: |00000000000000000000000000000000| delays: -
m5, b03: |00111111100000000000000000000000| delays: 05+-03
m5, b04: |00000000000000111111110000000000| delays: 17+-03
m5, b05: |00000000000000000000000000011111| delays: 29+-02
m5, b06: |00000000000000000000000000000000| delays: -
m5, b07: |00000000000000000000000000000000| delays: -
best: m5, b04 delays: 17+-03
m6, b00: |00000000000000000000000000000000| delays: -
m6, b01: |00000000000000000000000000000000| delays: -
m6, b02: |00000000000000000000000000000000| delays: -
m6, b03: |11111110000000000000000000000000| delays: 03+-03
m6, b04: |00000000000011111110000000000000| delays: 15+-03
m6, b05: |00000000000000000000000000111111| delays: 28+-02
m6, b06: |00000000000000000000000000000000| delays: -
m6, b07: |00000000000000000000000000000000| delays: -
best: m6, b04 delays: 15+-03
m7, b00: |00000000000000000000000000000000| delays: -
m7, b01: |00000000000000000000000000000000| delays: -
m7, b02: |00000000000000000000000000000000| delays: -
m7, b03: |11111110000000000000000000000000| delays: 03+-03
m7, b04: |00000000000011111110000000000000| delays: 15+-03
m7, b05: |00000000000000000000000001111110| delays: 27+-02
m7, b06: |00000000000000000000000000000000| delays: -
m7, b07: |00000000000000000000000000000000| delays: -
best: m7, b04 delays: 15+-03
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
Write: 0x40000000-0x40200000 2.0MiB
Read: 0x40000000-0x40200000 2.0MiB
bus errors: 0/256
addr errors: 0/8192
data errors: 520192/524288
Memtest KO
Memory initialization failed
--============= Console ================--
litex>
#!/usr/bin/env python3
#
# This file is part of LiteX-Boards.
#
# Copyright (c) 2014-2015 Sebastien Bourdeauducq <sb@m-labs.hk>
# Copyright (c) 2014-2020 Florent Kermarrec <florent@enjoy-digital.fr>
# Copyright (c) 2014-2015 Yann Sionneau <ys@m-labs.hk>
# SPDX-License-Identifier: BSD-2-Clause
import os
from migen import *
from litex.gen import *
from litex_boards.platforms import xilinx_kc705
from litex.soc.cores.clock import *
from litex.soc.integration.soc_core import *
from litex.soc.integration.builder import *
from litex.soc.cores.led import LedChaser
from litedram.modules import MT8JTF12864
from litedram.phy import s7ddrphy
from liteeth.phy import LiteEthPHY
from litepcie.phy.s7pciephy import S7PCIEPHY
from litepcie.software import generate_litepcie_software
# CRG ----------------------------------------------------------------------------------------------
class _CRG(LiteXModule):
def __init__(self, platform, sys_clk_freq):
self.rst = Signal()
self.cd_sys = ClockDomain()
self.cd_sys4x = ClockDomain()
self.cd_idelay = ClockDomain()
# # #
self.pll = pll = S7PLL(speedgrade=-2)
self.comb += pll.reset.eq(platform.request("cpu_reset") | self.rst)
pll.register_clkin(platform.request("clk200"), 200e6)
pll.create_clkout(self.cd_sys, sys_clk_freq)
pll.create_clkout(self.cd_sys4x, 4*sys_clk_freq)
pll.create_clkout(self.cd_idelay, 200e6)
platform.add_false_path_constraints(self.cd_sys.clk, pll.clkin) # Ignore sys_clk to pll.clkin path created by SoC's rst.
self.idelayctrl = S7IDELAYCTRL(self.cd_idelay)
# BaseSoC ------------------------------------------------------------------------------------------
class BaseSoC(SoCCore):
def __init__(self, sys_clk_freq=125e6,
with_ethernet = False,
local_ip = "192.168.1.50",
remote_ip = "",
eth_dynamic_ip = False,
with_led_chaser = True,
with_pcie = False,
with_sata = False,
with_video_colorbars = False,
with_video_framebuffer = False,
with_video_terminal = False,
**kwargs):
platform = xilinx_kc705.Platform()
# CRG --------------------------------------------------------------------------------------
self.crg = _CRG(platform, sys_clk_freq)
# SoCCore ----------------------------------------------------------------------------------
SoCCore.__init__(self, platform, sys_clk_freq, ident="LiteX SoC on KC705", **kwargs)
# DDR3 SDRAM -------------------------------------------------------------------------------
if not self.integrated_main_ram_size:
from litedram.common import PHYPadsReducer
self.ddrphy = s7ddrphy.K7DDRPHY(
#PHYPadsReducer(platform.request("ddram"), [0, 1, 2, 3, 4, 5, 6, 7]),
PHYPadsReducer(platform.request("ddram"), [0, 1, 2, 3]),
memtype = "DDR3",
nphases = 4,
sys_clk_freq = sys_clk_freq)
self.add_sdram("sdram",
phy = self.ddrphy,
module = MT8JTF12864(sys_clk_freq, "1:4"),
l2_cache_size = kwargs.get("l2_size", 8192)
)
# Ethernet ---------------------------------------------------------------------------------
if with_ethernet:
self.ethphy = LiteEthPHY(
clock_pads = self.platform.request("eth_clocks"),
pads = self.platform.request("eth"),
clk_freq = self.clk_freq)
self.add_ethernet(phy=self.ethphy, dynamic_ip=eth_dynamic_ip)
if local_ip:
local_ip = local_ip.split(".")
self.add_constant("LOCALIP1", int(local_ip[0]))
self.add_constant("LOCALIP2", int(local_ip[1]))
self.add_constant("LOCALIP3", int(local_ip[2]))
self.add_constant("LOCALIP4", int(local_ip[3]))
if remote_ip:
remote_ip = remote_ip.split(".")
self.add_constant("REMOTEIP1", int(remote_ip[0]))
self.add_constant("REMOTEIP2", int(remote_ip[1]))
self.add_constant("REMOTEIP3", int(remote_ip[2]))
self.add_constant("REMOTEIP4", int(remote_ip[3]))
# SPI Flash --------------------------------------------------------------------------------
#if with_spi_flash:
# from litespi.modules import N25Q128A13
# from litespi.opcodes import SpiNorFlashOpCodes as Codes
# self.add_spi_flash(mode="4x", module=N25Q128A13(Codes.READ_1_1_4), rate="1:1", with_master=True)
# PCIe -------------------------------------------------------------------------------------
if with_pcie:
self.pcie_phy = S7PCIEPHY(platform, platform.request("pcie_x4"),
data_width = 128,
bar0_size = 0x20000)
self.add_pcie(phy=self.pcie_phy, ndmas=1)
# SATA -------------------------------------------------------------------------------------
#if with_sata:
# from litex.build.generic_platform import Subsignal, Pins
# from litesata.phy import LiteSATAPHY
# # IOs
# _sata_io = [
# # SFP 2 SATA Adapter / https://shop.trenz-electronic.de/en/TE0424-01-SFP-2-SATA-Adapter
# ("sfp2sata", 0,
# Subsignal("tx_p", Pins("H2")),
# Subsignal("tx_n", Pins("H1")),
# Subsignal("rx_p", Pins("G4")),
# Subsignal("rx_n", Pins("G3")),
# ),
# ]
# platform.add_extension(_sata_io)
# # RefClk, Generate 150MHz from PLL.
# self.cd_sata_refclk = ClockDomain()
# self.crg.pll.create_clkout(self.cd_sata_refclk, 150e6)
# sata_refclk = ClockSignal("sata_refclk")
# platform.add_platform_command("set_property SEVERITY {{Warning}} [get_drc_checks REQP-52]")
# # PHY
# self.sata_phy = LiteSATAPHY(platform.device,
# refclk = sata_refclk,
# pads = platform.request("sfp2sata"),
# gen = "gen2",
# clk_freq = sys_clk_freq,
# data_width = 16)
# # Core
# self.add_sata(phy=self.sata_phy, mode="read+write")
# Leds -------------------------------------------------------------------------------------
if with_led_chaser:
self.leds = LedChaser(
pads = platform.request_all("user_led"),
sys_clk_freq = sys_clk_freq)
# Build --------------------------------------------------------------------------------------------
def main():
from litex.build.parser import LiteXArgumentParser
parser = LiteXArgumentParser(platform=xilinx_kc705.Platform, description="LiteX SoC on KC705.")
parser.add_target_argument("--sys-clk-freq", default=125e6, type=float, help="System clock frequency.")
parser.add_target_argument("--with-spi-flash", action="store_true", help="Enable SPI Flash (MMAPed).")
parser.add_target_argument("--remote-ip", default="192.168.1.100", help="Remote IP address of TFTP server.")
parser.add_target_argument("--local-ip", default="192.168.1.50", help="Local IP address.")
parser.add_target_argument("--eth-dynamic-ip", action="store_true", help="Enable dynamic Ethernet IP addresses setting.")
parser.add_target_argument("--with-pcie", action="store_true", help="Enable PCIe support.")
parser.add_target_argument("--driver", action="store_true", help="Generate PCIe driver.")
parser.add_target_argument("--with-sata", action="store_true", help="Enable SATA support (over SFP2SATA).")
ethopts = parser.target_group.add_mutually_exclusive_group()
ethopts.add_argument("--with-ethernet", action="store_true", help="Enable Ethernet support.")
sdopts = parser.target_group.add_mutually_exclusive_group()
sdopts.add_argument("--with-spi-sdcard", action="store_true", help="Enable SPI-mode SDCard support.")
sdopts.add_argument("--with-sdcard", action="store_true", help="Enable SDCard support.")
viopts = parser.target_group.add_mutually_exclusive_group()
viopts.add_argument("--with-video-terminal", action="store_true", help="Enable Video Terminal (HDMI).")
viopts.add_argument("--with-video-framebuffer", action="store_true", help="Enable Video Framebuffer (HDMI).")
viopts.add_argument("--with-video-colorbars", action="store_true", help="Enable Video Colorbars (HDMI).")
args = parser.parse_args()
soc = BaseSoC(
sys_clk_freq = args.sys_clk_freq,
with_spi_flash = args.with_spi_flash,
with_ethernet = args.with_ethernet,
local_ip = args.local_ip,
remote_ip = args.remote_ip,
eth_dynamic_ip = args.eth_dynamic_ip,
with_pcie = args.with_pcie,
with_sata = args.with_sata,
with_video_colorbars = args.with_video_colorbars,
with_video_framebuffer = args.with_video_framebuffer,
with_video_terminal = args.with_video_terminal,
**parser.soc_argdict
)
builder = Builder(soc, **parser.builder_argdict)
if args.build:
builder.build(**parser.toolchain_argdict)
if args.driver:
generate_litepcie_software(soc, os.path.join(builder.output_dir, "driver"))
if args.load:
prog = soc.platform.create_programmer()
prog.load_bitstream(builder.get_bitstream_filename(mode="sram"))
if __name__ == "__main__":
main()
@hansfbaier
Copy link

It looks like you have all modules active.
In my experience, openXC7 ususally only can get the timing for a limited number of modules right.
Try PHYPadsReducer and limit to the first module,
then the first two, and so on.

@hansfbaier
Copy link

But what astonishes me a bit here, is that calibration can find good values for all modules.

@hansfbaier
Copy link

I would try restricting to the first four modules. And see if that works.

@hansemro
Copy link
Author

@hansfbaier Good to know. I'll try again soon with PHYPadsReducer.

@hansemro
Copy link
Author

I am unable to get serial output with a newer LiteX build now... Tried both router1 and router2 since that can help in some cases, but without success.

@hansfbaier
Copy link

With the PHYPadsReducer?

@hansemro
Copy link
Author

hansemro commented Dec 20, 2023

Booted successfully. Looks like I still have to use PLL instead of MMCM for now. (LED chaser was not working with MMCM).

@hansemro
Copy link
Author

      __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2023 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Dec 19 2023 20:14:54
 BIOS CRC passed (7c2eb7c5)

 LiteX git sha1: b6e89c64

--=============== SoC ==================--
CPU:            VexRiscv @ 125MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            32-bit data
ROM:            128.0KiB
SRAM:           8.0KiB
L2:             8.0KiB
SDRAM:          256.0MiB 16-bit @ 1000MT/s (CL-7 CWL-6)
MAIN-RAM:       256.0MiB

--========== Initialization ============--
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Write leveling:
  tCK equivalent taps: 24
  Cmd/Clk scan (0-12)
  |111111111111| best: 6
  Setting Cmd/Clk delay to 6 taps.
  Data scan:
  m0: |11111110000000000000111111| delay: 00
  m1: |11111100000000000001111111| delay: 19
Write latency calibration:
m0:6 m1:0 
Write DQ-DQS training:
m0: |111111000000000000000000000000000| delays: 02+-02
m1: |000000000000001111111111100000000| delays: 18+-04
Read leveling:
  m0, b00: |00000000000000000000000000000000| delays: -
  m0, b01: |00000000000000000000000000000000| delays: -
  m0, b02: |00000000000000000000000000000000| delays: -
  m0, b03: |00001111111000000000000000000000| delays: 07+-02
  m0, b04: |00000000000000000111111100000000| delays: 20+-03
  m0, b05: |00000000000000000000000000000001| delays: -
  m0, b06: |00000000000000000000000000000000| delays: -
  m0, b07: |00000000000000000000000000000000| delays: -
  best: m0, b04 delays: 20+-03
  m1, b00: |00000000000000000000000000000000| delays: -
  m1, b01: |00000000000000000000000000000000| delays: -
  m1, b02: |00000000000000000000000000000000| delays: -
  m1, b03: |00011111111100000000000000000000| delays: 07+-04
  m1, b04: |00000000000000001111111100000000| delays: 19+-03
  m1, b05: |00000000000000000000000000000111| delays: 30+-01
  m1, b06: |00000000000000000000000000000000| delays: -
  m1, b07: |00000000000000000000000000000000| delays: -
  best: m1, b03 delays: 06+-03
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB     
   Read: 0x40000000-0x40200000 2.0MiB     
Memtest OK
Memspeed at 0x40000000 (Sequential, 2.0MiB)...
  Write speed: 47.4MiB/s
   Read speed: 62.4MiB/s

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
             Timeout
No boot medium found

--============= Console ================--

litex> 

@hansemro
Copy link
Author

With 4 modules, I run into "main analytical placer" stalling issue.

@hansfbaier
Copy link

@hansemro Using the nix toolchain?

@hansfbaier
Copy link

@hansemro MMCM uses 'magic values'. It looks like I need to weedle those out of vivado for different FPGA types/families.
Thanks for reporting this.

@hansemro
Copy link
Author

Using nix toolchain with my nextpnr fork and NEXTPNR_XILINX_PYTHON_DIR updated (for bba/chipdb generation).

@hansemro
Copy link
Author

Added xilinx_kc705.py and a litex build script.

@hansfbaier
Copy link

@hansemro What is your fork based on? I already corrected the old prjxray-db reference in the upstream nix toolchain.

@hansfbaier
Copy link

@hansemro Ah you use the main branch. Not recommended. Has the router2 lockup issue. stable-backports is the way to go.

@hansemro
Copy link
Author

This is upstream nix toolchain. I just have PATH updated to my experimental nextpnr-xilinx build. prjxray-db is not an issue here.

@hansfbaier
Copy link

What do you need the NEXTPNR_XILINX_PYTHON_DIR modification for? Can you point me to the code of your modification?

@hansfbaier
Copy link

hansfbaier commented Dec 20, 2023

@hansemro Yes, your branch is based on main. main is broken. Can you rebase it on the stable-backports branch? Then you could try the four module version again.

@hansemro
Copy link
Author

hansemro commented Dec 20, 2023

What do you need the NEXTPNR_XILINX_PYTHON_DIR modification for? Can you point me to the code of your modification?

https://github.com/openXC7/demo-projects/blob/cd5e3ebbfc047c1b6970a001a4b8ac9d4709ccaf/openXC7.mk#L35

NEXTPNR_XILINX_PYTHON_DIR is for xc7-lutram-dev build to ensure RAMS32 and RAMS64E are included in the chipdb. Not needed for normal use. Sorry for any confusion.

Addresses the following:

nextpnr-xilinx --router router2 --chipdb ../chipdb//xc7k325tffg900.bin --xdc xilinx_kc705.xdc --json xilinx_kc705.json --fasm xilinx_kc705.fasm
terminate called after throwing an instance of 'nextpnr_xilinx::assertion_failure'
  what():  Assertion failure: The internal IDs of nextpnr are inconsistent with the supplied chip database.

Yes, your branch is based on main.

Don't worry, I am using latest prjxray-db. But sure, I can rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment