Failed attempt at creating a dependency tracker / Makefile generator tool for FPGAs

README

This is my failed attempt at creating a dependency tracker / Makefile generator tool for FPGAs.

Originally I thought that it would be beneficial to create a Makefile that would generate outputs for FPGA hardware development just like is common for traditional software development. FPGA development makes use of more than a dozen command-line tools. The result was pretty poor. Here are some lessons learned:

1. In software, often there is only path to a.out. In FPGA development, there can be multiple paths.

2. The command-line tools have lots and lots of options. Many require one or more (temporary) input files to control the behavior of the tool. These input files are temporary because they need to be modified in order to perform a different action.

3. The Xilinx IDE, called ISE, is actually pretty good. There are problems like trying to recompile everything too much. But there are ways around these problems. Plus it is well documented.

4. Dependency tracking is a nightmare! In software development, there is unit testing, where you just want to see if the code is correct. In hardware development, there are 5 different types of unit testing (called simulation), and the input files to these 5 steps are all different! Naming and tracking all of these dependencies is very, very challenging.

Good luck to anyone that wants to make an FPGA build tool. You have my sympathy.

Cores.ini

; special sections

[INCLUDE]

[FPGA]
device = xc6slx45
devicefamily = spartan6
package = csg324
speedgrade = -2

; hardware cores

[async_4phase_handshake_master]
hdlfile=hdl/async_4phase_handshake_master.v
hdltype=verilog2001
synthesizable=True

[async_4phase_handshake_slave]
hdlfile=hdl/async_4phase_handshake_slave.v
requires=async_muller_c_element
synthesizable=True

[async_muller_c_element]
hdlfile=hdl/async_muller_c_element.v
synthesizable=True

[spi_master]
hdlfile=hdl/spi_master.v
synthesizable=True

[sync_4phase_handshake_master]
hdlfile=hdl/sync_4phase_handshake_master.v
synthesizable=True

[sync_4phase_handshake_slave]
hdlfile=hdl/sync_4phase_handshake_slave.v
synthesizable=True

[sync_signal]
hdlfile=hdl/sync_signal.v
synthesizable=True

[timer]
hdlfile=hdl/timer.v
synthesizable=True

; testbenches

[async_4phase_handshake_master_tb]
hdlfile=tb/async_4phase_handshake_master_tb.v
requires=async_4phase_handshake_master
testbench=True

[async_4phase_handshake_slave_tb]
hdlfile=tb/async_4phase_handshake_slave_tb.v
requires=async_4phase_handshake_slave
testbench=True

[async_muller_c_element_tb]
hdlfile=tb/async_muller_c_element_tb.v
requires=async_muller_c_element
testbench=True

[spi_master_tb]
hdlfile=tb/spi_master_tb.v
requires=spi_master
testbench=True

[sync_4phase_handshake_master_tb]
hdlfile=tb/sync_4phase_handshake_master_tb.v
requires=sync_4phase_handshake_master
testbench=True

[sync_4phase_handshake_slave_tb]
hdlfile=tb/sync_4phase_handshake_slave_tb.v
requires=sync_4phase_handshake_slave
testbench=True

[sync_signal_tb]
hdlfile=tb/sync_signal_tb.v
requires=sync_signal
testbench=True

[timer_tb]
hdlfile=tb/timer_tb.v
requires=timer
testbench=True

FPGAmake

#!/usr/bin/env python
#
# This Python script generates a Makefile to implement the Xilinx workflow using the dependencies recorded in a separate
# Cores.ini file.

import argparse
import ConfigParser
import glob
import os, os.path
import shutil
import stat
import subprocess
import sys

##############################################################################
# TEMPLATES 
##############################################################################

# Parameters:
#   NGCTARGETS
#   NGCRULES
#   XSTRULES
#   XSTPRJRULES
T_MAKEFILE=r"""# WARNING: AUTOMATICALLY GENERATED MAKEFILE. ALL EDITS WILL BE OVERWRITTEN.

# Cross-platform mkdir command.
MKDIR:=python mkdir.py

# NGC targets.
NGC:={NGCTARGETS}

.PHONY:
all: $(NGC)

# All outputs go into the build subdirectory.
# This makes cleanup very easy.
build:
	$(MKDIR) $@

.PHONY:
clean:
	-rm -rf build

# Rules to make Xilinx NGC netlists synthesized with XST.
{NGCRULES}

# Rules to make XST script files.
{XSTRULES}

# Rules to make XST project files.
{XSTPRJRULES}
"""

# Parameters:
#   module
T_NGCRULE="""build/{module}.ngc: build/{module}.xst
	$(MKDIR) build/{module}.xst.tmp
	$(MKDIR) build/{module}.xst.work
	cd build && xst -ifn {module}.xst
	rm -rf build/{module}.xst.tmp
	rm -rf build/{module}.xst.work
	rm -rf build/_xmsgs
	rm -f build/{module}.lso
	mv build/{module}.srp build/{module}.xst.srp
"""

# Parameters:
#   module
T_NGCTARGET="build/{module}.ngc"

# Parameters:
#   module
#   device: FPGA device number
#   package: FPGA package format
#   speedgrade: FPGA speed
#   vlgincdir: Verilog include directory, probably the same directory where the HDL source file is.
T_XSTRULE="""build/{module}.xst: build/{module}.xst.prj
	python mkxst.py $@ {module} {device} {package} {speedgrade} {vlgincdir} 
"""

# Parameters:
#   module
T_XSTTARGET="build/{module}.xst"

# Parameters:
#   module
#   language: (verilog,vhdl)
#   hdlfile: path to HDL file to compile
T_XSTPRJRULE="""build/{module}.xst.prj: build
	python mkxstprj.py $@ {language} {hdlfile}
"""

# Parameters:
#   module
T_XSTPRJTARGET="build/{module}.xst.prj"

##############################################################################
# FUNCTIONS 
##############################################################################

def MAKEFILE(config):
    return T_MAKEFILE.format(
        NGCTARGETS=NGCTARGETS(config),
        NGCRULES=NGCRULES(config),
        XSTRULES=XSTRULES(config),
        XSTPRJRULES=XSTPRJRULES(config))

def NGCTARGET(module):
    return T_NGCTARGET.format(module=module)

def NGCTARGETS(config):
    return '\\\n'.join(NGCTARGET(module) for module in config.modules.synthesizable)

def NGCRULE(module):
    return T_NGCRULE.format(module=module)

def NGCRULES(config):
    return '\n'.join(NGCRULE(module) for module in config.modules.synthesizable)

def XSTRULE(module,device,package,speedgrade,vlgincdir):
    return T_XSTRULE.format(
        module=module,
        device=device,
        package=package,
        speedgrade=speedgrade,
        vlgincdir=vlgincdir)

def XSTRULES(config):
    rules = []
    device = config.fpga.device
    package = config.fpga.package
    speedgrade = config.fpga.speedgrade
    for module in config.modules.synthesizable:
        hdlfile = config.modules[module].hdlfile
        vlgincdir = os.path.dirname(hdlfile)
        rules += [XSTRULE(module,device,package,speedgrade,vlgincdir)]
    return '\n'.join(rules)

def XSTPRJRULE(module,language,hdlfile):
    return T_XSTPRJRULE.format(
        module=module,
        language=language,
        hdlfile=hdlfile)

def XSTPRJRULES(config):
    rules = []
    for module in config.modules.synthesizable:
        hdlfile = config.modules[module].hdlfile
        if hdlfile.endswith('.v'):
            language = 'verilog'
        elif hdlfile.endswith('.vhd'):
            language = 'vhdl'
        else:
            raise Exception('unknown HDL type: %s' % hdlfile)
        rules += [XSTPRJRULE(module,language,hdlfile)]
    return '\n'.join(rules)

##############################################################################
# CLASSES 
##############################################################################

class Config(object):
    "Encapsulates the Cores.ini configuration file."
    def __init__(self, configParser=None):
        if configParser is None:
            configParser = ConfigParser.ConfigParser()
            configParser.read(['Cores.ini'])
        self.fpga = Config.FPGA(configParser)
        self.modules = Config.Modules(configParser)
    class FPGA(object):
        def __init__(self, configParser):
            self.device = configParser.get('FPGA','device')
            self.devicefamily = configParser.get('FPGA','devicefamily')
            self.package = configParser.get('FPGA','package')
            self.speedgrade = configParser.get('FPGA','speedgrade')
    class Modules(dict):
        def __init__(self, configParser):
            self.synthesizable = []
            self.testbench = []
            for module in configParser.sections():
                self[module] = Config.Module(module,configParser)
                if configParser.has_option(module,'synthesizable') and \
                   configParser.getboolean(module,'synthesizable'):
                    self.synthesizable += [module]
                if configParser.has_option(module,'testbench') and \
                   configParser.getboolean(module,'testbench'):
                    self.testbench += [module]
    class Module(object):
        def __init__(self, moduleName, configParser):
            self.name = moduleName
            self.hdlfile = None
            self.synthesizable = False
            self.testbench = False
            if configParser.has_option(moduleName,'hdlfile'):
                self.hdlfile = configParser.get(moduleName,'hdlfile')
            if configParser.has_option(moduleName,'synthesizable'):
                self.synthesizable = configParser.getboolean(moduleName,'synthesizable')
            if configParser.has_option(moduleName,'testbench'):
                self.testbench = configParser.getboolean(moduleName,'testbench')

##############################################################################
# MAIN SCRIPT 
##############################################################################

if __name__=='__main__':
    config = Config()
    with open('Makefile','w') as f:
        makefile = MAKEFILE(config)
        f.write(makefile)

mkdir.py

#!/usr/bin/env python

# Cross-platform mkdir command.

import os
import sys

if __name__=='__main__':
    if len(sys.argv) != 2:
        sys.exit('usage: mkdir.py <directory>')
    directory = sys.argv[1]
    try:
        os.makedirs(directory)
    except OSError:
        pass

mkxst.py

#!/usr/bin/env python

# Makes a Xilinx XST script file.

import os
import sys

TEMPLATE="""set -tmpdir {module}.xst.tmp
set -xsthdpdir {module}.xst.work
run
-ifn {module}.xst.prj
-ofn {module}.ngc
-p {device}-{package}{speedgrade}
-top {module}
-vlgincdir ../{vlgincdir}"""

if __name__=='__main__':
    if len(sys.argv) != 7:
        sys.exit('usage: mkxst.py <output> <module> <device> <package> <speedgrade> <vlgincdir>')
    output = sys.argv[1]
    parameters = dict(
        module=sys.argv[2],
        device=sys.argv[3],
        package=sys.argv[4],
        speedgrade=sys.argv[5],
        vlgincdir=sys.argv[6])
    with open(output,'w') as f:
        f.write(TEMPLATE.format(**parameters))

mkxstprj.py

#!/usr/bin/env python

# Makes a Xilinx XST project file.

import os
import sys

TEMPLATE="{language} work ../{hdlfile}"

if __name__=='__main__':
    if len(sys.argv) != 4:
        sys.exit('usage: mkxstprj.py <output> <language> <hdlfile>')
    output = sys.argv[1]
    parameters = dict(
        language=sys.argv[2],
        hdlfile=sys.argv[3])
    with open(output,'w') as f:
        f.write(TEMPLATE.format(**parameters))