Process data exported from the C-Therm TCi thermal conductivity analyzer.

$process-ctherm-tci-data.md

process_data.py

Process data exported from the C-Therm TCi thermal conductivity analyzer.

Usage

If you already have Python 3.11 and the Python launcher installed on Windows/Linux/MacOS, you may run process_data.py with pipx by first downloading it (or copy/pasting its contents into a locally-saved script) and running e.g.

py -3.11 -m pipx run process_data.py

In this case, pipx knows what packages process_data.py needs from inline metadata specified in a comment. You may need to install pipx and ensure that its packages are available in your PATH with

py -3.11 -m pip install pipx
py -3.11 -m pipx ensurepath

Python 3.11 or higher and the Python can be installed on Windows, Linux, or MacOS. If on Windows, install Python from here rather than from the more limited Microsoft Store install. If on MacOS or a UNIX-like system, install the Python Launcher or obtain different Python versions as needed from deadsnakes.

Data model

The structure of data.xml, exported from the C-Therm TCi instrument, is such that all data are contained within the NewDataSet node. Each test run is associated with a UserTest. This record stores the user, method, project, material group (indirectly), material, and lot. The relationships between nodes of interest are as follows:

• Each UserTest has a DataPool_id.
• The DataIntervals with matching DataPool_id have a DataInterval_id.
• A DataSample with matching DataInterval_id has the data for a single measurement.

This record stores the time, validity, ambient temperature, effusivity, and thermal conductivity. Additional fields may be obtained from the data by reference to this procedure.

Advanced usage

If you'd like to run tests, modify this script in-place, or develop it further, you may want to follow developer setup steps. Clone or download this Gist and run setup.ps1 in cross-platform PowerShell terminal window. If you open the downloaded Gist folder in VSCode, open a new terminal (Ctrl+`) and run setup.ps1 there. If on Windows, you may need to complete Task 1 in this guide to allow scripts to run. The setup.ps1 script essentially does the following:

• Updates .gitignore'd tooling from the template repo
• Installs uv if needed
• Determines the Python version to install from process_data.py inline metadata
• Synchronizes a virtual environment with the dependencies for process_data.py from its inline metadata

See a more in-depth guide for first-time setup instructions on a new machine, or if this is your first time using Python. Now you can run Python scripts in this Gist, in an activated terminal window, for example like python example.py. If you plan to modify this code or use it as a starting point for your own development, the template from which this Gist is derived details the additional VSCode tooling available.

.gitignore

__*
.*
pyproject.toml
!.gitignore

compile.py

"""Write compiled requirements to `requirements.txt` from inline script metadata."""

from datetime import UTC, datetime
from pathlib import Path
from platform import platform
from re import MULTILINE, finditer, search
from shlex import split
from subprocess import run
from sys import executable
from tomllib import loads
from typing import Any

SCRIPT = Path("process_data.py")
"""Path to the script to read inline metadata from."""
REQUIREMENTS = Path("requirements.txt")
"""Path to requirements for development of this script."""
PLATFORM = platform(terse=True).casefold().split("-")[0]
"""Platform identifier."""
DEV_REQUIREMENTS = ["pytest==8.1.1"]


def main():
    if metadata := read(Path(SCRIPT).read_text(encoding="utf-8")):
        REQUIREMENTS.write_text(
            "\n".join([
                f"# This file was autogenerated by `{Path(__file__).name}`.",
                f"# Compiled requirements sourced from inline metadata in `{SCRIPT}`.",
                f"# Requirements compiled on platform `{PLATFORM}`.",
                compile(metadata),
            ])
        )


def compile(metadata: dict[str, Any]) -> str:  # noqa: A001
    """Compile requirements from inline metadata.

    Args:
        metadata: Inline metadata.

    Returns:
        Compiled requirements.
    """
    py = m.group() if (m := search(VER, str(metadata.get("requires-python")))) else None
    sep = " "
    result = run(
        input="\n".join([*metadata.get("dependencies", []), *DEV_REQUIREMENTS]),
        args=split(
            sep.join([
                f"{Path(executable).as_posix()} -m uv",
                "pip compile --resolution lowest-direct",
                f"--exclude-newer {datetime.now(UTC).isoformat().replace('+00:00', 'Z')}",
                *([f"--python-version {py}"] if py else []),
                "-",
            ])
        ),
        capture_output=True,
        check=False,
        text=True,
    )
    if result.returncode:
        raise RuntimeError(result.stderr)
    return result.stdout


NAME = "script"
"""Name of the inline metadata block to read."""
VER = r"([1-9][0-9]*!)?(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*))*((a|b|rc)(0|[1-9][0-9]*))?(\.post(0|[1-9][0-9]*))?(\.dev(0|[1-9][0-9]*))?"
"""Canonical version specifier.

See: https://packaging.python.org/en/latest/specifications/version-specifiers/#public-version-identifiers"""


def read(script: str) -> dict[str, Any] | None:
    """Read inline metadata from script contents.

    Args:
        script: Contents of the script to read inline metadata from.

    Returns:
        Inline metadata if found.

    See: https://packaging.python.org/en/latest/specifications/inline-script-metadata/#reference-implementation"""
    matches = [
        m
        for m in finditer(
            r"^# /// (?P<type>[a-zA-Z0-9-]+)$\s(?P<content>(^#(| .*)$\s)+)^# ///$",
            script,
            flags=MULTILINE,
        )
        if m.group("type") == NAME
    ]
    if len(matches) > 1:
        raise ValueError(f"Multiple {NAME} blocks found")
    elif len(matches) == 1:
        content = "".join(
            line[2:] if line.startswith("# ") else line[1:]
            for line in matches[0].group("content").splitlines(keepends=True)
        )
        return loads(content)
    else:
        return None


if __name__ == "__main__":
    main()

LICENSE

MIT License

Copyright (c) 2023

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

process_data.py

# /// script
# requires-python = ">=3.11"
# dependencies = ["parsel>=1.8.1"]
# ///
"""Process data exported from the C-Therm TCi thermal conductivity analyzer."""

from csv import DictWriter
from pathlib import Path

from parsel import Selector

INPUT = Path("data.xml")
"""Test results data exported from the C-Therm TCi instrument."""

OUTPUT = Path("data.csv")
"""Output file."""

# Shorthand for ID fields used frequently in XPath expressions
MAT_ID = "Material_id"
MATGRP_ID = "Material_Group_id"
POOL_ID = "DataPool_id"
INT_ID = "DataInterval_id"


def main():
    root = Selector(text=INPUT.read_text(encoding="utf-8"), type="xml")
    data = root.xpath("/NewDataSet")
    records: list[dict[str, str]] = []
    for test in data.xpath("UserTest"):
        mat = data.xpath(f"Material[ {MAT_ID} = {cval(test, MAT_ID)} ]")
        mat_grp = data.xpath(f"Material_Group[ {MATGRP_ID} = {cval(mat, MATGRP_ID)} ]")
        test_record = {
            "User": cval(test, "User_nm"),
            "Method": cval(test, "TestMethod_nm"),
            "Project": cval(test, "Product_nm"),
            "Material Group": cval(mat_grp, "Material_Group_nm"),
            "Material": cval(test, "Material_nm"),
            "Lot": cval(test, "Material_Batch_nm"),
        }
        for interv in data.xpath(f"DataInterval[ {POOL_ID} = {cval(test, POOL_ID)} ]"):
            sample = data.xpath(f"DataSample[ {INT_ID} = {cval(interv, INT_ID)} ]")
            records.append(
                test_record
                | {
                    "Time": cval(sample, "Sample_Start_dt"),
                    "Valid": (
                        "true"
                        if all(
                            cval(elem, "Valid_yn").casefold() == "y"
                            for elem in (interv, sample)
                        )
                        else "false"
                    ),
                    "Ambient temperature (°C)": cval(sample, "AmbientTemp_nb"),
                    "Effusivity (W-s^(1/2)/m^2-K)": cval(sample, "Result_nb"),
                    "Thermal conductivity (W/m-k)": cval(sample, "K_Result_nb"),
                }
            )
    with open(OUTPUT, "w", encoding="utf-8", newline="") as file:
        writer = DictWriter(file, fieldnames=records[0].keys())
        writer.writeheader()
        writer.writerows(records)


def cval(elem: Selector, child: str) -> str:
    """Value of an element's child."""
    return elem.xpath(f"{child}/text()").get() or ""


if __name__ == "__main__":
    main()

requirements.txt

# This file was autogenerated by `compile.py`.
# Compiled requirements sourced from inline metadata in `process_data.py`.
# Requirements compiled on platform `windows`.
# This file was autogenerated by uv via the following command:
#    uv pip compile --resolution lowest-direct --exclude-newer 2024-03-19T00:10:47.230541Z --python-version 3.12 -
colorama==0.4.6
    # via pytest
cssselect==1.2.0
    # via parsel
iniconfig==2.0.0
    # via pytest
jmespath==1.0.1
    # via parsel
lxml==5.1.0
    # via parsel
packaging==24.0
    # via
    #   parsel
    #   pytest
parsel==1.8.1
pluggy==1.4.0
    # via pytest
pytest==8.1.1
w3lib==2.1.2
    # via parsel

setup.ps1

<#.SYNOPSIS
Sync Python dependencies.#>
Param([switch]$Compile)

$UV_VERSION = '0.1.22'
# ? Fail early
$ErrorActionPreference = 'Stop'
$PSNativeCommandUseErrorActionPreference = $true
$PSNativeCommandUseErrorActionPreference | Out-Null
# ? Source Python version
$ver_pattern = '^# requires-python = ">=(.+)"$'
$re = Get-Content 'process_data.py' -ErrorAction Ignore | Select-String -Pattern $ver_pattern
$Version = $re.Matches.Groups[1].value
$Version = $Version ? $Version : '3.12'

function Sync-Py {
    <#.SYNOPSIS
    Copy the template and update dependencies in a Python virtual environment.#>
    '***SYNCING' | Write-PyProgress
    'SYNCING PROJECT FROM TEMPLATE' | Write-PyProgress
    $tmp = New-TemporaryFile
    $tmpdir = "$($tmp.Directory)/$($tmp.BaseName)"
    git clone --depth 1 'https://github.com/blakeNaccarato/gist-template.git' $tmpdir
    $template = "$tmpdir/template"
    Get-ChildItem -File "$template/*" | Move-Item -Force
    if (! (Test-Path '.vscode')) { New-Item -ItemType Directory '.vscode' }
    Get-ChildItem -File "$template/.vscode/*" | Move-Item -Destination '.vscode' -Force
    $py = Get-Py $Version
    'INSTALLING UV' | Write-PyProgress
    Invoke-Expression "$py -m pip install uv==$UV_VERSION"
    if ($Compile) {
        'COMPILING' | Write-PyProgress
        Invoke-Expression "$py -m compile"
        'COMPILED' | Write-PyProgress -Done
    }
    'INSTALLING DEPENDENCIES' | Write-PyProgress
    Invoke-Expression "$py -m uv pip sync requirements.txt"
    '...DONE ***' | Write-PyProgress -Done
}

function Get-Py {
    <#.SYNOPSIS
    Get Python interpreter.#>
    Param([Parameter(ValueFromPipeline)][string]$Version)
    begin { $venvPath = '.venv' }
    process {
        $GlobalPy = Get-PyGlobal $Version
        if (!(Test-Path $venvPath)) {
            "CREATING VIRTUAL ENVIRONMENT: $venvPath" | Write-PyProgress
            Invoke-Expression "$GlobalPy -m venv $venvPath"
        }
        $VenvPy = Start-PyEnv $venvPath
        $foundVersion = Invoke-Expression "$VenvPy --version"
        if ($foundVersion |
                Select-String -Pattern "^Python $([Regex]::Escape($Version))\.\d+$") {
            "SYNCING VIRTUAL ENVIRONMENT: $(Resolve-Path $VenvPy -Relative)" |
                Write-PyProgress
            return $VenvPy
        }
        "REMOVING VIRTUAL ENVIRONMENT WITH INCORRECT PYTHON: $Env:VIRTUAL_ENV" |
            Write-PyProgress -Done
        Remove-Item -Recurse -Force $Env:VIRTUAL_ENV
        return Get-Py $Version
    }
}

function Get-PyGlobal {
    <#.SYNOPSIS
    Get global Python interpreter.#>
    Param([Parameter(Mandatory, ValueFromPipeline)][string]$Version)
    process {
        if ((Test-Command 'py') -and
        (py '--list' | Select-String -Pattern "^\s?-V:$([Regex]::Escape($Version))")) {
            return "py -$Version"
        }
        elseif (Test-Command "python$Version") { return "python$Version" }
        elseif (Test-Command 'python') { return 'python' }
        throw "Expected Python $Version, which does not appear to be installed. Ensure it is installed (e.g. from https://www.python.org/downloads/) and run this script again."
    }
}

function Start-PyEnv {
    <#.SYNOPSIS
    Activate and get the Python interpreter for the virtual environment.#>
    Param([Parameter(Mandatory, ValueFromPipeline)][string]$venvPath)
    process {
        if ($IsWindows) { $bin = 'Scripts'; $py = 'python.exe' }
        else { $bin = 'bin'; $py = 'python' }
        Invoke-Expression "$venvPath/$bin/Activate.ps1"
        return "$Env:VIRTUAL_ENV/$bin/$py"
    }
}

function Write-PyProgress {
    <#.SYNOPSIS
    Write progress and completion messages.#>
    Param([Parameter(Mandatory, ValueFromPipeline)][string]$Message,
        [switch]$Done)
    begin { $Color = $Done ? 'Green' : 'Yellow' }
    process {
        if (!$Done) { Write-Host }
        Write-Host "$Message$($Done ? '' : '...')" -ForegroundColor $Color
    }
}

function Test-Command {
    <#.SYNOPSIS
    Like `Get-Command` but errors are ignored.#>
    return Get-Command @args -ErrorAction 'Ignore'
}

Sync-Py

test_process_data.py

def test_process_data(): ...