bogdan

Rename token types and centralize space-aware token joining

Commit: https://github.com/hosmada/emr/commit/2c27eb5381d082e0a9745085d384b8fbe74e83e6

What changed:

• Renamed two Token#token_type values for clarity: word → japanese_word and foreign → latin_word.
• Added SuikaTokenizer.join(tokens), a shared helper that reconstructs text from tokens correctly instead of naively joining with a fixed separator. It only inserts a space between two consecutive "space-delimited" tokens (Latin words, abbreviations, and numbers) — Japanese tokens never get a space.
• Wired Meeting#utterances and ReviewWorkflow#phi_review_data to use this new helper instead of tokens.map(&:text).join(" ").
• Updated the token test factory to auto-classify token_type from text via the real tokenizer instead of hardcoding a default.

bogdan

#!/usr/bin/env ruby
# frozen_string_literal: true

# Speech-to-Text v2 diarization test (chirp_3 model, Japanese by default).
#
# KEY FINDING: chirp_3 + diarizationConfig only works in the "us" multi-region.
# It is rejected everywhere else tested (global, us-central1, europe-west4,
# asia-southeast1, europe-west3) with either "model does not exist" or
# "Permission denied ... It is no longer generally available." Only the
# "us" multi-region endpoint accepts it, and only via the async

bogdan

# syntax=docker/dockerfile:1
# check=error=true;skip=SecretsUsedInArgOrEnv

# This Dockerfile is designed for production, not development. Use with Kamal or build'n'run by hand:
# docker build -t realphotos_marketplace .
# docker run -d -p 80:80 -e RAILS_MASTER_KEY=<value from config/master.key> --name realphotos_marketplace realphotos_marketplace

# For a containerized dev environment, see Dev Containers: https://guides.rubyonrails.org/getting_started_with_devcontainer.html

# Make sure RUBY_VERSION matches the Ruby version in .ruby-version

bogdan

steps:
  # Pull cache images for both build stage and final image (ignore failure on first run)
  - id: "pull-cache"
    name: "gcr.io/cloud-builders/docker"
    entrypoint: /bin/bash
    args:
      - -c
      - |
        docker pull ${_IMAGE_NAME}:build-cache || true && \
        docker pull ${_IMAGE_NAME}:latest || true

bogdan

Meeting 596 PHI Token Analysis

Token(s)	Reading	Context	Assessment
鈴木	Suzuki	鈴木さん こんにちは	Patient surname — PHI
さん	-san	suffix after 鈴木	Honorific — not PHI
鈴木 一郎	Suzuki Ichiro	鈴木一郎と言います (my name is Suzuki Ichiro)	Patient full name — PHI
1989 年 の 10 月 15 日	October 15, 1989	生まれです (born on)	Patient DOB — PHI, but の, 年, 月, 日 are grammatical particles/units
島田	Shimada	島田先生	Doctor's surname — debatable (provider name)
先生	sensei/doctor	title after 島田	Title only — not PHI

bogdan

Third-Party Face Removal from Video

Variants for handling third-party faces that appear in recorded video segments.

---

1. Cut the Segment

Remove the time range entirely from the final export.

bogdan

Morphology — Conversation Token Format

Specification for the JSON data structure produced by speech-to-text extraction from doctor/patient video recordings. The output is a flat, time-ordered array of tokens — the smallest meaningful units of spoken language: words, numbers, and punctuation.

---

Top-level structure

[

bogdan

require 'open3'
require 'json'

module InsightFace
  SCRIPT = Rails.root.join("bin/insightface").to_s
  CMD = if Rails.env.production?
    ["/app/.heroku/python/bin/python3", SCRIPT]
  else
    ["uv", "run", SCRIPT]

bogdan

#!/usr/bin/env python3
"""
Face detection via InsightFace (ONNX-based, no TensorFlow).

Usage:
  bin/insightface <image_path>
  cat image.jpg | bin/insightface

Outputs detected faces as JSON to stdout (same schema as bin/retinaface).

bogdan

How We Keep Our API and Frontend Automatically In Sync

Our backend is a Ruby REST API built with Grape. Our frontend is TypeScript. Keeping them in sync manually is a recipe for silent drift — a renamed field here, a removed param there, and suddenly the frontend is sending requests the server no longer understands or rendering fields that no longer exist.

This post describes the pipeline we built so that the server code is the single source of truth, type mismatches are caught before they ship, and backward incompatibilities surface as compiler errors rather than production bugs. The specific tools are Ruby-flavored, but the pattern is language-agnostic and maps directly to FastAPI, NestJS, or any framework that can generate an OpenAPI spec from code.

---

The Pipeline at a Glance