Skip to main content

Data flows

How data moves through Complied: from the outside world into the system, between its modules, and back out. The page starts with the whole system and narrows to each flow, then covers the trust boundaries and the five end-to-end workflows. It is written for developers; for the same workflows from an operator's chair see running a project.

Notation: rounded boxes are external parties, rectangles are processes, cylinders are stores. Arrows carry data, not control.

Context​

Complied does not e-file with HPD. assemble-filing-package makes sure every decided HPD document exists and records a filing_packages row with join rows; a person files the paperwork. Lab results are typed in by staff against the sample rows.

Level 1: the system​

Process 3 is the Decide, Work, File and Money workspace. It is not a gated phase machine: projects.phase is a label the header dropdown writes, and a trigger records the history.

Level 2: ingest​

One edge function, sync-orchestrator, drives one domain loop, runFeedCitywide (domain/src/ingestion/). Eight feeds run, each idempotent on (agency, source_id). The full pipeline is in ingestion.

The feeds are HPD violations, HPD complaints, HPD litigation, DOB violations, ECB violations, OATH hearings, DOF liens and NYC 311. Cron fires one request per feed so each stays inside the platform gateway timeout; a manual run may omit the feed id. After the feeds, the orchestrator refreshes the building rollups and stamps map_data_version, which tells the map exporter to publish a new snapshot. Citywide identity loads (Layer 1) and historical backfill run from CLI scripts against a direct Postgres connection, not from the function. Obligations are not an ingest job: their rules run on read.

Level 2: project workspace​

origin is immutable. Creating from a violation writes project_event_links in the same call. Creating from an obligation writes (or links) the obligations ledger row with the new project_id. There is no UI that links or unlinks events after creation, so an obligation or occupant-request project opens Decide with nothing to decide. The phase label sits beside this pipeline, not in it.

Level 2: field capture and documents​

Every generator renders a PDF, writes it to the documents bucket and inserts one documents row per document code through insertDocument() in _shared/documents.ts. The XRF run writes an INSP-RPT and a separate XRF-AFF; the paint-chip run writes PC-RPT, PC-AFF and PC-LAB. generate-abatement-report exists as an edge function with no frontend caller. HTML-to-PDF rendering goes to the render service (pdf-reports). File layout and access are in documents.

Level 2: money​

A proposal is priced from the derived services and the rate cards, edited on the Proposal card, saved to proposals, then rendered to a PDF as a separate step. An invoice is priced from what was captured (non-calibration XRF readings, samples, abatement components) and the same price lists, not from the proposal. A service with no matching rate card contributes no line, so an invoice can total $0 and still return success. invoices_stamp_billed_client fixes the billed client when the invoice leaves draft; the portal sees only non-draft invoices billed to its client.

Trust boundaries​

  • The browser holds only the anon key and the user's JWT. Everything it reads or writes is filtered by RLS.
  • An edge function that runs as the service role authorizes its caller first: a user JWT that passes has_permission(key) (or is a platform operator), or the service-role bearer. Money functions also require the caller to be in the owner tenant. sync-orchestrator accepts a scoped CRON_SECRET from pg_cron, or a platform operator or service role by hand.
  • Functions that take no ordinary user JWT set verify_jwt = false in supabase/config.toml and check their own credential: sync-orchestrator (cron secret, or an operator or service-role bearer), process-email-outbox and process-notification-digest (service-role bearer), handle-email-suppression (Resend's Svix signature; it fails closed when its secret is unset) and handle-email-unsubscribe (a one-time token). request-password-reset is unauthenticated by necessity, since its caller cannot sign in.
  • setProjectPhase is not an edge function. It is a client update under RLS, and the projects_check_update_permissions trigger requires advance_projects.
  • The map service never holds tenant data. It answers from a snapshot of public rows and fetches the caller's own tenant overlay from Postgres with the caller's JWT (map).

Workflow 1: staff project, violation to commercially done​

The phase dropdown can be set at any point and is not a step in this flow; setting closed does not wait on HPD or on payment and does not make the project read-only. A collaborator tenant with field_execution can sit in the inspector lane for that project only and never enters the money lane. project-walkthrough traces this flow table by table.

Workflow 2: agency ingest​

There is no UI actor: the job runs nightly from pg_cron (08:10 UTC, one request per feed). Quarantined rows can be resolved with resolve_unlinked_event; there is no dedicated page for it. /map and /violations read this data and never sync it.

Workflow 3: client portal​

The portal is read-only and provisioned by staff. A portal user sees a building through tenant_buildings.client_id, a project through its building, a document only when it was shared with that client, and an invoice only when it is not a draft and is billed to that client. It cannot see inspections, rate cards, licenses or vendor payments (proven in the db-tests leakage suite). Portal sign-in uses Supabase Auth. The emailed password reset (request-password-reset) looks up profiles only, so it serves staff logins.

Workflow 4: notifications and email​

Every email enters through enqueue_email(), which only the service role may call. The worker reads queued rows that are due, re-checks suppression, renders the template, takes the sender identity from tenant_branding when set, sends through Resend and logs the attempt; failures retry with backoff up to five attempts. A bounce or complaint webhook, or an unsubscribe link (handle-email-unsubscribe), writes email_suppressions, which is global by address. notify also writes a second notification_log row with channel = 'sms' and status = 'queued' when SMS is enabled; no SMS provider is called. The digest function rolls up the last 24 hours of documents, phase changes and invoices for each recipient whose digest time matches.

Workflow 5: cross-tenant collaboration​

An owner with manage_collaborators invites another tenant onto one project with a role (field_execution, abatement, clearance_sampling or lab_coordination). The project row is visible to any active grant. Field-table writes are role-scoped through has_active_collaboration_grant_for_role(): field_execution reaches inspections, XRF, floor plans and inspection_events; lab_coordination also reaches samples and chain of custody; abatement also reaches abatement_components. A collaborator additionally needs its own permission key (for example perform_field_work) on its own side. Field-report documents are readable by a collaborator whose role is listed on the document type. The 28 RCNY abatement versus clearance independence rule produces a warning only, at invite time and at visit time. Details: permissions.