Complied Map — developer guide

Audience: engineers joining the /map work or shipping it. Status: finalized on branch map-query-rebuild (2026-09-24). Not on production yet. Companion docs: MAP_MODEL.md (product/filter model), MAP_QUERY_BENCHMARK_2026-09-24.md (why DuckDB), map-service/AGENTS.md (service ops + env vars).

This guide is the how it works / where to change what walkthrough. Start here if you need to debug a request, add a filter, or understand why prod must not get the frontend before the service.

---

1. One sentence

The map has one filter object (in the URL). That filter goes to one query service (map-service/, DuckDB over a nightly Parquet copy of open city records). The response has totals, the ranked list, and the pins — all from the same filtered set, so they cannot disagree.

URL facets  →  MapQueryFilter  →  POST /map/query  →  { totals, in_view, ranked, pins }

If those three disagree, something is wrong. Do not "fix" it by filtering only in the browser.

---

2. What this branch finalized

Five commits, in order:

Commit	What it locked in
8cc1ca9	One filter → one answer. Charlie's department → category → order → status model. Dropped the old vector-tile / unfiltered-totals path.
9072062	Prod-scale benchmark: Postgres too slow on broad filters; DuckDB fast with identical answers.
08dfb67	Query moves to map-service/ (DuckDB + Parquet). Postgres keeps vocabulary + mapping only.
9835ad1	Opening a building: /map/building, richer popup + expandable rail, ?b= deep link, photos.
17581ee	Smart search: one English sentence → a filter (POST /map/search).

Still open (do not treat as done): production deploy of map-service, browser QA of the UI, dropping the old unused tile RPCs from Postgres. See docs/LIFECYCLE_ASSESSMENT.md §8 #20.

---

3. Architecture (big picture)

┌─────────────────────────────────────────────────────────────────────────┐
│  Browser (/map)                                                          │
│  CompliedMapPage → facets (URL) → facetsToFilter → queryMap()            │
│  MapTopBar / MapRecordFilters / MapSmartSearch / MapResultsRail / …      │
└───────────────────────────────┬─────────────────────────────────────────┘
                                │  Bearer <user's Supabase JWT>
                                │  VITE_MAP_SERVICE_URL
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  map-service (Node, :8787 locally)                                       │
│  1. Forward JWT → PostgREST map_tenant_overlay()  (tracked + contested)  │
│  2. Compile filter → SQL (sql.ts)                                         │
│  3. Run against in-memory DuckDB snapshot                                 │
│  Endpoints: /map/query  /map/building  /map/taxonomy  /map/search         │
└───────────────────────────────┬─────────────────────────────────────────┘
                                │  reads Parquet from disk or S3
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  Nightly export                                                          │
│  sync-orchestrator / ingest CLIs → mark_map_stale()                      │
│  exporter waits until stamp is quiet ≥ 10 min → writes Parquet           │
│  latest.json written last (= publish)                                    │
└───────────────────────────────┬─────────────────────────────────────────┘
                                │  SELECT from
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  Postgres (source of truth)                                              │
│  public_events, buildings                                                │
│  map_event_source (view: open rows + lifted raw_data fields)             │
│  map_departments / map_categories / map_category_orders (menus)          │
│  map_data_version + mark_map_stale()                                     │
│  map_tenant_overlay() (SECURITY INVOKER, caller's JWT only)              │
└─────────────────────────────────────────────────────────────────────────┘

Rule of thumb

Concern	Lives in
"What does this filter mean?"	docs/MAP_MODEL.md, src/lib/mapFilter.ts, taxonomy tables
"Turn filter into SQL"	only map-service/src/sql.ts
"Serve / export / auth to Postgres"	map-service/
"URL ↔ UI state"	src/components/map/mapFacets.ts
"HTTP to the service"	src/data/mapQuery.ts
"Page orchestration"	src/pages/CompliedMapPage.tsx

---

4. File map (where to look)

Frontend

Path	Role
src/pages/CompliedMapPage.tsx	Owns filter state, fires queryMap / building / taxonomy / search, URL sync (?b=, facets)
src/components/map/mapFacets.ts	Facets, lenses, URL encode/decode, facetsToFilter, facetsFromSearch
src/components/map/MapTopBar.tsx	Lens + scope + filter chrome
src/components/map/MapRecordFilters.tsx	Department → category → order + status pickers
src/components/map/MapSmartSearch.tsx	"Ask the map" box
src/components/map/MapResultsRail.tsx	Ranked list, expand-in-place rows
src/components/map/CompliedMapView.tsx	Mapbox canvas: pins only (no second unfiltered layer)
src/components/map/MapBuildingDetails.tsx	Shared popup / expanded-row body
src/components/map/BuildingPhoto.tsx	Mapillary street or Mapbox aerial
src/components/map/buildingDisplay.ts	Labels, urgency line, agency ranking for one building
src/lib/mapFilter.ts	MapQueryFilter, status vocabulary, collapse of What-tree, summaries
src/lib/mapObligations.ts	Tier B obligation facets → buildingIds (browser-side ladder)
src/lib/buildingPhoto.ts	Photo URL helpers
src/data/mapQuery.ts	All map-service HTTP + tracked-point overlay + building extras from Postgres

map-service

Path	Role
src/sql.ts	Filter → SQL. Ports the old map_filter_sql / map_query. Also building detail + taxonomy.
src/snapshot.ts	One loaded DuckDB; overlay tables; hot-swap lifecycle
src/overlay.ts	JWT → map_tenant_overlay() via PostgREST
src/exporter.ts	Postgres → Parquet; staleness rule
src/storage.ts	Local dir or s3:// snapshot store
src/search.ts + searchVocabulary.ts + llm.ts	Smart search
src/app.ts	Routes (no HTTP server)
src/server.ts	node:http + poll/export timers
src/cli-export.ts	npm run export

Postgres migrations (this branch)

Migration	Role
20260924120000_map_query.sql	Taxonomy tables + map_event_source view (trimmed; no longer the Postgres query path)
20260924150000_map_duckdb_service.sql	Drops leftover Postgres query path if present; adds map_data_version, mark_map_stale(), map_tenant_overlay()
20260925120000_map_postponed_status_detail.sql	Redefines map_event_source (same columns): postponed reads status_detail (OPEN_POSTPONEMENT_GRANTED, with or without _OVERDUE) instead of raw_data->>'currentstatusid'

Tests worth knowing

Suite	Covers
map-service npm test	SQL compiler, fixture world, HTTP, tenant line (two tenants cannot see each other)
src/lib/mapFilter.test.ts	What-tree collapse, summaries
src/components/map/mapFacets.test.ts	URL codec, lenses
src/components/map/smartSearch.test.ts	Frontend ↔ service vocabulary stay in sync
db-tests/.../leakage.test.ts	map_tenant_overlay cross-tenant leakage

---

5. A request, step by step

5.1 Map load / filter change

1. User changes a chip (or lands on a URL).
2. CompliedMapPage reads facets from the query string (decodeFacets).
3. Optional: obligation facets → wait for portfolio facts → list of matching tracked building ids.
4. facetsToFilter(facets, { buildingIds? }) builds a plain MapQueryFilter.
5. queryMap({ filter, viewport, zoom, sort }) POSTs to map-service with the session JWT.
6. Service:
   • Resolves overlay (tracked building ids + contested event ids for this tenant).
   • Compiles SQL once (compileMapQuery).
   • Writes overlay tables if needed, runs the query, returns one JSON document.
7. Page paints totals, rail (ranked), and pins. Same document feeds all three.

Viewport only affects in_view, ranked, and which pins you see. Totals are citywide (within the filter's borough / scope / etc.), not "what's on screen."

5.2 Open a building

1. Click pin or rail row → selection + optional ?b=<buildingId> in the URL.
2. getMapBuildingDetail({ filter, buildingId }) → POST /map/building.
3. Same predicates as the map query: matching counts always equal the pin.
4. getMapBuildingExtras hits Postgres (BBL, zip, client, projects) — snapshot does not carry those.
5. Photo: Mapillary if VITE_MAPILLARY_TOKEN is set, else Mapbox aerial. Browser → Mapillary/Mapbox only; map-service never sees it.

5.3 Smart search

1. User types a sentence in "Ask the map".
2. searchMap(q) → POST /map/search (needs MAP_SEARCH_* on the service; else 501).
3. Model sees fixed vocabulary + live taxonomy (~3.3k tokens), returns JSON schema.
4. Service sanitize() drops anything not in the vocabulary.
5. Browser applies via facetsFromSearch (same URL codec as manual filters) — never raw model JSON into state.

---

6. The filter model (short)

Full product semantics: MAP_MODEL.md. Developer notes:

Axis	Filter fields	Notes
What	depts, categories, orders	ORed among themselves. normalizeWhat (map-service/src/shared/whatTree.ts, shared by the menus and smart search) keeps the lists from overlapping: children of a chosen parent are dropped, and a department with every category chosen folds into the department. Orders never fold into their category (it also matches null-order-code records). Orders are dept:code.
Status	statuses, agencyStatuses	ORed with each other, ANDed with What. Agency statuses are dept:NAME (|-separated in URL — HPD names contain commas).
Where	boroughs, yearFrom/yearTo, minUnits/maxUnits	Building facts, ANDed.
Whose	scope: all | tracked | prospects	Tracked/prospects need overlay.
Owed (Tier B)	obligations in facets → buildingIds	Browser computes; forces scope = tracked.

Empty What + empty Status = every building (still narrowed by Where / scope / buildingIds).

Lenses are named presets + colors (MAP_LENSES in mapFacets.ts). Not a separate query path. Adding a lens = one array entry.

URL gotchas

• Map uses state, not status (on /violations, status means OPEN/CLOSED/DISMISSED).
• Cleared-from-lens facets write as none so "cleared" ≠ "unset".
• Saved views are v: 2. Selection deep link is ?b=<uuid>.

---

7. Auth and the tenant line (do not break this)

map-service holds no service role key and no JWT secret.

For scope, contested, and anything tenant-shaped:

1. Browser sends the user's access token.
2. Service calls PostgREST map_tenant_overlay() with that same token.
3. Function is SECURITY INVOKER and matches tenant_id = current_tenant_id() (impersonation works because current_tenant_id() does).
4. Overlay rows land in in-memory DuckDB tables keyed by that tenant id; every query filters on it.
5. There is no shared result cache, so there is no cache key for tenant data to leak through.

If PostgREST refuses the token, the map refuses the request.

Tests: map-service/src/snapshot.test.ts ("the tenant line"), db-tests leakage test for map_tenant_overlay.

Never:

• Accept tenant_id from the request body.
• Fetch overlay with a service key "for convenience".
• Cache a public query result that already folded in tenant predicates.

---

8. Snapshot pipeline

1. Any public-data sync finishes → mark_map_stale() stamps map_data_version.changed_at.
2. Exporter (exportIfStale / MAP_EXPORT_WATCH): if stamp is newer than the snapshot and quiet for MAP_EXPORT_QUIET_MINUTES (default 10), export.
3. Export reads map_event_source, buildings, taxonomy → Parquet under snapshots/<id>/, then writes latest.json last.
4. Server polls latest.json, loads the new DuckDB beside the old one, swaps, retires the old when in-flight requests finish.

Local: npm run dev starts map-service with export watch; first snapshot appears seconds after start. Prod scale: ~2 min export, ~235 MB Parquet for ~10M open rows. Memory briefly doubles while two snapshots coexist (~2 GB class machine).

---

9. HTTP API (map-service)

All map routes (except /health) require Authorization: Bearer <supabase access token>.

Method	Path	Body	Returns
POST	/map/query	{ filter, viewport, zoom, sort, limit? }	totals, in_view, ranked (≤ 500), pins (points or grid cells), sync_at, as_of, build_ms, cached: false
POST	/map/building	{ filter, buildingId }	Building facts + matching / open / departments / orders
GET	/map/taxonomy	—	Departments → categories → orders + status counts for the current snapshot
POST	/map/search	{ q }	Sanitized facets + summary / unmatched / empty (501 if search not configured)
GET	/health	—	{ ok, snapshot } — no auth

Pins: individual buildings when ≤ 3,000 matches in view; otherwise grid cells (~1/8 map tile, fixed world grid). Ranked order is server-side over the in-view set — changing sort is exact, not "rearrange these 500."

---

10. Local development

# From repo root — starts Supabase check, Vite, functions, AND map-service with export watch
npm run dev

# Map only pieces
cd map-service && npm install && npm start    # :8787
cd map-service && npm run export              # snapshot now
cd map-service && npm test

Frontend expects VITE_MAP_SERVICE_URL. In dev, src/data/mapQuery.ts falls back to http://127.0.0.1:8787. Production builds have no fallback — unset URL = map hard-fails.

Optional:

• VITE_MAPILLARY_TOKEN — street-level photos on open building.
• MAP_SEARCH_PROVIDER + MAP_SEARCH_API_KEY (+ model / base URL) — smart search. Key stays on the service; never VITE_.

Env table: map-service/AGENTS.md.

---

11. Production rollout (order matters)

Merging this branch to main before the service is live breaks /map, because Cloudflare Pages will ship a frontend that calls VITE_MAP_SERVICE_URL with no fallback.

Safe order (also in map-service/AGENTS.md):

1. Apply migrations 20260924120000 + 20260924150000 + 20260925120000 on prod.
2. Deploy map-service (EC2/container with MAP_EXPORT_WATCH=1, or Lambda + S3 — see AGENTS).
3. Run the first export; confirm /health shows a snapshot.
4. Set VITE_MAP_SERVICE_URL (and optional Mapillary / search vars) on the app Cloudflare Pages project.
5. Redeploy sync-orchestrator (calls mark_map_stale).
6. Then merge / push the frontend to main.

Docs-only changes do not need this dance — see §14.

---

12. How to change common things

I want to…	Do this
Add a lens	Entry in MAP_LENSES (mapFacets.ts). No SQL change.
Add a status key	MAP_STATUS_OPTIONS + SQL predicates in sql.ts + taxonomy if needed. Keep frontend and searchVocabulary.ts aligned (smartSearch test will fail if they drift).
Remap HPD order → category	Taxonomy seed / tables in the migration / DB — not hardcoded TS. Taxonomy is served live with counts.
Change pin grid threshold	sql.ts (pins CASE). Watch DuckDB "plans both CASE branches" gotcha in AGENTS.
Change ranked limit	MAP_RANKED_LIMIT + service limit handling.
Touch obligation logic	mapObligations.ts only — keep one ladder with /deadlines.
Deploy / ops the service	map-service/AGENTS.md "Production" + env table.

---

13. What was deliberately left behind

• Old vector-tile RPCs (map_tile, map_grid_cells, map_citywide_totals, map_buildings_in_view) are unused by the app but still in the DB until a separate drop migration. Do not resurrect client code that calls them.
• The first Postgres map_events + map_query + cache path existed only on this branch, was benchmarked, then removed. Prod that never applied the fat migration only needs the trimmed taxonomy + DuckDB migration.
• Benchmark scripts in scripts/map-bench/ still time the old Postgres path on complied_bench — historical, not the live path.

---

14. Publishing these docs (without shipping the map)

The documentation site is a separate Cloudflare Pages project from the app:

	App	Docs
Cloudflare project	complied	complied-docs
Live URL	app host	https://docs.compliednyc.com
Root directory	repo root (Vite)	website/
Production branch	main	main

Canonical markdown lives under docs/ (and root README / REQUIREMENTS). website/scripts/sync-docs.mjs copies selected files into a gitignored website/docs/ at build time. Edit sources here; never author inside website/docs/.

You can ship docs without shipping the map frontend/service:

1. Preview on this branch — push map-query-rebuild (or open a PR). Cloudflare Pages for complied-docs builds previews for non-main branches the same way as any Pages project. Open the preview URL for the docs project (not the app project). Confirm in the Cloudflare dashboard → complied-docs → Deployments.
2. Docs-only to production — open a PR that only changes docs/**, website/** (sync table / sidebars), and maybe AGENTS.md pointers. Merge that to main. complied-docs auto-deploys; the app project does not gain the map rebuild. Safe anytime.
3. Manual preview — cd website && npm ci && npm run build && npx wrangler pages deploy build --project-name=complied-docs --branch=<preview-name>.

Do not point the app Pages project at website/. Details: website/README.md.

---

15. Related reading order

1. This guide (you are here).
2. MAP_MODEL.md — filter semantics Charlie asked for.
3. MAP_QUERY_BENCHMARK_2026-09-24.md — why DuckDB won.
4. map-service/AGENTS.md — run/deploy/env.
5. docs/LIFECYCLE_ASSESSMENT.md §8 #20 — remaining ship checklist.