fio-l1-nyc-csvwkt-addresses

Analyst notes

Description

Tests CSV-with-WKT reading plus Arrow-schema literacy on the GeoParquet write side. The agent has to parse the WKT geometry column, then coerce `recorded_at` to `timestamp[us]` and `unit_count` to `int32` exactly, not the pyarrow defaults (`timestamp[ns]`, `int64`). The prompt deliberately does not mention the leftover `geometry_wkt` text column, so the agent has to recognise on its own that storing both the parsed geometry and the original WKT string defeats the point of the conversion.

Approach

1. Read the all-quoted CSV and treat every column as text on the way in.
2. Parse the WKT column into Point geometry in EPSG:4326.
3. Cast `recorded_at` to a microsecond timestamp and `unit_count` to int32, keeping the other address text columns as strings so leading-zero postcodes survive.
4. Drop the original WKT text column so the output carries the geometry only once.
5. Write `addresses.geoparquet` with the row count matching the input exactly.

Pitfalls

• Leaving the `geometry_wkt` text column in alongside the parsed geometry doubles storage and confuses downstream consumers.
• Letting pandas default `recorded_at` to `timestamp[ns]` makes DuckDB and other tools cast on every query, which is exactly what the persona is trying to avoid.
• Calling `.astype(int)` on `unit_count` lands int64 instead of the requested int32, so the SUM aggregations widen and the file gets larger than it needs to be.
• Letting type inference re-type `postcode` or `number` to integer mangles leading-zero ZIPs and fraction-style house numbers like `1/2`.
• Silently dropping malformed rows during CSV read breaks the persona's downstream `SUM(unit_count)` so the row count has to match the input exactly.
• Forgetting to declare EPSG:4326 on the GeoParquet, or reprojecting the WKT to a metric CRS, throws the geometry comparison off.

Map

Recent runs task v1