spa-l2-lagos-hotspot-overlaps

Rerun

Let me understand the task:

1. We have `lagos_landuse` (land-use polygons with population density attribute) and `lagos_hex_grid` (hex grid)
2. Need to find top 10% of hex cells ranked by area-weighted mean population density
3. Exclude tiny polygons under 100 m² (noise)
4. Output two files:
   - `hotspots.geoparquet` - GeoParquet with Polygon/MultiPolygon geometry in Nigeria's national grid, columns: hex_id, rank, area_weighted_density, geometry
   - `hotspot_ranking.parquet` - plain Parquet, columns: hex_id, rank, area_weighted_density, n_overlap_polygons, n_slivers_filtered

Let me first explore the data to understand the structure.

Let me understand the task:

1. I have `lagos_landuse.geojson` with land-use polygons in EPSG:4326. It has `pop_density` attribute.
2. I have `lagos_hex_grid.geojson` with a hex grid in EPSG:4326. It has `hex_id` attribute.
3. I need to:
   - Exclude tiny land-use polygons under 100 m² (noise).
   - For each hex cell, compute the area-weighted mean population density across overlapping land-use polygons.
   - Rank hex cells by area-weighted mean density (descending), take top 10%.
   - Produce two files:
     a. `hotspots.geoparquet`: GeoParquet with columns `hex_id`, `rank`, `area_weighted_density`, `geometry` in Nigeria's national grid (EPSG:26391 - Minna / Nigeria West Belt... or EPSG:26393? Let me think... Nigeria's national grid is typically EPSG:26391 (Minna / Nigeria West) or EPSG:26393 (Minna / Nigeria East). But actually the common one for Lagos area is EPSG:26391 (Minna / Nigeria West Belt) which covers the western part of Nigeria including Lagos. Let me check.)

Actually, Nigeria's national grid is "Minna / Nigeria West Belt" (EPSG:26391) for the western part and "Minna / Nigeria East Belt" (EPSG:26393) for the eastern part. Lagos is in the western part so it would be EPSG:26391. But let me just check if there's another common CRS. The user said "Nigeria's national grid" - this is likely EPSG:26391 (Minna / Nigeria West Belt) since Lagos is in western Nigeria.

Wait, there's also EPSG:26392 (Minna / Nigeria Mid Belt) and EPSG:26393 (Minna / Nigeria East Belt). Let me use 26391 for Lagos.

Actually, let me just compute everything and produce the outputs.

b. `hotspot_ranking.parquet`: plain Parquet (no geometry) with columns `hex_id`, `rank`, `area_weighted_density`, `n_overlap_polygons` (int), `n_slivers_filtered` (int).

The `n_overlap_polygons` is the number of land-use polygons that overlap with each hex cell (after filtering slivers).

The `n_slivers_filtered` is the number of polygons under 100 m² that were excluded per hex.

Wait, let me re-read: "Tiny polygons under 100 m² are noise and should be excluded." So we filter out land-use polygons with area < 100 m² before doing the analysis. The `n_slivers_filtered` column should be the count of such tiny polygons per hex that were excluded.

Hmm, actually let me think about this more carefully. The `n_slivers_filtered` is per hex cell. It could mean the number of polygons that overlap with that hex cell but were filtered out because they're under 100 m². But since filtering happens globally (we remove all tiny polygons before the overlap analysis), the count per hex of "how many of these tiny polygons overlapped this hex" would be needed.

Actually, let me think step by step:

1. First, for each land-use polygon, compute its area (in a metric CRS).
2. Filter out polygons with area < 100 m².
3. For the filtered-out polygons, we still need to count per hex cell how many of them overlap with each hex (to report `n_slivers_filtered`).
4. For the remaining polygons, compute the area-weighted mean pop_density per hex cell.

Let me plan the solution:

1. Read both files.
2. Reproject land-use to EPSG:26391 (or similar metric CRS) to compute areas.
3. Identify tiny polygons (area < 100 m²) - these are "slivers".
4. For each tiny polygon, find which hex cells it overlaps with, and count per hex (n_slivers_filtered).
5. For non-tiny polygons, for each hex cell, compute:
   - Intersection area between the hex and each overlapping land-use polygon
   - Area-weighted mean density = sum(density_i * intersection_area_i) / sum(intersection_area_i)
6. Sort by area-weighted density descending, assign rank 1..N
7. Take top 10% (ceil).
8. Write files.

Let me write the script.

For Nigeria's national grid - Lagos is in the western part of Nigeria. The standard is EPSG:26391 (Minna / Nigeria West Belt).

Let me verify by looking at the coordinates. Lagos hex data is around 3.25°E to 3.5°E, 6.4°N to 6.45°N. EPSG:26391 would be appropriate.

Actually, let me think about whether I should use 26391 or some projected CRS. Let me just use 26391.

Let me write the solve script.

Actually, I need to be careful about the `n_slivers_filtered` column. Let me re-read:

"`n_slivers_filtered` (integer)" - this is the number of polygons that were filtered out (under 100 m²) that overlap with each hex cell.

So per hex cell, count of tiny polygons overlapping that cell.

Let me write the script.

The Nigeria national grid for the western belt is EPSG:26391 (Minna / Nigeria West Belt). Let me also check if there's a unified national grid EPSG:26392 or EPSG:26393. For Lagos, the West Belt (26391) is appropriate.

Actually, let me think about what "Nigeria's national grid" means. Nigeria uses the Minna datum with three belts:
- EPSG:26391 - Nigeria West Belt (zone 1)
- EPSG:26392 - Nigeria Mid Belt (zone 2) 
- EPSG:26393 - Nigeria East Belt (zone 3)

Lagos is in the West, so EPSG:26391 is correct.

Now let me write the solution script.

Plan:
1. Read both files
2. Convert both to Nigeria national grid (EPSG:26391) for area calculations
3. Filter out land-use polygons with area < 100 m²
4. For each hex cell, find overlapping land-use polygons
5. Compute area-weighted mean population density:
   - For each hex cell, find intersection area with each overlapping polygon
   - area_weighted_density = sum(overlap_area_i * pop_density_i) / sum(overlap_area_i)
6. Rank hex cells by area_weighted_density descending
7. Take top 10%
8. Output hotspots.geoparquet (with geometry) and hotspot_ranking.parquet

Let me also track n_overlap_polygons (number of overlapping polygons used in calculation) and n_slivers_filtered (number of polygons excluded due to area < 100 m²).

Wait - "Tiny polygons under 100 m² are noise and should be excluded." - This means filtering out land-use polygons whose area is < 100 m² before doing the overlap analysis.

And n_slivers_filtered = number of those tiny polygons that were excluded.

n_overlap_polygons = number of polygons that overlap each hex cell (after filtering).

Let me code this up carefully.

Wait, let me check the output carefully. The hex grid had 1782 cells but only 1032 had overlapping polygons. That's fine. But I need to verify:

1. Both files share exactly the same set of hex_id values.
2. The CRS of hotspots.geoparquet is Nigeria national grid (EPSG:26391).
3. rank is unique, starts at 1, ascending = descending density.
4. n_overlap_polygons is integer.
5. n_slivers_filtered is integer.

Let me verify these things.

Everything checks out perfectly:

1. **hotspots.geoparquet** - 104 rows, CRS is EPSG:26391 (Minna / Nigeria West Belt - Nigeria national grid), columns: `hex_id`, `rank`, `area_weighted_density`, `geometry`
2. **hotspot_ranking.parquet** - 104 rows, same `hex_id` set, columns: `hex_id`, `rank`, `area_weighted_density`, `n_overlap_polygons` (int64), `n_slivers_filtered` (int64)
3. Both files have exactly the same 104 hex_ids
4. Rank is unique starting at 1, ascending corresponds to descending area_weighted_density
5. `n_overlap_polygons` and `n_slivers_filtered` are integers

Let me just verify the files exist on disk.