crs-l3-tokyo-jgd-crossings
Map (reference ← swipe → agent)
0:00
I'm rebuilding the road-safety dashboard for Tokyo's 23 special wards. For each ward I need to know how often the drivable road network crosses the ward boundary (please skip service roads), plus a 50 m buffer around every crossing for jurisdictional reporting. Pull the data from current OSM via Overpass. Run the engineering layers in the regional metric coordinate system so the distances and areas come out in real metres.
Save everything to a single `tokyo_crossings.gpkg` with five layers:
- `wards_jgd` (Polygon): the 23 ward boundaries.
- `crossing_points` (Point): one point at each place a road crosses a ward boundary.
- `crossing_buffers_50m` (Polygon): a 50 m buffer around each crossing point.
- `buffer_ward_intersection` (Polygon): each buffer clipped to the ward whose boundary it crossed.
- `ward_crossing_density_wgs84` (Polygon): one row per ward with `ward_id`, `crossing_count`, `ward_area_km2`, and `crossings_per_km2`. This is what the public dashboard reads, so it goes in WGS84 while the other four stay in the metric grid.
Use each ward's OSM relation id as `ward_id`, and carry the English and native ward names as `ward_name_en` and `ward_name` on every layer that's tied to a specific ward.
0:00
System info
{
"adapter": "openrouter",
"model": "deepseek/deepseek-v4-flash",
"base_url": "https://openrouter.ai/api/v1"
}
0:07
Let me break down this task: 1. Get the 23 special wards of Tokyo from OSM via Overpass 2. Get the drivable road network (excluding service roads) for Tokyo 3. Find crossings where roads intersect ward boundaries 4. Create 50m buffers around crossing points 5. Clip buffers to the ward they crossed 6. Calculate crossing density per ward 7. Save everything in a GeoPackage with 5 layers Let me start by understanding the coordinate system situation. "Regional metric coordinate system" - for Tokyo, this would be JGD2011 / Japan Plane Rectangular CS IX (zone 9) or JGD2000 / Japan Plane Rectangular CS IX. Let me use EPSG:6679 (JGD2011 / Japan Plane Rectangular CS IX) which covers Tokyo. Actually, let me check what's commonly used. The Japan Plane Rectangular Coordinate System zone 9 (EPSG:2451 for JGD2000, EPSG:6679 for JGD2011) covers Tokyo. Let me write a Python script to do this. First, let me understand the Overpass query structure. I need: - Relations for the 23 special wards of Tokyo - Ways for drivable roads (highway=*) but not service roads - Find intersections Let me write this step by step. Actually, Overpass can be complex. Let me use osmnx or direct overpass API calls. Let me think about the approach: 1. Use OSMnx or Overpass API to get the ward boundaries 2. Get the road network 3. Find intersection points between roads and ward boundaries 4. Buffer etc. Let me write a comprehensive Python script. First, let me check what's installed.
0:07
Bash
pip list 2>/dev/null | grep -iE "geopandas|shapely|pyproj|osmnx|overpy|duckdb|py...
duckdb 1.5.3 fiona 1.10.1 geopandas 1.1.3 osmnx 2.1.0 pyogrio 0.12.1 pyproj 3.7.2 shapely 2.1.2
0:32
Good, all the necessary libraries are available. Let me plan this task step by step. 1. **Get Tokyo's 23 special wards boundaries** from OSM using Overpass. The 23 special wards of Tokyo (東京23区) are OSM relations. I need to query Overpass API to get them. 2. **Get the drivable road network** for Tokyo's 23 wards area using OSM. Need to exclude service roads (highway=service). 3. **Find crossings** where roads cross ward boundaries. 4. **Create 50m buffers** around each crossing point. 5. **Clip buffers** to the ward whose boundary was crossed. 6. **Calculate densities** and reproject to WGS84. Let me start by figuring out the OSM relation IDs for Tokyo's 23 special wards. The 23 special wards of Tokyo (東京23区) are: 1. Chiyoda (千代田区) - 3558217 2. Chuo (中央区) - 3560101 3. Minato (港区) - 3581265 4. Shinjuku (新宿区) - 3573780 5. Bunkyo (文京区) - 3560085 6. Taito (台東区) - 3558232 7. Sumida (墨田区) - 3581279 8. Koto (江東区) - 3558575 9. Shinagawa (品川区) - 3581301 10. Meguro (目黒区) - 3558388 11. Ota (大田区) - 3581297 12. Setagaya (世田谷区) - 3558426 13. Shibuya (渋谷区) - 3573754 14. Nakano (中野区) - 3573802 15. Suginami (杉並区) - 3573732 16. Toshima (豊島区) - 3560080 17. Kita (北区) - 3558312 18. Arakawa (荒川区) - 3558368 19. Itabashi (板橋区) - 3558392 20. Nerima (練馬区) - 3558402 21. Adachi (足立区) - 3558305 22. Katsushika (葛飾区) - 3558295 23. Edogawa (江戸川区) - 3558304 Let me verify these with an Overpass query first. Actually, I can get them all in one Overpass query by looking for relations with admin_level=9 and name:en containing "ward" or similar. Let me write a Python script to do everything. The coordinate system for Japan is JGD2011 / Japan Plane Rectangular CS IX (EPSG:6671) or JGD2000 / Japan Plane Rectangular CS IX (EPSG:2451) for the Tokyo area. Actually, let me think about the metric coordinate system. For Tokyo, the appropriate metric coordinate system is: - JGD2011 / Japan Plane Rectangular CS IX (EPSG:6671) - covers Tokyo - Or JGD2000 / Japan Plane Rectangular CS IX (EPSG:2451) I'll use EPSG:6671 (JGD2011 / Japan Plane Rectangular CS IX) as it's the most up-to-date. Let me write the full script. First, I need to query Overpass for: 1. The 23 special wards (relations with admin_level=9, or specific relation IDs) 2. The road network within the area of these wards (highways that are not service roads) Let me think about how to find crossings. The approach: - Get ward boundaries as lines - Get roads as lines - Find intersection points between ward boundaries and roads Actually, a better approach: - Each ward boundary is a polygon. The boundary is the exterior ring. - For each pair of adjacent wards, find where roads cross the shared boundary. - Or simpler: dissolve all wards into one boundary, then find where roads cross it. Wait, the task says "for each ward" - so I need crossings per ward. A road crossing a ward boundary is entering/exiting that ward. So for each ward, I find where roads intersect the ward boundary polygon's boundary (exterior ring). Let me think more carefully. If ward A and ward B share a boundary, a road crossing from A to B would cross both ward boundaries at the same point. So if I just count road-boundary intersections per ward, I'd double-count. The task says: "for each ward I need to know how often the drivable road network crosses the ward boundary". So it's per ward. Each crossing point belongs to exactly one ward? No, it belongs to two wards (the two sides of the boundary). Actually, let me think about this differently. A crossing point is where a road crosses a ward boundary. For each such crossing, there are two wards involved. The crossing belongs to both wards' boundary. The "buffer around every crossing" - each buffer could extend into both wards. The task says "each buffer clipped to the ward whose boundary it crossed" - so for each crossing point, we create a buffer and clip it to the ward whose boundary was crossed. But which ward? If a road crosses from ward A to ward B, the crossing point is on both boundaries. So we'd have two clips - one for ward A and one for ward B? Actually, I think the intent is: - For each ward, find where roads cross its boundary. - For each such crossing point, create a 50m buffer. - Clip that buffer to the ward. So if a road crosses the boundary between ward A and ward B, that crossing point is on ward A's boundary AND ward B's boundary. We'd create a buffer for ward A (clipped to ward A) and a buffer for ward B (clipped to ward B). But that means the crossing_points layer might have duplicate points (one for each ward it touches). Let me think... Actually, maybe simpler: find all unique road-boundary crossing points. Then for each crossing point, determine which ward(s) it's on the boundary of. Create buffers clipped to each ward. Let me plan the implementation: 1. Query Overpass for the 23 wards (admin_level=9 relations in Tokyo) 2. Get their boundaries as polygons 3. Query Overpass for roads (excluding service roads) within the bounding box of Tokyo 4. For each ward: a. Get its boundary line b. Find intersection points between roads and the boundary c. Record these as crossing points with the ward's info 5. Create a unique set of crossing points 6. Create 50m buffers around each 7. For each buffer, clip to the ward(s) whose boundary it crossed 8. Calculate ward-level stats: crossing count, area, density 9. Save to GeoPackage with 5 layers Let me start coding. Actually, for the Overpass query, I can use osmnx to get the data. osmnx is great for this. Let me write the script. For the metric coordinate system: Tokyo area is in Japan Plane Rectangular CS IX. I'll use JGD2011 / Japan Plane Rectangular CS IX (EPSG:6671). Let me write the complete script now. Actually, I realize I should first check what relation IDs OSM has for Tokyo's wards. Let me query Overpass to find them. Let me write a comprehensive script.
0:32
Write
/work/download_wards.py
content · 28 lines · py
"""Query Overpass to find the 23 special wards of Tokyo."""
import requests
import json
overpass_url = "https://overpass-api.de/api/interpreter"
# Query for all admin_level=9 relations in Tokyo with name:en containing "Special Ward" or similar
# Actually let's get them by querying the "Tokyo" city/region and finding its child admin_level=9
query = """
[out:json][timeout:60];
(
area["name:en"="Tokyo"]["admin_level=4"];
rel(area)["admin_level"="9"]["name:en"];
);
out body;
>;
"""
r = requests.post(overpass_url, data={"data": query})
data = r.json()
print(f"Number of elements: {len(data.get('elements', []))}")
for elem in data.get('elements', []):
if elem['type'] == 'relation' and 'tags' in elem:
tags = elem['tags']
name_en = tags.get('name:en', '')
name = tags.get('name', '')
print(f" Relation {elem['id']}: {name_en} / {name}")Wrote /work/download_wards.py (882 chars).
0:37
Let me run this to see what we get.
0:37
Bash
cd /work && python download_wards.py
Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/requests/models.py", line 1116, in json
return complexjson.loads(self.text, **kwargs)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/json/__init__.py", line 352, in loads
return _default_decoder.decode(s)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File "/usr/local/lib/python3.14/json/decoder.py", line 345, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/json/decoder.py", line 363, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/download_wards.py", line 20, in <module>
data = r.json()
File "/usr/local/lib/python3.14/site-packages/requests/models.py", line 1120, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
0:41
The request might be failing. Let me debug a bit.
0:41
Bash
cd /work && python -c "
cd /work && python -c "
import requests
query = '[out:json][timeout:60]; area[\"name:en\"=\"Tokyo\"][\"admin_level\"=\"4\"]; rel(area)[\"admin_level\"=\"9\"][\"name:en\"]; out body; >;'
r = requests.post('https://overpass-api.de/api/interpreter', data={'data': query})
print(r.status_code)
print(r.text[:500])
"
406 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address> </body></html>
0:47
The request might have issues with the format. Let me try a different approach - maybe using a simpler query or using osmnx which wraps Overpass nicely.
0:47
Bash
cd /work && python -c "
cd /work && python -c "
import requests
query = '[out:json][timeout:60];area[\"admin_level\"=\"4\"][\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
r = requests.post('https://overpass-api.de/api/interpreter', data={'data': query})
print(r.status_code, len(r.text))
if r.status_code == 200:
import json
d = r.json()
for e in d.get('elements', []):
if e['type']=='relation' and 'tags' in e:
print(e['id'], e['tags'].get('name:en',''), e['tags'].get('name',''))
"
406 371
0:53
The Overpass API is returning 406. Let me try with curl or different headers.
0:53
Bash
curl -s -o /tmp/out.json -w '%{http_code}' -H 'Content-Type: application/x-www-f...
bash: line 1: curl: command not found
1:00
The 406 error is likely due to content-type issues. Let me try setting headers in the Python request.
1:00
Bash
cd /work && python -c "
cd /work && python -c "
import requests
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
query = '[out:json][timeout:60];area[\"admin_level\"=\"4\"][\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
r = requests.post('https://overpass-api.de/api/interpreter', data={'data': query}, headers=headers)
print(r.status_code, len(r.text))
if r.status_code == 200:
import json
d = r.json()
for e in d.get('elements', []):
if e['type']=='relation' and 'tags' in e:
print(e['id'], e['tags'].get('name:en',''), e['tags'].get('name',''))
"
406 371
1:05
Let me try a different Overpass API endpoint or check if the query format is wrong.
1:05
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try overpass-api.de with proper GET method
query = '[out:json][timeout:60];area[\"admin_level\"=\"4\"][\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
r = requests.get('https://overpass-api.de/api/interpreter', params={'data': query})
print(r.status_code, len(r.text))
print(r.text[:200] if len(r.text)<500 else r.text[:500])
"
406 371 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate repres
1:13
The 406 error is about content negotiation. Let me try using the Overpass Turbo endpoint instead, or maybe I need to set the Accept header.
1:13
Bash
cd /work && python -c "
cd /work && python -c "
import requests
headers = {'Accept': 'application/json'}
query = '[out:json][timeout:60];area[\"admin_level\"=\"4\"][\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
r = requests.post('https://overpass-api.de/api/interpreter', data={'data': query}, headers=headers)
print(r.status_code, len(r.text))
if r.status_code == 200:
import json
d = r.json()
for e in d.get('elements', []):
if e['type']=='relation' and 'tags' in e:
print(e['id'], e['tags'].get('name:en',''), e['tags'].get('name',''))
"
406 371
1:23
Let me try a different Overpass server or use the library osmnx to fetch this data. osmnx already handles Tokyo well.
1:23
Bash
cd /work && python -c "
cd /work && python -c "
import osmnx as ox
# Try to get Tokyo special wards using OSMnx
# First let's see what we can get
gdf = ox.features_from_place('Tokyo, Japan', tags={'admin_level':'9'})
print(gdf.shape)
print(gdf.columns.tolist())
print(gdf[['name','name:en','admin_level']].head(30))
"
/usr/local/lib/python3.14/site-packages/osmnx/_overpass.py:271: UserWarning: This area is 650 times your configured Overpass max query area size. It will automatically be divided up into multiple sub-queries accordingly. This may take a long time.
multi_poly_proj = utils_geo._consolidate_subdivide_geometry(poly_proj)
Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
(self._dns_host, self.port),
...<2 lines>...
socket_options=self.socket_options,
)
File "/usr/local/lib/python3.14/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/usr/local/lib/python3.14/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
~~~~~~~~~~~~^^^^
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 788, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 488, in _make_request
raise new_e
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 464, in _make_request
self._validate_conn(conn)
~~~~~~~~~~~~~~~~~~~^^^^^^
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 1106, in _validate_conn
conn.connect()
~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/urllib3/connection.py", line 759, in connect
self.sock = sock = self._new_conn()
~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/urllib3/connection.py", line 219, in _new_conn
raise NewConnectionError(
self, f"Failed to establish a new connection: {e}"
) from e
urllib3.exceptions.NewConnectionError: HTTPSConnection(host='overpass-api.de', port=443): Failed to establish a new connection: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/requests/adapters.py", line 696, in send
resp = conn.urlopen(
method=request.method,
...<9 lines>...
chunked=chunked,
)
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 842, in urlopen
retries = retries.increment(
method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
)
File "/usr/local/lib/python3.14/site-packages/urllib3/util/retry.py", line 543, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by NewConnectionError("HTTPSConnection(host='overpass-api.de', port=443): Failed to establish a new connection: [Errno 111] Connection refused"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 5, in <module>
gdf = ox.features_from_place('Tokyo, Japan', tags={'admin_level':'9'})
File "/usr/local/lib/python3.14/site-packages/osmnx/features.py", line 279, in features_from_place
return features_from_polygon(polygon, tags)
File "/usr/local/lib/python3.14/site-packages/osmnx/features.py", line 332, in features_from_polygon
return _create_gdf(response_jsons, polygon, tags)
File "/usr/local/lib/python3.14/site-packages/osmnx/features.py", line 406, in _create_gdf
for response_json in response_jsons:
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/osmnx/_overpass.py", line 432, in _download_overpass_features
yield _overpass_request(OrderedDict(data=query_str))
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/osmnx/_overpass.py", line 469, in _overpass_request
response = requests.post(
url,
...<3 lines>...
**settings.requests_kwargs,
)
File "/usr/local/lib/python3.14/site-packages/requests/api.py", line 134, in post
return request("post", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.14/site-packages/requests/api.py", line 71, in request
return session.request(method=method, url=url, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/requests/sessions.py", line 651, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.14/site-packages/requests/sessions.py", line 784, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.14/site-packages/requests/adapters.py", line 729, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by NewConnectionError("HTTPSConnection(host='overpass-api.de', port=443): Failed to establish a new connection: [Errno 111] Connection refused"))
2:33
It seems overpass-api.de is refusing connections from this environment. Let me try a different approach - maybe use a different endpoint or download data from a mirror. Let me check if there are other overpass instances available.
2:33
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try overpass.kumi.systems
hosts = ['https://overpass-api.de/api/interpreter', 'https://overpass.kumi.systems/api/interpreter', 'https://overpass.openstreetmap.fr/api/interpreter']
for h in hosts:
try:
query = '[out:json][timeout:60];area[\"admin_level\"=\"4\"][\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
r = requests.get(h, params={'data': query}, timeout=10)
print(f'{h}: {r.status_code} {len(r.text)}')
except Exception as e:
print(f'{h}: {type(e).__name__}: {e}')
"
https://overpass-api.de/api/interpreter: 406 371 https://overpass.kumi.systems/api/interpreter: ReadTimeout: HTTPSConnectionPool(host='overpass.kumi.systems', port=443): Read timed out. (read timeout=10) https://overpass.openstreetmap.fr/api/interpreter: 403 53
2:50
The overpass-api.de returns 406. Let me try using the overpass turbo query format differently - maybe the issue is the API URL or the content-type. Let me try with the right content-type header.
2:50
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Let's try with proper form data content type
query = '[out:json][timeout:60];area[\"admin_level\"=\"4\"][\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
r = requests.post('https://overpass-api.de/api/interpreter',
data=query.encode('utf-8'),
headers={'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'})
print(r.status_code, len(r.text))
print(r.text[:300])
"
406 371 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.6
2:57
The Overpass API is returning 406 Not Acceptable. This seems to be a content negotiation issue. Let me try using `requests.Session` with proper headers, or perhaps the issue is that the query string needs to be URL-encoded differently. Wait, the 406 means the server can't produce content matching the Accept header sent by the client. The requests library sends `Accept: */*` by default, which should be fine. But overpass-api.de might have issues with certain client characteristics. Let me try a different approach - using `urllib` or checking if there's a proxy/VPN issue.
2:57
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try with Accept header explicitly set for JSON
headers = {
'Accept': 'application/json, text/plain, */*',
'Content-Type': 'application/x-www-form-urlencoded',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64)'
}
query = '[out:json][timeout:60];area[\"admin_level\"=\"4\"][\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
r = requests.post('https://overpass-api.de/api/interpreter',
data={'data': query},
headers=headers)
print(r.status_code, len(r.text))
if r.status_code == 200:
print(r.text[:500])
else:
print(r.text[:300])
"
406 371 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.6
3:07
The HTTP 406 error is strange. Let me check if maybe there's a proxy issue or we need to use a different approach. Let me try using OSMnx which uses the overpass API but perhaps through a different mechanism, or let me try a simple direct curl-like approach using urllib.
3:07
Bash
cd /work && python -c "
cd /work && python -c "
import urllib.request, urllib.parse
query = '[out:json][timeout:60];area[\"admin_level\"=\"4\"][\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
data = urllib.parse.urlencode({'data': query}).encode()
req = urllib.request.Request('https://overpass-api.de/api/interpreter', data=data, method='POST')
req.add_header('Accept', '*/*')
req.add_header('Content-Type', 'application/x-www-form-urlencoded')
try:
resp = urllib.request.urlopen(req, timeout=30)
print(resp.status, resp.read()[:500])
except Exception as e:
print(f'Error: {e}')
# Try reading error
if hasattr(e, 'read'):
print(e.read()[:500])
"
Error: HTTP Error 406: Not Acceptable b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html><head>\n<title>406 Not Acceptable</title>\n</head><body>\n<h1>Not Acceptable</h1>\n<p>An appropriate representation of the requested resource could not be found on this server.</p>\n<hr>\n<address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address>\n</body></html>\n'
3:12
The 406 error seems to be related to the Overpass query format. Let me try a simpler query first to see if the server is reachable at all.
3:12
Bash
cd /work && python -c "
cd /work && python -c "
import urllib.request, urllib.parse
# Try a very simple query
query = '[out:json];node(1);out;'
data = urllib.parse.urlencode({'data': query}).encode()
req = urllib.request.Request('https://overpass-api.de/api/interpreter', data=data, method='POST')
req.add_header('Accept', '*/*')
req.add_header('Content-Type', 'application/x-www-form-urlencoded')
try:
resp = urllib.request.urlopen(req, timeout=30)
print(resp.status, resp.read()[:300])
except Exception as e:
print(f'Error: {e}')
if hasattr(e, 'read'):
print(e.read()[:500])
"
Error: HTTP Error 406: Not Acceptable b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html><head>\n<title>406 Not Acceptable</title>\n</head><body>\n<h1>Not Acceptable</h1>\n<p>An appropriate representation of the requested resource could not be found on this server.</p>\n<hr>\n<address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address>\n</body></html>\n'
3:18
The overpass API is returning 406 for all requests. Let me try a different approach. Maybe I can use a different mirror or check if there's a way to make this work. Let me try using the `requests` library with different content encoding or try the Austrian instance.
3:18
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try alternative mirror
query = '[out:json][timeout:60];area[\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
urls = [
'https://overpass-api.de/api/interpreter',
'https://overpass.kumi.systems/api/interpreter',
'https://overpass.openstreetmap.fr/api/interpreter',
'https://overpass.rz-bremen.de/api/interpreter',
]
for url in urls:
try:
# Use a raw POST with data=string
r = requests.post(url, data=query.encode('utf-8'),
headers={'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'},
timeout=30)
print(f'{url}: {r.status_code} {len(r.text)}')
if r.status_code == 200 and 'json' in r.text[:50]:
print('Got JSON!')
except Exception as e:
print(f'{url}: {type(e).__name__}: {e}')
"
https://overpass-api.de/api/interpreter: 406 371 https://overpass.kumi.systems/api/interpreter: ReadTimeout: HTTPSConnectionPool(host='overpass.kumi.systems', port=443): Read timed out. (read timeout=30) https://overpass.openstreetmap.fr/api/interpreter: 403 53 https://overpass.rz-bremen.de/api/interpreter: ConnectTimeout: HTTPSConnectionPool(host='overpass.rz-bremen.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by ConnectTimeoutError(<HTTPSConnection(host='overpass.rz-bremen.de', port=443) at 0x7f10a5326e90>, 'Connection to overpass.rz-bremen.de timed out. (connect timeout=30)'))
4:25
The Overpass API is giving 406 errors. This might be a server configuration issue. Let me check if there are any network restrictions in this environment.
4:25
Bash
pip install --user osm2geojson 2>/dev/null; cd /work && python -c "
pip install --user osm2geojson 2>/dev/null; cd /work && python -c "
import requests
# Let's try with a different content type approach - maybe just use GET
query = '[out:json][timeout:60];area[\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
# Try directly via GET with the data in the URL
import urllib.parse
encoded = urllib.parse.quote(query)
try:
r = requests.get(f'https://overpass-api.de/api/interpreter?data={encoded}', timeout=30)
print(r.status_code, len(r.text))
print(r.text[:300])
except Exception as e:
print(f'Error: {e}')
"
Collecting osm2geojson Downloading osm2geojson-0.3.2-py3-none-any.whl.metadata (8.4 kB) Requirement already satisfied: shapely>=2.0.0 in /usr/local/lib/python3.14/site-packages (from osm2geojson) (2.1.2) Requirement already satisfied: requests>=2.25.0 in /usr/local/lib/python3.14/site-packages (from osm2geojson) (2.34.2) Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.14/site-packages (from requests>=2.25.0->osm2geojson) (3.4.7) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.14/site-packages (from requests>=2.25.0->osm2geojson) (3.18) Requirement already satisfied: urllib3<3,>=1.26 in /usr/local/lib/python3.14/site-packages (from requests>=2.25.0->osm2geojson) (2.7.0) Requirement already satisfied: certifi>=2023.5.7 in /usr/local/lib/python3.14/site-packages (from requests>=2.25.0->osm2geojson) (2026.5.20) Requirement already satisfied: numpy>=1.21 in /usr/local/lib/python3.14/site-packages (from shapely>=2.0.0->osm2geojson) (2.4.6) Downloading osm2geojson-0.3.2-py3-none-any.whl (18 kB) Installing collected packages: osm2geojson Successfully installed osm2geojson-0.3.2 406 371 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.6
4:33
The overpass API is returning 406. This might be because the server requires a specific User-Agent. Let me try with a proper User-Agent header.
4:33
Bash
cd /work && python -c "
cd /work && python -c "
import requests
query = '[out:json][timeout:60];area[\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
headers = {
'User-Agent': 'Mozilla/5.0 (compatible; GISBot/1.0; +http://example.com)',
'Accept': 'application/json',
'Content-Type': 'application/x-www-form-urlencoded'
}
r = requests.post('https://overpass-api.de/api/interpreter', data={'data': query}, headers=headers, timeout=60)
print(r.status_code, len(r.text))
if r.status_code == 200:
print(r.text[:500])
else:
print(r.text[:500])
"
406 371 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address> </body></html>
4:40
The 406 error is persistent. Let me try using `https://overpass-api.de/api/interpreter` with a different approach. Maybe the issue is that this particular server is blocking based on some header or IP. Let me try the `http://` version or a different API entirely. Actually, maybe the issue is that the overpass-api.de server is being blocked from this environment. Let me try using osm2geojson or a different Overpass endpoint. Wait - let me try with `http://overpass-api.de/api/interpreter` (not HTTPS) or try other endpoints.
4:40
Bash
cd /work && python -c "
cd /work && python -c "
import requests
query = '[out:json][timeout:60];area[\"name\"=\"東京都\"];(rel(area)[\"admin_level\"=\"9\"];);out body;>;'
# Try HTTP instead of HTTPS
r = requests.post('http://overpass-api.de/api/interpreter', data={'data': query}, timeout=60)
print(r.status_code, len(r.text))
if r.status_code == 200:
print(r.text[:500])
else:
print(r.text[:500])
"
406 370 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 80</address> </body></html>
4:46
The 406 error is strange. Let me check if there's a proxy or firewall issue. Let me try a different approach - maybe I need to use a different Overpass endpoint or approach. Let me check if the network allows outbound connections. Actually, let me look at this differently - maybe the issue is that the server blocks certain User-Agent strings. Let me also try with `wget` or check if we have network access at all.
4:46
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Test basic internet access
r = requests.get('https://api.github.com', timeout=10)
print(r.status_code, len(r.text))
"
200 2396
4:54
The Overpass API is returning 406 Not Acceptable. Let me check if maybe there's a proxy or something. Let me try a simpler query first to see if the issue is with my query or the server itself. Actually, I recall that the Overpass API sometimes requires certain Accept headers. Let me try a few things. Wait - maybe the issue is that the server is using content negotiation and the Python requests library somehow doesn't send proper Accept headers. Let me try adding `*/*` to Accept.
4:54
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try a really minimal query
query = '[out:json];node(1);out;'
r = requests.post('https://overpass-api.de/api/interpreter',
data=query.encode('utf-8'),
headers={
'Content-Type': 'application/x-www-form-urlencoded',
'Accept': '*/*',
},
timeout=30)
print(r.status_code, r.reason)
print(r.text[:500])
"
406 Not Acceptable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address> </body></html>
4:59
It seems the Overpass API is blocking requests from this environment. Let me try a different approach - maybe use OSM data from a different source, or try the Overpass API with a different endpoint/format. Let me check if there's a proxy or something blocking the requests. Actually, the 406 error suggests the server is rejecting the Accept headers. Let me try more carefully. Actually, let me read the error more carefully. The 406 Not Acceptable error means the server can't produce a response matching the Accept header sent. The default Accept header of requests might be causing issues. Let me try sending no Accept header at all, or try with the raw urllib with a minimal set of headers.
4:59
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Minimal headers
r = requests.post('https://overpass-api.de/api/interpreter',
data='data=[out:json];node(1);out;',
timeout=30)
print(f'Status: {r.status_code}')
print(f'Headers: {dict(r.request.headers)}')
print(f'Response headers: {dict(r.headers)}')
print(r.text[:500])
"
Status: 406
Headers: {'User-Agent': 'python-requests/2.34.2', 'Accept-Encoding': 'gzip, deflate, zstd', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '28'}
Response headers: {'Date': 'Thu, 18 Jun 2026 03:22:28 GMT', 'Server': 'Apache/2.4.67 (Debian)', 'Content-Length': '371', 'Connection': 'close', 'Content-Type': 'text/html; charset=iso-8859-1'}
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<title>406 Not Acceptable</title>
</head><body>
<h1>Not Acceptable</h1>
<p>An appropriate representation of the requested resource could not be found on this server.</p>
<hr>
<address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address>
</body></html>
5:06
The Overpass API is returning 406 Not Acceptable. This is odd - the server seems to reject our requests. Let me try a different approach - maybe I need to use a different endpoint or there's a different way to format the request. Let me check if there's an issue with the request format. The standard approach for Overpass is to POST with `data=<query>` in the body.
Wait, looking at the request headers - the Content-Length is only 28, which means it sent `data=[out:json];node(1);out;` as the body, not `data=[out:json];node(1);out;` as a form parameter.
Let me try with `data={'data': query}` properly.
5:06
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# The issue is that I was sending the raw string as the body, not as a form parameter
# Let's send it properly with data= as form param
query = '[out:json];node(1);out;'
r = requests.post('https://overpass-api.de/api/interpreter', data={'data': query}, timeout=30)
print(f'Status: {r.status_code}')
print(f'Request body: {r.request.body[:100]}')
print(r.text[:500])
"
Status: 406 Request body: data=%5Bout%3Ajson%5D%3Bnode%281%29%3Bout%3B <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address> </body></html>
5:12
The overpass-api.de server is returning 406 for all queries. This might be an IP block or some issue with the server. Let me try a different approach - maybe I can use a different Overpass instance or download the data from geofabrik or another source. Actually, let me check if there's a geofabrik extract available, or I could try the main OSM API.
5:12
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try a different Overpass instance
urls = [
'https://overpass.kumi.systems/api/interpreter',
'https://overpass.openstreetmap.fr/api/interpreter',
'https://overpass.rz-bremen.de/api/interpreter',
]
query = '[out:json];node(1);out;'
for url in urls:
try:
r = requests.post(url, data={'data': query}, timeout=10)
print(f'{url}: {r.status_code} {len(r.text)}')
if r.status_code == 200:
print(r.text[:200])
except Exception as e:
print(f'{url}: {type(e).__name__}')
"
https://overpass.kumi.systems/api/interpreter: ReadTimeout https://overpass.openstreetmap.fr/api/interpreter: 403 53 https://overpass.rz-bremen.de/api/interpreter: ConnectTimeout
5:40
The overpass-api.de is returning 406 for all requests. Let me check if there's a different approach needed - maybe the server requires a certain Accept header negotiation. Let me try adding "Accept: application/json" specifically.
5:40
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Maybe the issue is Accept header negotiation - let's be very explicit
query = '[out:json];node(1);out;'
r = requests.post('https://overpass-api.de/api/interpreter',
data={'data': query},
headers={
'Accept': 'application/json, text/javascript, */*; q=0.01',
'Accept-Language': 'en-US,en;q=0.5',
},
timeout=30)
print(f'Status: {r.status_code}')
print(r.text[:500])
# Also try with curl-equivalent via requests
import json
r2 = requests.post('https://overpass-api.de/api/interpreter',
data=query,
headers={'Content-Type': 'text/plain'},
timeout=30)
print(f'Status2: {r2.status_code}')
print(r2.text[:500])
"
Status: 406 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address> </body></html> Status2: 406 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address> </body></html>
5:47
The overpass-api.de server is returning 406 Not Acceptable for all requests. This is unusual. Let me check if maybe there's a proxy or something else going on. Let me check if maybe this server's IP is blocked. Let me try wget or something else. Wait - maybe the issue is that overpass-api.de has been reconfigured to require a User-Agent header or some other specific header. Let me check by looking at what a proper request looks like. Actually, the 406 error from Apache typically means the server can't serve the requested content type. In this case, I'm sending data as `application/x-www-form-urlencoded` and expecting `application/json` back. Some Overpass instances might need specific handling. Let me try to access a different mirror or use osmnx with a custom endpoint.
5:47
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Check if overpass-api.de is totally blocking or just needs different headers
# Try sending as application/json
query = '[out:json];node(1);out;'
r = requests.post('https://overpass-api.de/api/interpreter',
json={'data': query},
timeout=30)
print(f'Status: {r.status_code}')
print(r.text[:500])
"
Status: 406 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address> </body></html>
5:56
Hmm, the overpass-api.de is returning 406 for all requests from this environment. Let me try a different approach - maybe I can use the OSM API directly to get the relation data, or use a different Overpass endpoint. Let me check if there are other mirrors or if we can use a different method. Actually, looking at the error more carefully, it says "406 Not Acceptable - An appropriate representation of the requested resource could not be found on this server." This is typically Apache's response when the client sends an Accept header that can't be satisfied. But we're sending `Accept: */*` which should match anything... Wait, maybe the issue is that Apache is configured to block certain User-Agent strings. The `python-requests/2.34.2` might be blocked. Let me try with a different User-Agent. Actually, I tried that earlier with "Mozilla/5.0" and it still gave 406. Let me try yet another approach. Actually, looking at the response headers from earlier: - 'Content-Type': 'text/html; charset=iso-8859-1' The 406 is coming from Apache, not from the Overpass application itself. This suggests the request is being blocked at a proxy/CDN level. Let me try using a different host, or maybe try `http://overpass-api.de` (plain HTTP), or try with different headers. Let me try the `overpass.kumi.systems` with a higher timeout, or try an entirely different approach: download the data from a pre-downloaded file or use the OSM API directly.
5:56
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try with different User-Agent and no Accept header modification (default is */*)
# The issue might be the IP being blocked. Let's try HTTP (not HTTPS)
query = '[out:json];node(1);out;'
r = requests.post('http://overpass-api.de/api/interpreter',
data={'data': query},
headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36'},
timeout=30)
print(f'Status: {r.status_code}')
print(r.text[:500])
"
Status: 406 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 80</address> </body></html>
6:06
The overpass-api.de server is returning 406 for all requests. This might be a temporary issue or IP-based blocking. Let me try using a different approach - maybe we can use a public OSM snapshot or a different API endpoint. Let me try the `overpass.openstreetmap.fr` endpoint with proper authentication, or try to download data from other sources. Actually, let me first check if perhaps the issue is that the Overpass server is rejecting requests because of some Apache mod_security rule triggered by the query content. Let me try an extremely minimal query. Actually, wait - I got 406 for `node(1);out;` too, which is the simplest possible query. So the server is just rejecting all our requests, probably due to IP blocking. Let me try some alternatives: 1. Use a different Overpass instance 2. Use OSMnx offline capabilities 3. Download pre-processed data Let me try downloading from geofabrik or other sources, or use the overpass-api with a different approach. Actually, let me try the public instance at overpass.kumi.systems with a longer timeout.
6:06
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try kumi with longer timeout
query = '[out:json];node(1);out;'
try:
r = requests.post('https://overpass.kumi.systems/api/interpreter',
data={'data': query},
timeout=60)
print(f'Status: {r.status_code}')
print(r.text[:500])
except Exception as e:
print(f'Error: {e}')
"
Status: 504
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>
<title>OSM3S Response</title>
</head>
<body>
<p>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</p>
<p><strong style="color:#FF0000
6:53
The Overpass API instances seem to have issues. Let me try a different approach - maybe I can use the OSMnx library with a different Overpass endpoint, or I can download the data from a different source. Let me try to configure OSMnx to use a different overpass endpoint. Actually, the issue might be that overpass-api.de is blocking certain User-Agents or IPs. Let me try with a proper browser User-Agent and see if that helps.
6:53
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Let's try using osmnx's built-in function but redirect to kumi
import osmnx as ox
# First get the bounds of Tokyo as a polygon
gdf = ox.geocode_to_gdf('Tokyo, Japan')
print(gdf.shape)
print(gdf.columns.tolist())
print(gdf[['name','geometry']].head())
"
(1, 17)
['geometry', 'bbox_west', 'bbox_south', 'bbox_east', 'bbox_north', 'place_id', 'osm_type', 'osm_id', 'lat', 'lon', 'class', 'type', 'place_rank', 'importance', 'addresstype', 'name', 'display_name']
name geometry
0 Tokyo MULTIPOLYGON (((135.85369 20.42357, 135.8699 2...
7:02
The geocode worked! So the internet connection is fine. The issue is with the Overpass API. Let me try to use osmnx but configure it to use a different endpoint. Or maybe I should try a different approach - use the Nominatim geocoding results and then try to directly query Overpass with a different approach. Let me check if I can use osmnx to get the features from a polygon (the Tokyo boundary) rather than using the place name directly.
7:02
Bash
cd /work && python -c "
cd /work && python -c "
import osmnx as ox
# Use the polygon of Tokyo to query for admin_level=9 boundaries
tokyo = ox.geocode_to_gdf('Tokyo, Japan')
poly = tokyo.iloc[0].geometry
# Try to get admin_level=9 features
gdf = ox.features_from_polygon(poly, tags={'admin_level':'9'})
print(gdf.shape)
print(gdf.columns.tolist())
cols = ['name','name:en','admin_level','boundary','osm_type','osm_id']
for c in cols:
if c in gdf.columns:
print(f'{c}: {gdf[c].head(10).tolist()}')
"
/usr/local/lib/python3.14/site-packages/osmnx/_overpass.py:271: UserWarning: This area is 650 times your configured Overpass max query area size. It will automatically be divided up into multiple sub-queries accordingly. This may take a long time.
multi_poly_proj = utils_geo._consolidate_subdivide_geometry(poly_proj)
Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
(self._dns_host, self.port),
...<2 lines>...
socket_options=self.socket_options,
)
File "/usr/local/lib/python3.14/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/usr/local/lib/python3.14/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
~~~~~~~~~~~~^^^^
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 788, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 488, in _make_request
raise new_e
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 464, in _make_request
self._validate_conn(conn)
~~~~~~~~~~~~~~~~~~~^^^^^^
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 1106, in _validate_conn
conn.connect()
~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/urllib3/connection.py", line 759, in connect
self.sock = sock = self._new_conn()
~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/urllib3/connection.py", line 219, in _new_conn
raise NewConnectionError(
self, f"Failed to establish a new connection: {e}"
) from e
urllib3.exceptions.NewConnectionError: HTTPSConnection(host='overpass-api.de', port=443): Failed to establish a new connection: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/requests/adapters.py", line 696, in send
resp = conn.urlopen(
method=request.method,
...<9 lines>...
chunked=chunked,
)
File "/usr/local/lib/python3.14/site-packages/urllib3/connectionpool.py", line 842, in urlopen
retries = retries.increment(
method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
)
File "/usr/local/lib/python3.14/site-packages/urllib3/util/retry.py", line 543, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by NewConnectionError("HTTPSConnection(host='overpass-api.de', port=443): Failed to establish a new connection: [Errno 111] Connection refused"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 9, in <module>
gdf = ox.features_from_polygon(poly, tags={'admin_level':'9'})
File "/usr/local/lib/python3.14/site-packages/osmnx/features.py", line 332, in features_from_polygon
return _create_gdf(response_jsons, polygon, tags)
File "/usr/local/lib/python3.14/site-packages/osmnx/features.py", line 406, in _create_gdf
for response_json in response_jsons:
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/osmnx/_overpass.py", line 432, in _download_overpass_features
yield _overpass_request(OrderedDict(data=query_str))
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/osmnx/_overpass.py", line 469, in _overpass_request
response = requests.post(
url,
...<3 lines>...
**settings.requests_kwargs,
)
File "/usr/local/lib/python3.14/site-packages/requests/api.py", line 134, in post
return request("post", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.14/site-packages/requests/api.py", line 71, in request
return session.request(method=method, url=url, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/requests/sessions.py", line 651, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.14/site-packages/requests/sessions.py", line 784, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.14/site-packages/requests/adapters.py", line 729, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by NewConnectionError("HTTPSConnection(host='overpass-api.de', port=443): Failed to establish a new connection: [Errno 111] Connection refused"))
8:11
The overpass-api.de seems to be refusing connections. But earlier we got 406 responses, now we get connection refused. Let me try configuring osmnx to use a different endpoint. Actually, wait - the OSMnx error says "Connection refused" for overpass-api.de:443, while earlier my direct requests got 406 responses. Maybe there's some blocking happening. Let me try a different approach entirely - using OSMnx with a custom endpoint setting, or downloading data via different means. Let me try the kumi.systems endpoint with osmnx.
8:11
Bash
cd /work && python -c "
cd /work && python -c "
import osmnx as ox
# Configure osmnx to use kumi.systems
ox.settings.overpass_endpoint = 'https://overpass.kumi.systems/api/interpreter'
ox.settings.timeout = 120
# Get Tokyo polygon first
tokyo = ox.geocode_to_gdf('Tokyo, Japan')
poly = tokyo.iloc[0].geometry
# Try with a small query first
try:
gdf = ox.features_from_polygon(poly, tags={'admin_level':'9'})
print(gdf.shape)
print(gdf.columns.tolist())
except Exception as e:
print(f'Error: {type(e).__name__}: {e}')
"
Error: ConnectionError: HTTPSConnectionPool(host='overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by NewConnectionError("HTTPSConnection(host='overpass-api.de', port=443): Failed to establish a new connection: [Errno 111] Connection refused"))
/usr/local/lib/python3.14/site-packages/osmnx/_overpass.py:271: UserWarning: This area is 650 times your configured Overpass max query area size. It will automatically be divided up into multiple sub-queries accordingly. This may take a long time. multi_poly_proj = utils_geo._consolidate_subdivide_geometry(poly_proj)
9:17
It seems like osmnx is ignoring the settings override and still connecting to overpass-api.de. Let me check the osmnx source code to see what's happening, and perhaps directly use requests with kumi.systems.
9:17
Bash
cd /work && python -c "
cd /work && python -c "
# Check osmnx default endpoint
import osmnx as ox
print('Default endpoint:', ox.settings.overpass_endpoint)
print('Settings timeout:', ox.settings.timeout)
print('Requests per second:', ox.settings.requests_per_second)
"
Traceback (most recent call last):
File "<string>", line 4, in <module>
print('Default endpoint:', ox.settings.overpass_endpoint)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'osmnx.settings' has no attribute 'overpass_endpoint'
9:25
The overpass-api.de server is returning 406 Not Acceptable errors, which suggests it might be blocking based on some header. Let me check what version of osmnx we have and how to configure the endpoint.
9:25
Bash
cd /work && python -c "
cd /work && python -c "
import osmnx as ox
print(ox.__version__)
# Check all settings
for attr in dir(ox.settings):
if not attr.startswith('_'):
try:
val = getattr(ox.settings, attr)
if not callable(val):
print(f'{attr} = {val}')
except:
pass
"
2.1.0
TYPE_CHECKING = False
all_oneway = False
annotations = _Feature((3, 7, 0, 'beta', 1), None, 16777216)
bidirectional_network_types = ['walk']
cache_folder = ./cache
cache_only_mode = False
data_folder = ./data
default_access = ["access"!~"private"]
default_crs = epsg:4326
doh_url_template = https://8.8.8.8/resolve?name={hostname}
elevation_url_template = https://maps.googleapis.com/maps/api/elevation/json?locations={locations}&key={key}
http_accept_language = en
http_referer = OSMnx Python package (https://github.com/gboeing/osmnx)
http_user_agent = OSMnx Python package (https://github.com/gboeing/osmnx)
imgs_folder = ./images
lg = <module 'logging' from '/usr/local/lib/python3.14/logging/__init__.py'>
log_console = False
log_file = False
log_filename = osmnx
log_level = 20
log_name = OSMnx
logs_folder = ./logs
max_query_area_size = 2500000000
nominatim_key = None
nominatim_url = https://nominatim.openstreetmap.org/
overpass_memory = None
overpass_rate_limit = True
overpass_settings = [out:json][timeout:{timeout}]{maxsize}
overpass_url = https://overpass-api.de/api
requests_kwargs = {}
requests_timeout = 180
use_cache = True
useful_tags_node = ['highway', 'junction', 'railway', 'ref']
useful_tags_way = ['access', 'area', 'bridge', 'est_width', 'highway', 'junction', 'landuse', 'lanes', 'maxspeed', 'name', 'oneway', 'ref', 'service', 'tunnel', 'width']
9:31
I see - osmnx uses `overpass_url` not `overpass_endpoint`. Let me try setting that to kumi.systems.
9:31
Bash
cd /work && python -c "
cd /work && python -c "
import osmnx as ox
# Configure osmnx to use different endpoint
ox.settings.overpass_url = 'https://overpass.kumi.systems/api'
ox.settings.requests_timeout = 600
tokyo = ox.geocode_to_gdf('Tokyo, Japan')
poly = tokyo.iloc[0].geometry
try:
gdf = ox.features_from_polygon(poly, tags={'admin_level':'9'})
print('Success! Shape:', gdf.shape)
print(gdf.columns.tolist())
cols = ['name','name:en','admin_level','boundary','osm_type','osm_id']
for c in cols:
if c in gdf.columns:
print(f'{c}: {gdf[c].head(25).tolist()}')
except Exception as e:
print(f'Error: {type(e).__name__}: {e}')
import traceback
traceback.print_exc()
"
<exit_code>-1</exit_code> <error>Command timed out after 600000ms</error>
19:42
The overpass servers are not responding from this environment. Let me try a different approach - I can download a pre-existing OSM extract or use a different data source. Let me check if there's any OSM data already available, or try using the OSM SF bay or planet file. Alternatively, I could try using Nominatim to get the ward boundaries directly, or use a GeoJSON source. Actually, let me check if the Python environment can reach any Overpass endpoint. The issue might be that overpass-api.de is giving 406 errors, and kumi.systems is timing out. Let me try another approach - maybe I can use the OSM API directly or use a different endpoint. Wait, looking more carefully at the error - overpass-api.de returned 406 "Not Acceptable" for POST requests. This is unusual. Let me check if there's a proxy issue or a different version of the API. Actually, the 406 might be because the server is blocking based on some criteria. Let me try to use the OSM API directly (not Overpass) to get the relations, or use a different API endpoint. Actually, let me think about this differently. There are many ways to get ward boundaries for Tokyo without using Overpass. I could: 1. Use Natural Earth data 2. Use a pre-packaged GeoJSON from a GitHub repo 3. Use the OSM API to get specific relations 4. Use PyPI package like `japan-topography` or similar Let me try to directly get the OSM relation data via the OSM API (api.openstreetmap.org).
19:42
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try OSM API directly to get Tokyo wards
# The 23 special wards have relation IDs. Let's find them.
# First, get the Tokyo relation
r = requests.get('https://api.openstreetmap.org/api/0.6/relation/1543125', timeout=30)
print(r.status_code)
print(r.text[:1000])
"
200 <?xml version="1.0" encoding="UTF-8"?> <osm version="0.6" generator="openstreetmap-cgimap 2.1.0 (1317 spike-08.openstreetmap.org)" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/"> <relation id="1543125" visible="true" version="382" changeset="180192345" timestamp="2026-03-21T08:07:52Z" user="Gin-an S" uid="22613668"> <member type="node" ref="265018692" role="admin_centre"/> <member type="way" ref="285975526" role="outer"/> <member type="way" ref="357954032" role="outer"/> <member type="way" ref="269179478" role="outer"/> <member type="way" ref="355233089" role="outer"/> <member type="way" ref="262103468" role="outer"/> <member type="way" ref="355231075" role="outer"/> <member type="way" ref="355231074" role="outer"/> <member type="way" ref="269179481" role="outer"/> <member type="way" ref="269179477" role="outer"/> <member type="way" ref="355233087" role="outer
19:54
Great! I can use the OSM API directly. Let me find the 23 special wards of Tokyo. The Tokyo prefecture relation ID is 1543125. The special wards are admin_level=9 boundaries within Tokyo. Let me look them up. Actually, I know the 23 special wards of Tokyo. They are: 1. Adachi (足立区) 2. Arakawa (荒川区) 3. Bunkyo (文京区) 4. Chiyoda (千代田区) 5. Chuo (中央区) 6. Edogawa (江戸川区) 7. Itabashi (板橋区) 8. Katsushika (葛飾区) 9. Kita (北区) 10. Koto (江東区) 11. Meguro (目黒区) 12. Minato (港区) 13. Nakano (中野区) 14. Nerima (練馬区) 15. Ota (大田区) 16. Setagaya (世田谷区) 17. Shibuya (渋谷区) 18. Shinagawa (品川区) 19. Shinjuku (新宿区) 20. Suginami (杉並区) 21. Sumida (墨田区) 22. Taito (台東区) 23. Toshima (豊島区) These are relations with admin_level=9. Let me use the OSM API to query for relations by bounding box or by API queries. Actually, I can use the Overpass API... but it's refusing connections. Let me try using the OSM API directly. Let me use the OSM API to get the relation data for each ward. But first, I need to find the relation IDs. Actually, let me try using the Nominatim API to find the 23 special wards.
19:54
Bash
cd /work && python -c "
cd /work && python -c "
# Let's find the 23 special wards via Nominatim
import requests
# Tokyo special wards are admin_level=9 in OSM
# Let's query Nominatim for the list
r = requests.get('https://nominatim.openstreetmap.org/search?q=Tokyo+special+wards&format=json&limit=50',
headers={'User-Agent': 'Mozilla/5.0'}, timeout=30)
if r.status_code == 200:
data = r.json()
for d in data:
print(d.get('osm_id'), d.get('osm_type'), d.get('display_name')[:80])
print('---')
# Also try searching for each ward individually
wards_en = [
'Adachi', 'Arakawa', 'Bunkyo', 'Chiyoda', 'Chuo', 'Edogawa', 'Itabashi',
'Katsushika', 'Kita', 'Koto', 'Meguro', 'Minato', 'Nakano', 'Nerima',
'Ota', 'Setagaya', 'Shibuya', 'Shinagawa', 'Shinjuku', 'Suginami',
'Sumida', 'Taito', 'Toshima'
]
for w in wards_en:
r = requests.get(f'https://nominatim.openstreetmap.org/search?city={w}&county=Tokyo&country=Japan&format=json&limit=1',
headers={'User-Agent': 'Mozilla/5.0'}, timeout=30)
if r.status_code == 200:
data = r.json()
if data:
print(f'{w}: osm_id={data[0].get(\"osm_id\")}, type={data[0].get(\"osm_type\")}')
else:
print(f'{w}: NOT FOUND')
"
---
20:42
The Nominatim search didn't return useful results. Let me try a different approach - use the OSM API directly to find the relations. Actually, I know the Tokyo 23 special wards OSM relation IDs. Let me look them up. The Tokyo prefecture relation is 1543125. The 23 special wards are: 1. Chiyoda (relation...) 2. Chuo 3. Minato 4. Shinjuku 5. Bunkyo 6. Taito 7. Sumida 8. Koto 9. Shinagawa 10. Meguro 11. Ota 12. Setagaya 13. Shibuya 14. Nakano 15. Suginami 16. Toshima 17. Kita 18. Arakawa 19. Itabashi 20. Nerima 21. Adachi 22. Katsushika 23. Edogawa Let me query the OSM API to find them. Actually, let me try a different approach - use Overpass XML format in case the issue is with the content negotiation. Wait, actually the 406 error on overpass-api.de is strange. Let me check if there's an issue with how requests library sends the Accept header. The default Accept is `*/*` which should be fine... Let me try a different approach - use the OSM API to get the members of the Tokyo relation, which should include the ward boundaries.
20:42
Bash
cd /work && python -c "
cd /work && python -c "
# Get all members of Tokyo relation to find ward relations
import requests
r = requests.get('https://api.openstreetmap.org/api/0.6/relation/1543125/full', timeout=30)
print(r.status_code)
# Parse XML
import xml.etree.ElementTree as ET
root = ET.fromstring(r.content)
# Find all relations
for rel in root.findall('.//relation'):
tags = {t.get('k'): t.get('v') for t in rel.findall('tag')}
if tags.get('admin_level') == '9':
print(f'Relation {rel.get(\"id\")}: {tags.get(\"name:en\",\"\")} / {tags.get(\"name\",\"\")}')
"
200
21:12
Hmm, the /full endpoint didn't return all relations. Let's try a different approach - maybe we need to get the members of the Tokyo relation and then fetch each one.
21:12
Bash
cd /work && python -c "
cd /work && python -c "
import requests
import xml.etree.ElementTree as ET
# Get Tokyo relation members
r = requests.get('https://api.openstreetmap.org/api/0.6/relation/1543125', timeout=30)
print('Status:', r.status_code)
root = ET.fromstring(r.content)
rel = root.find('.//relation')
print('Tags:')
tags = {t.get('k'): t.get('v') for t in rel.findall('tag')}
for k, v in sorted(tags.items()):
print(f' {k} = {v}')
print()
print('Members:')
for m in rel.findall('member'):
print(f' type={m.get(\"type\")} ref={m.get(\"ref\")} role={m.get(\"role\")}')
"
Status: 200 Tags: ISO3166-2 = JP-13 admin_level = 4 alt_name:be = Токіё alt_name:hr = Tokyo alt_name:la = Tocium boundary = administrative is_in:country = Japan is_in:country_code = JP name = 東京都 name:be = Токіа name:br = Tokyo name:cnr = Tokio name:cs = Tokio name:de = Tokio name:en = Tokyo name:eo = Tokio name:es = Tokio name:fi = Tokio name:fr = Tokyo name:hr = Tokio name:ia = Tokyo name:io = Tokyo name:it = Tokyo name:ja = 東京都 name:ja-Hira = とうきょうと name:ja-Latn = Tōkyō-to name:ko = 도쿄도 name:la = Tokium name:mi = Tōkio name:nl = Tokio name:pl = Tokio name:pt = Tóquio name:ru = Токио name:szl = Tokjo name:tr = Tokyo name:uk = Префектура Токіо name:zh = 东京都/東京都 name:zh-Hans = 东京都 name:zh-Hant = 東京都 official_name:it = Metropoli di Tokyo place = province ref = 130001 type = boundary website = https://www.metro.tokyo.lg.jp/ wikidata = Q1490 wikipedia = ja:東京都 Members: type=node ref=265018692 role=admin_centre type=way ref=285975526 role=outer type=way ref=357954032 role=outer type=way ref=269179478 role=outer type=way ref=355233089 role=outer type=way ref=262103468 role=outer type=way ref=355231075 role=outer type=way ref=355231074 role=outer type=way ref=269179481 role=outer type=way ref=269179477 role=outer type=way ref=355233087 role=outer type=way ref=355233088 role=outer type=way ref=353333449 role=outer type=way ref=355233085 role=outer type=way ref=353333452 role=outer type=way ref=353333446 role=outer type=way ref=285973711 role=outer type=way ref=204519328 role=outer type=way ref=262103428 role=outer type=way ref=262103456 role=outer type=way ref=262103432 role=outer type=way ref=262103459 role=outer type=way ref=262103429 role=outer type=way ref=262103431 role=outer type=way ref=262103458 role=outer type=way ref=325387789 role=outer type=way ref=262103430 role=outer type=way ref=356300531 role=outer type=way ref=361145583 role=outer type=way ref=361216857 role=outer type=way ref=475417984 role=outer type=way ref=475417983 role=outer type=way ref=564029331 role=outer type=way ref=1222683859 role=outer type=way ref=1391972172 role=outer type=way ref=1391972175 role=outer type=way ref=1391723583 role=outer type=way ref=1391723582 role=outer type=way ref=355038055 role=outer type=way ref=1391723595 role=outer type=way ref=1391723592 role=outer type=way ref=1391723586 role=outer type=way ref=1391269180 role=outer type=way ref=1391275041 role=outer type=way ref=1391283420 role=outer type=way ref=1391283426 role=outer type=way ref=1391283423 role=outer type=way ref=1391283424 role=outer type=way ref=1391467460 role=outer type=way ref=268557504 role=outer type=way ref=355051820 role=outer type=way ref=1409933240 role=outer type=way ref=1411708893 role=outer type=way ref=355051970 role=outer type=way ref=1411708896 role=outer type=way ref=355097036 role=outer type=way ref=1412633444 role=outer type=way ref=1412633447 role=outer type=way ref=1424306533 role=outer type=way ref=1423024383 role=outer type=way ref=1425959512 role=outer type=way ref=1425959515 role=outer type=way ref=1429639822 role=outer type=way ref=1430223215 role=outer type=way ref=1430223219 role=outer type=way ref=268545112 role=outer type=way ref=1431609528 role=outer type=way ref=1431609536 role=outer type=way ref=1431609527 role=outer type=way ref=1435217711 role=outer type=way ref=268227537 role=outer type=way ref=1435217709 role=outer type=way ref=1436380974 role=outer type=way ref=1435217700 role=outer type=way ref=1436380971 role=outer type=way ref=1435220560 role=outer type=way ref=1436380978 role=outer type=way ref=1436380981 role=outer type=way ref=654409021 role=outer type=way ref=1443129974 role=outer type=way ref=1443129971 role=outer type=way ref=355114855 role=outer type=way ref=1443129981 role=outer type=way ref=1443129964 role=outer type=way ref=1443129954 role=outer type=way ref=1443129967 role=outer type=way ref=1443129948 role=outer type=way ref=1433486042 role=outer type=way ref=1433486035 role=outer type=way ref=1433486037 role=outer type=way ref=1444814695 role=outer type=way ref=1446634530 role=outer type=way ref=1444678659 role=outer type=way ref=1444678675 role=outer type=way ref=1444678674 role=outer type=way ref=1444678661 role=outer type=way ref=1444678660 role=outer type=way ref=1444678668 role=outer type=way ref=1444678671 role=outer type=way ref=1444818674 role=outer type=way ref=355114853 role=outer type=way ref=1453004886 role=outer type=way ref=1453004887 role=outer type=way ref=1452757476 role=outer type=way ref=1452999266 role=outer type=way ref=355114849 role=outer type=way ref=1452757487 role=outer type=way ref=1452757490 role=outer type=way ref=1352440064 role=outer type=way ref=1352440070 role=outer type=way ref=1352508823 role=outer type=way ref=267164296 role=outer type=way ref=1352508827 role=outer type=way ref=1352508820 role=outer type=way ref=267163987 role=outer type=way ref=1352486524 role=outer type=way ref=1352508838 role=outer type=way ref=1352508818 role=outer type=way ref=355213607 role=outer type=way ref=1350350054 role=outer type=way ref=1350350056 role=outer type=way ref=1350307431 role=outer type=way ref=1350307427 role=outer type=way ref=1350307428 role=outer type=way ref=1308358698 role=outer type=way ref=1308358692 role=outer type=way ref=267024970 role=outer type=way ref=254159772 role=outer type=way ref=254159278 role=outer type=way ref=254159275 role=outer type=way ref=254159279 role=outer type=way ref=1228037870 role=outer type=way ref=267020331 role=outer type=way ref=1320832153 role=outer type=way ref=1320832152 role=outer type=way ref=1320832151 role=outer type=way ref=1320832150 role=outer type=way ref=1320832149 role=outer type=way ref=343691211 role=outer type=way ref=254126987 role=outer type=way ref=1320832148 role=outer type=way ref=1323863853 role=outer type=way ref=1323744399 role=outer type=way ref=1325261540 role=outer type=way ref=1325568209 role=outer type=way ref=1325261525 role=outer type=way ref=1325261532 role=outer type=way ref=254126435 role=outer type=way ref=1325261534 role=outer type=way ref=1219264171 role=outer type=way ref=254126432 role=outer type=way ref=1219264170 role=outer type=way ref=254125380 role=outer type=way ref=254125374 role=outer type=way ref=254125379 role=outer type=way ref=254125383 role=outer type=way ref=254125375 role=outer type=way ref=254125376 role=outer type=way ref=254124661 role=outer type=way ref=1221210853 role=outer type=way ref=266765067 role=outer type=way ref=254124647 role=outer type=way ref=254124649 role=outer type=way ref=1221210851 role=outer type=way ref=1221210844 role=outer type=way ref=254103881 role=outer type=way ref=1221210846 role=outer type=way ref=1221231841 role=outer type=way ref=1221231845 role=outer type=way ref=254103890 role=outer type=way ref=254103891 role=outer type=way ref=254103893 role=outer type=way ref=254086494 role=outer type=way ref=1357104056 role=outer type=way ref=1222383272 role=outer type=way ref=1222383263 role=outer type=way ref=130770864 role=outer type=way ref=1357104078 role=outer type=way ref=1222383261 role=outer type=way ref=693992296 role=outer type=way ref=693992298 role=outer type=way ref=1357104087 role=outer type=way ref=1319005304 role=outer type=way ref=254086492 role=outer type=way ref=1357104106 role=outer type=way ref=1318934471 role=outer type=way ref=254086496 role=outer type=way ref=1318933162 role=outer type=way ref=1318933167 role=outer type=way ref=199571326 role=outer type=way ref=1318933173 role=outer type=way ref=1318095641 role=outer type=way ref=1318095639 role=outer type=way ref=1317569093 role=outer type=way ref=1317569098 role=outer type=way ref=1317569087 role=outer type=way ref=1310779772 role=outer type=way ref=1310779778 role=outer type=way ref=1310779769 role=outer type=way ref=1310779775 role=outer type=way ref=1310779785 role=outer type=way ref=1310809994 role=outer type=way ref=1310809979 role=outer type=way ref=1310809988 role=outer type=way ref=1310966628 role=outer type=way ref=1310966625 role=outer type=way ref=556503229 role=outer type=way ref=1310966642 role=outer type=way ref=1310966635 role=outer type=way ref=1310973412 role=outer type=way ref=1310973397 role=outer type=way ref=1310973395 role=outer type=way ref=779910053 role=outer type=way ref=219620009 role=outer type=way ref=779910067 role=outer type=way ref=1311024508 role=outer type=way ref=778411659 role=outer type=way ref=778732243 role=outer type=way ref=778542385 role=outer type=way ref=778411654 role=outer type=way ref=1311024511 role=outer type=way ref=778411650 role=outer type=way ref=1312361062 role=outer type=way ref=778411647 role=outer type=way ref=778411672 role=outer type=way ref=778411678 role=outer type=way ref=1312361065 role=outer type=way ref=1312361067 role=outer type=way ref=1312361072 role=outer type=way ref=199571231 role=outer type=way ref=1313882564 role=outer type=way ref=1312361059 role=outer type=way ref=1313882567 role=outer type=way ref=1379884669 role=outer type=way ref=1379318200 role=outer type=way ref=1378674363 role=outer type=way ref=1313882580 role=outer type=way ref=1378674350 role=outer type=way ref=254086493 role=outer type=way ref=1378674361 role=outer type=way ref=1375320804 role=outer type=way ref=1313882561 role=outer type=way ref=1375320810 role=outer type=way ref=1313918971 role=outer type=way ref=1375320805 role=outer type=way ref=727555257 role=outer type=way ref=355229463 role=outer type=way ref=355229462 role=outer type=way ref=359454122 role=outer type=way ref=358998006 role=outer type=way ref=285975524 role=outer type=way ref=266521383 role=outer type=way ref=1320748074 role=outer type=way ref=219512081 role=outer type=way ref=1221758516 role=outer type=way ref=1187575698 role=outer type=way ref=228183375 role=outer type=way ref=1187575703 role=outer type=way ref=1187575701 role=outer type=way ref=219515120 role=outer type=way ref=219512080 role=outer type=way ref=1187575696 role=outer type=way ref=228217966 role=outer type=way ref=1189782178 role=outer type=way ref=1189785841 role=outer type=way ref=228217963 role=outer type=way ref=228214929 role=outer type=way ref=1189785839 role=outer type=way ref=1314069271 role=outer type=way ref=200164002 role=outer type=way ref=1314069273 role=outer type=way ref=1189785847 role=outer type=way ref=1189785850 role=outer type=way ref=955225506 role=outer type=way ref=1189785854 role=outer type=way ref=962868685 role=outer type=way ref=962874837 role=outer type=way ref=1169508350 role=outer type=way ref=1169394573 role=outer type=way ref=1225131130 role=outer type=way ref=1169394576 role=outer type=way ref=1314072160 role=outer type=way ref=1314072163 role=outer type=way ref=200164606 role=outer type=way ref=1314072157 role=outer type=way ref=1169394598 role=outer type=way ref=1314165614 role=outer type=way ref=962877656 role=outer type=way ref=1315625977 role=outer type=way ref=932671665 role=outer type=way ref=1315627623 role=outer type=way ref=1315627621 role=outer type=way ref=932671663 role=outer type=way ref=1315627618 role=outer type=way ref=1315640307 role=outer type=way ref=947988657 role=outer type=way ref=1315640305 role=outer type=way ref=947990646 role=outer type=way ref=1315640321 role=outer type=way ref=648700095 role=outer type=way ref=1169335472 role=outer type=way ref=1169335468 role=outer type=way ref=1315640329 role=outer type=way ref=254067305 role=outer type=way ref=1315640338 role=outer type=way ref=200163367 role=outer type=way ref=1169335437 role=outer type=way ref=1342561941 role=outer type=way ref=1340662120 role=outer type=way ref=1224773998 role=outer type=way ref=1169635223 role=outer type=way ref=1340662124 role=outer type=way ref=1169635220 role=outer type=way ref=1340662107 role=outer type=way ref=1340662083 role=outer type=way ref=1169635196 role=outer type=way ref=1169635178 role=outer type=way ref=1340662086 role=outer type=way ref=1169731187 role=outer type=way ref=200163534 role=outer type=way ref=1340402679 role=outer type=way ref=1339158328 role=outer type=way ref=1169731082 role=outer type=way ref=1337289838 role=outer type=way ref=355213604 role=outer type=way ref=1337289865 role=outer type=way ref=1222757998 role=outer type=way ref=1222758007 role=outer type=way ref=1222758020 role=outer type=way ref=1222758010 role=outer type=way ref=1222758019 role=outer type=way ref=312850200 role=outer type=way ref=1222758018 role=outer type=way ref=1222280847 role=outer type=way ref=1222280860 role=outer type=way ref=1222280859 role=outer type=way ref=1222280855 role=outer type=way ref=1222280843 role=outer type=way ref=1222492947 role=outer type=way ref=312850183 role=outer type=way ref=1174838480 role=outer type=way ref=1174838487 role=outer type=way ref=1174838478 role=outer type=way ref=1222501276 role=outer type=way ref=1222501277 role=outer type=way ref=1222501266 role=outer type=way ref=1222683866 role=outer type=way ref=200164031 role=outer type=way ref=1226691758 role=outer type=way ref=1226691775 role=outer type=way ref=287004404 role=outer type=way ref=1226676292 role=outer type=way ref=1226676295 role=outer type=way ref=1226676304 role=outer type=way ref=1226676307 role=outer type=way ref=1226676310 role=outer type=way ref=1225951993 role=outer type=way ref=1225951997 role=outer type=way ref=1226224707 role=outer type=way ref=1226224702 role=outer type=way ref=1226459183 role=outer type=way ref=1225951998 role=outer type=way ref=1226571627 role=outer type=way ref=1226571616 role=outer type=way ref=1226571625 role=outer type=way ref=343839080 role=outer type=way ref=200162985 role=outer type=way ref=1226224691 role=outer type=way ref=1226224690 role=outer type=way ref=1226224697 role=outer type=way ref=1225952011 role=outer type=way ref=1225952029 role=outer type=way ref=1225952030 role=outer type=way ref=494960105 role=outer type=way ref=494960100 role=outer type=way ref=494960106 role=outer type=way ref=200163695 role=outer type=way ref=1183879657 role=outer type=way ref=1176295276 role=outer type=way ref=1175506458 role=outer type=way ref=200163963 role=outer type=way ref=1175508552 role=outer type=way ref=1176302119 role=outer type=way ref=1176302107 role=outer type=way ref=1176302116 role=outer type=way ref=200163969 role=outer type=way ref=1179774405 role=outer type=way ref=1179774404 role=outer type=way ref=587419831 role=outer type=way ref=200163824 role=outer type=way ref=1229054746 role=outer type=way ref=200164390 role=outer type=way ref=200164378 role=outer type=way ref=1232811271 role=outer type=way ref=1232811272 role=outer type=way ref=200164557 role=outer type=way ref=200164374 role=outer type=way ref=343839086 role=outer type=way ref=355051843 role=outer type=way ref=354979621 role=outer type=way ref=254107499 role=outer type=way ref=354863117 role=outer type=way ref=338652710 role=outer type=way ref=338651744 role=outer type=way ref=354863121 role=outer type=way ref=90483042 role=outer type=way ref=266369019 role=outer type=way ref=271667932 role=outer type=way ref=354863119 role=outer type=way ref=1408466500 role=outer type=way ref=1408466503 role=outer type=way ref=266373685 role=outer type=way ref=1408466508 role=outer type=way ref=268599343 role=outer type=way ref=1408466514 role=outer type=way ref=268599342 role=outer type=way ref=1408466516 role=outer type=relation ref=5100276 role=subarea type=relation ref=1759475 role=subarea type=relation ref=3606141 role=subarea type=relation ref=3606090 role=subarea type=relation ref=3605944 role=subarea type=relation ref=3605930 role=subarea type=relation ref=1761717 role=subarea type=relation ref=3554304 role=subarea type=relation ref=1760119 role=subarea type=relation ref=1543055 role=subarea type=relation ref=1759474 role=subarea type=relation ref=1543056 role=subarea type=relation ref=1758878 role=subarea type=relation ref=1758891 role=subarea type=relation ref=3554015 role=subarea type=relation ref=1761743 role=subarea type=relation ref=1761718 role=subarea type=relation ref=1760124 role=subarea type=relation ref=1760038 role=subarea type=relation ref=1760078 role=subarea type=relation ref=1758858 role=subarea type=relation ref=1759477 role=subarea type=relation ref=1759506 role=subarea type=relation ref=1758936 role=subarea type=relation ref=1758897 role=subarea type=relation ref=1760040 role=subarea type=relation ref=1758888 role=subarea type=relation ref=1758947 role=subarea type=relation ref=1761742 role=subarea type=relation ref=3807613 role=subarea type=relation ref=5299312 role=subarea type=relation ref=1761834 role=subarea type=relation ref=5301559 role=subarea type=relation ref=5301643 role=subarea type=relation ref=5301640 role=subarea type=relation ref=5301641 role=subarea type=relation ref=1761771 role=subarea type=relation ref=5301639 role=subarea type=relation ref=5301790 role=subarea type=relation ref=5301789 role=subarea type=relation ref=5302158 role=subarea type=relation ref=5302157 role=subarea type=relation ref=5302160 role=subarea type=relation ref=5302161 role=subarea type=relation ref=5302159 role=subarea type=relation ref=5302329 role=subarea type=relation ref=5302328 role=subarea type=relation ref=5302330 role=subarea type=relation ref=5302326 role=subarea type=relation ref=5302327 role=subarea type=relation ref=5303898 role=subarea type=relation ref=1761820 role=subarea type=relation ref=1761804 role=subarea type=relation ref=5301344 role=subarea
21:27
The Tokyo relation has many subarea relations. Let me find the 23 special wards specifically. The special wards should have admin_level=9. Let me look up a few of these relation IDs to see which ones are the 23 special wards. Actually, I can query the OSM API for each relation to check their tags. But first let me look for known IDs. The 23 special wards of Tokyo are: 1. Adachi (足立区) - relation 1761717 2. Arakawa (荒川区) - relation 1760124 3. Bunkyo (文京区) - relation 1760040 4. Chiyoda (千代田区) - relation 1760119 5. Chuo (中央区) - relation 1543055 6. Edogawa (江戸川区) - relation 1761718 7. Itabashi (板橋区) - relation 1761743 8. Katsushika (葛飾区) - relation 1761742 9. Kita (北区) - relation 1761804 10. Koto (江東区) - relation 1543056 11. Meguro (目黒区) - relation 1759475 12. Minato (港区) - relation 1759474 13. Nakano (中野区) - relation 1761820 14. Nerima (練馬区) - relation 1761771 15. Ota (大田区) - relation 1758878 16. Setagaya (世田谷区) - relation 1758891 17. Shibuya (渋谷区) - relation 1758897 18. Shinagawa (品川区) - relation 1759477 19. Shinjuku (新宿区) - relation 1758858 20. Suginami (杉並区) - relation 1758936 21. Sumida (墨田区) - relation 1758947 22. Taito (台東区) - relation 1758888 23. Toshima (豊島区) - relation 1759506 These are the 23 special wards. Let me verify a few. Actually, I notice there are many subarea relations. Let me just get the admin_level=9 relations from the Tokyo relation members.
21:27
Bash
cd /work && python -c "
cd /work && python -c "
import requests
import xml.etree.ElementTree as ET
# Known 23 special ward relation IDs from OSM
# Let's verify them
wards = {
1761717: 'Adachi',
1760124: 'Arakawa',
1760040: 'Bunkyo',
1760119: 'Chiyoda',
1543055: 'Chuo',
1761718: 'Edogawa',
1761743: 'Itabashi',
1761742: 'Katsushika',
1761804: 'Kita',
1543056: 'Koto',
1759475: 'Meguro',
1759474: 'Minato',
1761820: 'Nakano',
1761771: 'Nerima',
1758878: 'Ota',
1758891: 'Setagaya',
1758897: 'Shibuya',
1759477: 'Shinagawa',
1758858: 'Shinjuku',
1758936: 'Suginami',
1758947: 'Sumida',
1758888: 'Taito',
1759506: 'Toshima',
}
for rid, expected in wards.items():
r = requests.get(f'https://api.openstreetmap.org/api/0.6/relation/{rid}', timeout=30)
root = ET.fromstring(r.content)
rel = root.find('.//relation')
tags = {t.get('k'): t.get('v') for t in rel.findall('tag')}
name_en = tags.get('name:en', '')
admin = tags.get('admin_level', '')
print(f'{rid}: {name_en:20s} admin={admin} expected={expected:15s} match={name_en.lower().startswith(expected.lower())}')
"
1761717: Minato admin=7 expected=Adachi match=False 1760124: Adachi admin=7 expected=Arakawa match=False 1760040: Arakawa admin=7 expected=Bunkyo match=False 1760119: Nerima admin=7 expected=Chiyoda match=False 1543055: Suginami admin=7 expected=Chuo match=False 1761718: Katsushika admin=7 expected=Edogawa match=False 1761743: Edogawa admin=7 expected=Itabashi match=False 1761742: Chiyoda admin=7 expected=Katsushika match=False 1761804: Musashino admin=7 expected=Kita match=False 1543056: Nakano admin=7 expected=Koto match=False 1759475: Komae admin=7 expected=Meguro match=False 1759474: Setagaya admin=7 expected=Minato match=False 1761820: Mitaka admin=7 expected=Nakano match=False 1761771: Tachikawa admin=7 expected=Nerima match=False 1758878: Bunkyō admin=7 expected=Ota match=False 1758891: Sumida admin=7 expected=Setagaya match=False 1758897: Chuo admin=7 expected=Shibuya match=False 1759477: Shibuya admin=7 expected=Shinagawa match=False 1758858: Shinjuku admin=7 expected=Shinjuku match=True 1758936: Meguro admin=7 expected=Suginami match=False 1758947: Ota admin=7 expected=Sumida match=False 1758888: Taito admin=7 expected=Taito match=True 1759506: Toshima admin=7 expected=Toshima match=True
21:41
Interesting - the admin_level for these is 7, not 9! And some of my ID mappings were wrong. Let me get the correct ones. The 23 special wards of Tokyo have admin_level=7 in OSM (they're cities/special wards, not just administrative subdivisions at level 9). Let me look up the correct IDs by searching for each ward name in Tokyo.
21:41
Bash
cd /work && python -c "
cd /work && python -c "
import requests
import xml.etree.ElementTree as ET
# Let's find the actual 23 special wards of Tokyo.
# They are admin_level=7 and subarea of Tokyo (relation 1543125)
# Let me get all relations that are subarea members of Tokyo
# Get Tokyo relation with all members
r = requests.get('https://api.openstreetmap.org/api/0.6/relation/1543125', timeout=30)
root = ET.fromstring(r.content)
rel = root.find('.//relation')
# Find members with role=subarea that are relations
subarea_ids = []
for m in rel.findall('member'):
if m.get('type') == 'relation' and m.get('role') == 'subarea':
subarea_ids.append(m.get('ref'))
print(f'Found {len(subarea_ids)} subarea relations')
print(subarea_ids)
# Now get details for each
ward_list = []
for rid in subarea_ids:
r = requests.get(f'https://api.openstreetmap.org/api/0.6/relation/{rid}', timeout=30)
root = ET.fromstring(r.content)
rel = root.find('.//relation')
tags = {t.get('k'): t.get('v') for t in rel.findall('tag')}
name_en = tags.get('name:en', '')
name = tags.get('name', '')
admin = tags.get('admin_level', '')
boundary = tags.get('boundary', '')
print(f'{rid}: name_en={name_en:30s} name={name:20s} admin_level={admin} boundary={boundary}')
# Only keep the 23 special wards: admin_level=7, boundary=administrative
ward_list.append((rid, name_en, name, admin, boundary))
print()
print('--- Filtered to special wards ---')
special = [w for w in ward_list if w[3] == '7']
for w in special:
print(f'{w[0]}: {w[1]:30s} {w[2]:20s}')
"
Found 54 subarea relations ['5100276', '1759475', '3606141', '3606090', '3605944', '3605930', '1761717', '3554304', '1760119', '1543055', '1759474', '1543056', '1758878', '1758891', '3554015', '1761743', '1761718', '1760124', '1760038', '1760078', '1758858', '1759477', '1759506', '1758936', '1758897', '1760040', '1758888', '1758947', '1761742', '3807613', '5299312', '1761834', '5301559', '5301643', '5301640', '5301641', '1761771', '5301639', '5301790', '5301789', '5302158', '5302157', '5302160', '5302161', '5302159', '5302329', '5302328', '5302330', '5302326', '5302327', '5303898', '1761820', '1761804', '5301344'] 5100276: name_en=Machida name=町田市 admin_level=7 boundary=administrative 1759475: name_en=Komae name=狛江市 admin_level=7 boundary=administrative 3606141: name_en=Ogasawara Subprefecture name=小笠原支庁 admin_level=5 boundary=administrative 3606090: name_en=Hachijō Subprefecture name=八丈支庁 admin_level=5 boundary=administrative 3605944: name_en=Miyake Subprefecture name=三宅支庁 admin_level=5 boundary=administrative 3605930: name_en=Oshima Subprefecture name=大島支庁 admin_level=5 boundary=administrative 1761717: name_en=Minato name=港区 admin_level=7 boundary=administrative 3554304: name_en=Shinagawa name=品川区 admin_level=7 boundary=administrative 1760119: name_en=Nerima name=練馬区 admin_level=7 boundary=administrative 1543055: name_en=Suginami name=杉並区 admin_level=7 boundary=administrative 1759474: name_en=Setagaya name=世田谷区 admin_level=7 boundary=administrative 1543056: name_en=Nakano name=中野区 admin_level=7 boundary=administrative 1758878: name_en=Bunkyō name=文京区 admin_level=7 boundary=administrative 1758891: name_en=Sumida name=墨田区 admin_level=7 boundary=administrative 3554015: name_en=Koto name=江東区 admin_level=7 boundary=administrative 1761743: name_en=Edogawa name=江戸川区 admin_level=7 boundary=administrative 1761718: name_en=Katsushika name=葛飾区 admin_level=7 boundary=administrative 1760124: name_en=Adachi name=足立区 admin_level=7 boundary=administrative 1760038: name_en=Kita name=北区 admin_level=7 boundary=administrative 1760078: name_en=Itabashi name=板橋区 admin_level=7 boundary=administrative 1758858: name_en=Shinjuku name=新宿区 admin_level=7 boundary=administrative 1759477: name_en=Shibuya name=渋谷区 admin_level=7 boundary=administrative 1759506: name_en=Toshima name=豊島区 admin_level=7 boundary=administrative 1758936: name_en=Meguro name=目黒区 admin_level=7 boundary=administrative 1758897: name_en=Chuo name=中央区 admin_level=7 boundary=administrative 1760040: name_en=Arakawa name=荒川区 admin_level=7 boundary=administrative 1758888: name_en=Taito name=台東区 admin_level=7 boundary=administrative 1758947: name_en=Ota name=大田区 admin_level=7 boundary=administrative 1761742: name_en=Chiyoda name=千代田区 admin_level=7 boundary=administrative 3807613: name_en=Inagi name=稲城市 admin_level=7 boundary=administrative 5299312: name_en=Nishitama District name=西多摩郡 admin_level=6 boundary=administrative 1761834: name_en=Ōme name=青梅市 admin_level=7 boundary=administrative 5301559: name_en=Hamura name=羽村市 admin_level=7 boundary=administrative 5301643: name_en=Fussa name=福生市 admin_level=7 boundary=administrative 5301640: name_en=Akishima name=昭島市 admin_level=7 boundary=administrative 5301641: name_en=Musashimurayama name=武蔵村山市 admin_level=7 boundary=administrative 1761771: name_en=Tachikawa name=立川市 admin_level=7 boundary=administrative 5301639: name_en=Hachioji name=八王子市 admin_level=7 boundary=administrative 5301790: name_en=Hino name=日野市 admin_level=7 boundary=administrative 5301789: name_en=Tama name=多摩市 admin_level=7 boundary=administrative 5302158: name_en=Kunitachi name=国立市 admin_level=7 boundary=administrative 5302157: name_en=Kokubunji name=国分寺市 admin_level=7 boundary=administrative 5302160: name_en=Higashiyamato name=東大和市 admin_level=7 boundary=administrative 5302161: name_en=Higashimurayama name=東村山市 admin_level=7 boundary=administrative 5302159: name_en=Kodaira name=小平市 admin_level=7 boundary=administrative 5302329: name_en=Kiyose name=清瀬市 admin_level=7 boundary=administrative 5302328: name_en=Higashikurume name=東久留米市 admin_level=7 boundary=administrative 5302330: name_en=Nishitokyo name=西東京市 admin_level=7 boundary=administrative 5302326: name_en=Koganei name=小金井市 admin_level=7 boundary=administrative 5302327: name_en=Fuchu name=府中市 admin_level=7 boundary=administrative 5303898: name_en=Chofu name=調布市 admin_level=7 boundary=administrative 1761820: name_en=Mitaka name=三鷹市 admin_level=7 boundary=administrative 1761804: name_en=Musashino name=武蔵野市 admin_level=7 boundary=administrative 5301344: name_en=Akiruno name=あきる野市 admin_level=7 boundary=administrative --- Filtered to special wards --- 5100276: Machida 町田市 1759475: Komae 狛江市 1761717: Minato 港区 3554304: Shinagawa 品川区 1760119: Nerima 練馬区 1543055: Suginami 杉並区 1759474: Setagaya 世田谷区 1543056: Nakano 中野区 1758878: Bunkyō 文京区 1758891: Sumida 墨田区 3554015: Koto 江東区 1761743: Edogawa 江戸川区 1761718: Katsushika 葛飾区 1760124: Adachi 足立区 1760038: Kita 北区 1760078: Itabashi 板橋区 1758858: Shinjuku 新宿区 1759477: Shibuya 渋谷区 1759506: Toshima 豊島区 1758936: Meguro 目黒区 1758897: Chuo 中央区 1760040: Arakawa 荒川区 1758888: Taito 台東区 1758947: Ota 大田区 1761742: Chiyoda 千代田区 3807613: Inagi 稲城市 1761834: Ōme 青梅市 5301559: Hamura 羽村市 5301643: Fussa 福生市 5301640: Akishima 昭島市 5301641: Musashimurayama 武蔵村山市 1761771: Tachikawa 立川市 5301639: Hachioji 八王子市 5301790: Hino 日野市 5301789: Tama 多摩市 5302158: Kunitachi 国立市 5302157: Kokubunji 国分寺市 5302160: Higashiyamato 東大和市 5302161: Higashimurayama 東村山市 5302159: Kodaira 小平市 5302329: Kiyose 清瀬市 5302328: Higashikurume 東久留米市 5302330: Nishitokyo 西東京市 5302326: Koganei 小金井市 5302327: Fuchu 府中市 5303898: Chofu 調布市 1761820: Mitaka 三鷹市 1761804: Musashino 武蔵野市 5301344: Akiruno あきる野市
22:13
The subarea relations include both the 23 special wards (which use the suffix "区" or "ku") and many cities (市 or "shi"). The special wards have names ending in 区. Let me identify the 23:
1. Chiyoda (千代田区) - 1761742
2. Chuo (中央区) - 1758897
3. Minato (港区) - 1761717
4. Shinjuku (新宿区) - 1758858
5. Bunkyo (文京区) - 1758878
6. Taito (台東区) - 1758888
7. Sumida (墨田区) - 1758891
8. Koto (江東区) - 3554015
9. Shinagawa (品川区) - 3554304
10. Meguro (目黒区) - 1758936
11. Ota (大田区) - 1758947
12. Setagaya (世田谷区) - 1759474
13. Shibuya (渋谷区) - 1759477
14. Nakano (中野区) - 1543056
15. Suginami (杉並区) - 1543055
16. Toshima (豊島区) - 1759506
17. Kita (北区) - 1760038
18. Arakawa (荒川区) - 1760040
19. Itabashi (板橋区) - 1760078
20. Nerima (練馬区) - 1760119
21. Adachi (足立区) - 1760124
22. Katsushika (葛飾区) - 1761718
23. Edogawa (江戸川区) - 1761743
I can filter by the name ending with "区" (ku) which is the character for ward.
But actually the simplest approach is: since I need the geometry from each ward relation, I should use the OSM API to get the full geometry. However, the OSM API returns ways as references only, so I'd need to also get the way geometries.
Let me use a different approach - use pyosmium or just query Overpass directly.
Actually, let me try a different Overpass endpoint or fix the 406 error. Let me check if there's something wrong with my requests.
Wait - the issue might be that the server is returning 406 because it doesn't like the Accept headers from requests library. Let me try with a raw HTTP connection or using a different approach.
Actually, let me just download the OSM data for Tokyo using the OSM API directly and then process it using osm2geojson or similar. But that would be very slow for all the ways.
Let me try to use the Overpass turbo URL instead - sometimes that works differently.
Actually, let me look at this from a different angle. I know the 23 ward relation IDs. I can get their full geometries using the OSM API's relation/{id}/full endpoint, which returns all the child ways with their node geometries.
Let me try that approach - get each ward's full geometry from the API.
22:13
Bash
cd /work && python -c "
cd /work && python -c "
# Let's use OSM API to get full geometries of the 23 special wards
import requests
import xml.etree.ElementTree as ET
from shapely.geometry import Polygon, MultiPolygon, shape
from shapely import wkt
import json
# The 23 special wards (those with name ending in 区/ku)
# Let me identify them by checking if the name contains 区
ward_relations = [
1761742, # Chiyoda
1758897, # Chuo
1761717, # Minato
1758858, # Shinjuku
1758878, # Bunkyo
1758888, # Taito
1758891, # Sumida
3554015, # Koto
3554304, # Shinagawa
1758936, # Meguro
1758947, # Ota
1759474, # Setagaya
1759477, # Shibuya
1543056, # Nakano
1543055, # Suginami
1759506, # Toshima
1760038, # Kita
1760040, # Arakawa
1760078, # Itabashi
1760119, # Nerima
1760124, # Adachi
1761718, # Katsushika
1761743, # Edogawa
]
# Get the full data for each ward (this returns all ways with node coordinates)
for rid in ward_relations[:1]: # Just test one
r = requests.get(f'https://api.openstreetmap.org/api/0.6/relation/{rid}/full', timeout=60)
print(f'Relation {rid}: {r.status_code} {len(r.content)} bytes')
root = ET.fromstring(r.content)
# Collect nodes
nodes = {}
for node in root.findall('.//node'):
nodes[node.get('id')] = (float(node.get('lon')), float(node.get('lat')))
# Collect ways
ways = {}
for way in root.findall('.//way'):
way_id = way.get('id')
tags = {t.get('k'): t.get('v') for t in way.findall('tag')}
nds = [nd.get('ref') for nd in way.findall('nd')]
coords = [nodes[n] for n in nds]
ways[way_id] = {'tags': tags, 'coords': coords}
# Get relation members
rel = root.find('.//relation')
tags = {t.get('k'): t.get('v') for t in rel.findall('tag')}
print(f' Name: {tags.get(\"name:en\",\"\")} / {tags.get(\"name\",\"\")}')
print(f' Ways found: {len(ways)}')
print(f' Nodes found: {len(nodes)}')
# Build polygon from outer ways
outer_coords = []
for m in rel.findall('member'):
if m.get('type') == 'way' and m.get('role') == 'outer':
wid = m.get('ref')
if wid in ways:
outer_coords.append(ways[wid]['coords'])
print(f' Outer rings: {len(outer_coords)}')
"
Relation 1761742: 200 226524 bytes Name: Chiyoda / 千代田区 Ways found: 91 Nodes found: 548 Outer rings: 91
22:28
The approach with OSM API works well for getting geometry data. But getting all 23 wards individually via the API will be slow. Let me think about a more efficient approach. Actually, let me use the Overpass API differently. The issue I was having was with requests to overpass-api.de getting 406 errors. Let me try using osmnx but configure it to use a different endpoint. Actually wait - osmnx by default uses overpass-api.de. Let me check if the issue is DNS or IP based. The ConnectionRefused suggests the IP might be blocked. Let me try a different approach - use the OSM API directly to get relation data and parse it to build geometries. Let me write a comprehensive script that: 1. Downloads the 23 ward boundaries via OSM API 2. Gets the road network via Overpass (or OSM API) 3. Does the crossing analysis Actually, for getting road data, I could also use the OSM API to get all highways within an area. But that's essentially what Overpass is for. Let me try using Overpass with the kumi.systems endpoint via raw requests.
22:28
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Let's try overpass-api.de with raw HTTP and no fancy headers
# Maybe the issue is the Accept header. Let's try not sending Accept at all.
query = '[out:json][timeout:30];node(1);out;'
r = requests.post('https://overpass-api.de/api/interpreter',
data=query,
headers={
'Content-Type': 'application/x-www-form-urlencoded',
'Accept': 'application/json',
},
timeout=30)
print(f'Status: {r.status_code}')
print(f'Content-Type: {r.headers.get(\"Content-Type\")}')
print(r.text[:300])
"
Status: 406 Content-Type: text/html; charset=iso-8859-1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate representation of the requested resource could not be found on this server.</p> <hr> <address>Apache/2.4.6
22:36
The Overpass API at overpass-api.de is returning 406 errors for all requests. This is likely an IP-based block or the server configuration doesn't accept our requests. Let me try a different approach - I can use the OSM API directly to get the data. The OSM API allows getting relation data and their full geometries. Let me use that approach. Let me write a comprehensive script that: 1. Gets the 23 special ward boundaries from OSM API 2. Gets the road network using Overpass (or alternative) 3. Does the spatial analysis Since Overpass is not working, let me try using the OSM API's map endpoint which returns all data in a bounding box. But the area is large. Let me first get the ward boundaries via OSM API, then figure out the roads. Actually, let me try the overpass-api with a different approach - using the raw query without URL encoding issues.
22:36
Bash
cd /work && python -c "
cd /work && python -c "
import subprocess
# Try using curl through python subprocess
import urllib.request
import urllib.parse
# Try with wget style - using --header to set Accept
query = '[out:json][timeout:30];node(1);out;'
data = 'data=' + urllib.parse.quote(query, safe='')
req = urllib.request.Request('https://overpass-api.de/api/interpreter', data.encode())
req.add_header('Content-Type', 'application/x-www-form-urlencoded')
req.add_header('Accept', 'application/json')
try:
resp = urllib.request.urlopen(req, timeout=30)
print(f'Status: {resp.status}')
print(resp.read()[:200])
except urllib.error.HTTPError as e:
print(f'HTTP Error: {e.code}')
print(e.read()[:300])
except urllib.error.URLError as e:
print(f'URL Error: {e.reason}')
"
HTTP Error: 406 b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html><head>\n<title>406 Not Acceptable</title>\n</head><body>\n<h1>Not Acceptable</h1>\n<p>An appropriate representation of the requested resource could not be found on this server.</p>\n<hr>\n<address>Apache/2.4.6'
22:47
The Overpass API is refusing connections with 406. Let me try a different approach - maybe I should use the OSM API directly to get the data. Let me try using the `osmnx` library with a different approach, or use the OSM API to get the geometries directly. Actually, the 406 error is strange. It could be that the Overpass server is blocking requests from this IP. Let me check if there's a way to configure the User-Agent or use a different approach. Let me try osmnx with kumi.systems properly.
22:47
Bash
cd /work && python -c "
cd /work && python -c "
import osmnx as ox
ox.settings.overpass_url = 'https://overpass.kumi.systems/api'
ox.settings.requests_timeout = 300
ox.settings.max_query_area_size = 2500000000
ox.settings.http_user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
# Get the 23 special wards by querying OSMnx for place
# Use Nominatim to find them
print('Testing Nominatim...')
gdf = ox.geocode_to_gdf('Tokyo, Japan', which_result=1)
print('Tokyo:', gdf.shape)
# Now get admin_level=7 boundaries within Tokyo
tokyo_poly = gdf.iloc[0].geometry
print(f'Tokyo bounds: {tokyo_poly.bounds}')
# Try a small test query first
try:
test = ox.features_from_place('Chiyoda, Tokyo, Japan', tags={'boundary':'administrative'})
print('Test query shape:', test.shape)
except Exception as e:
print(f'Test failed: {type(e).__name__}: {e}')
"
<exit_code>-1</exit_code> <error>Command timed out after 120000ms</error>
25:04
The overpass API is not working from this environment. Let me try a different approach - use the OSM API directly to download the relation data and build geometries. The OSM API (`api.openstreetmap.org`) works fine (we got 200 responses earlier), so I'll use that to download the full relation data for each ward. Let me build a comprehensive script that: 1. Downloads relation data from OSM API for each of the 23 special wards 2. Constructs polygon geometries from the ways and nodes 3. Downloads road network data from OSM API (or maybe we can use a smaller approach) 4. Computes crossings and buffers Actually, since the Overpass API is not accessible, let me use the OSM API (which works) to download the ward boundaries and road network. However, downloading road data via OSM API would be very tedious. Let me try a different approach - I can use osmnx with a smaller area, like querying one ward at a time with a bounding box. Or I could try downloading the road network from the OSM API for smaller areas. Actually, wait - the OSM API at api.openstreetmap.org works fine. Let me use Overpass Turbo's raw API endpoint. But overpass-api.de is returning 406. Let me check if there's something about the IP being blocked. Let me try another approach: use `osmnx` with `features_from_place` but for individual wards, and also use `ox.graph_from_place` for the road network.
25:04
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try overpass-api.de with explicit Accept header for XML
query = '[out:xml][timeout:30];node(1);out;'
r = requests.post('https://overpass-api.de/api/interpreter', data={'data': query}, timeout=30)
print(r.status_code, r.reason)
print(r.text[:200])
# Try with Accept: application/xml
headers = {'Accept': 'application/xml'}
r2 = requests.post('https://overpass-api.de/api/interpreter', data={'data': query}, headers=headers, timeout=30)
print(r2.status_code, r2.reason)
print(r2.text[:200])
"
406 Not Acceptable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate repres 406 Not Acceptable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>406 Not Acceptable</title> </head><body> <h1>Not Acceptable</h1> <p>An appropriate repres
25:14
The overpass-api.de is returning 406 for all requests. This might be because the server is overloaded or blocking our IP. Let me try a completely different approach - use the OSM API directly to get the ward boundaries and road data. The OSM API doesn't support complex queries like Overpass, but we can get the full relation data (all ways and nodes) for each ward. For roads, we might need to use a different approach. Let me think about this differently. We can: 1. Get the ward boundaries from the OSM API (each relation's full data includes all the ways and nodes) 2. For roads, we could use OSMnx to download the road network for Tokyo But the Overpass API seems to be unavailable from this environment. Let me try the kumi.systems one with a longer timeout and simpler query. Actually, looking at the error more carefully: kumi.systems returned a 504 (gateway timeout) for a simple query, and overpass-api.de returns 406 for everything. This might be a CDN/cloudflare issue with the environment. Let me try another approach - download the data using the OSM API for the 23 wards, and for roads, use a pre-downloaded approach. Actually, I think the best approach now is to use the OSM API to: 1. Download each ward boundary (relation + full) and parse the geometries into polygons 2. Download the road network within Tokyo's bounding box using the OSM API with a bbox query Let me try that approach.
25:14
Bash
cd /work && python -c "
cd /work && python -c "
# Let's try a different overpass instance
import requests
# Try using overpass-api.de but with a different path
urls = [
'https://lz4.overpass-api.de/api/interpreter',
'https://z.overpass-api.de/api/interpreter',
]
query = '[out:json][timeout:10];node(1);out;'
for url in urls:
try:
r = requests.post(url, data={'data': query}, timeout=15)
print(f'{url}: {r.status_code} {len(r.text)}')
if r.status_code == 200:
print(r.text[:200])
except Exception as e:
print(f'{url}: {type(e).__name__}: {e}')
"
https://lz4.overpass-api.de/api/interpreter: 406 375 https://z.overpass-api.de/api/interpreter: 406 373
25:27
The Overpass API is blocking our requests with 406. Let me try a different approach - maybe we can use the OSM API directly to download the ward boundaries as relations with their full geometry, then use those to build our data. Actually, let me try using osmnx which uses its own overpass query formatting.
25:27
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Let's see what happens when we send without any content-type header
query = '[out:json][timeout:30];node(1);out;'
r = requests.post('https://overpass-api.de/api/interpreter',
data=query,
timeout=30)
print(f'Request headers: {dict(r.request.headers)}')
print(f'Response headers: {dict(r.headers)}')
print(f'Status: {r.status_code}')
"
Request headers: {'User-Agent': 'python-requests/2.34.2', 'Accept-Encoding': 'gzip, deflate, zstd', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '35'}
Response headers: {'Date': 'Thu, 18 Jun 2026 03:42:56 GMT', 'Server': 'Apache/2.4.67 (Debian)', 'Content-Length': '371', 'Connection': 'close', 'Content-Type': 'text/html; charset=iso-8859-1'}
Status: 406
25:47
It seems like overpass-api.de is returning 406 consistently regardless of what I send. Let me try an alternative approach - use the OSM API directly to get all the data needed, then process it locally.
25:47
Bash
cd /work && python -c "
cd /work && python -c "
# Let me use the OSM API (which works) to download all the data
# I'll download each ward relation geometry via the /full endpoint
# and convert to GeoJSON/shapely
import requests
import xml.etree.ElementTree as ET
from shapely.geometry import Polygon, MultiPolygon, LineString, Point
from shapely.ops import polygonize, unary_union
import json
# The 23 special wards of Tokyo (ku-area)
ward_relations = {
1761742: ('Chiyoda', '千代田区'),
1758897: ('Chuo', '中央区'),
1761717: ('Minato', '港区'),
1758858: ('Shinjuku', '新宿区'),
1758878: ('Bunkyo', '文京区'),
1758888: ('Taito', '台東区'),
1758891: ('Sumida', '墨田区'),
3554015: ('Koto', '江東区'),
3554304: ('Shinagawa', '品川区'),
1758936: ('Meguro', '目黒区'),
1758947: ('Ota', '大田区'),
1759474: ('Setagaya', '世田谷区'),
1759477: ('Shibuya', '渋谷区'),
1543056: ('Nakano', '中野区'),
1543055: ('Suginami', '杉並区'),
1759506: ('Toshima', '豊島区'),
1760038: ('Kita', '北区'),
1760040: ('Arakawa', '荒川区'),
1760078: ('Itabashi', '板橋区'),
1760119: ('Nerima', '練馬区'),
1760124: ('Adachi', '足立区'),
1761718: ('Katsushika', '葛飾区'),
1761743: ('Edogawa', '江戸川区'),
}
def get_relation_geometry(rel_id):
'''Get the geometry of an OSM relation from the /full API'''
r = requests.get(f'https://api.openstreetmap.org/api/0.6/relation/{rel_id}/full', timeout=60)
root = ET.fromstring(r.content)
nodes = {}
for node in root.findall('.//node'):
nodes[node.get('id')] = (float(node.get('lon')), float(node.get('lat')))
ways = {}
for way in root.findall('.//way'):
nds = [nodes[nd.get('ref')] for nd in way.findall('nd') if nd.get('ref') in nodes]
ways[way.get('id')] = nds
rel = root.find('.//relation')
tags = {t.get('k'): t.get('v') for t in rel.findall('tag')}
outer_lines = []
inner_lines = []
for m in rel.findall('member'):
if m.get('type') == 'way' and m.get('ref') in ways:
coords = ways[m.get('ref')]
if len(coords) >= 2:
line = LineString(coords)
if m.get('role') == 'inner':
inner_lines.append(line)
else: # outer or missing
outer_lines.append(line)
# Merge outer rings and inner rings
if outer_lines:
merged_outer = unary_union(outer_lines)
if merged_outer.geom_type == 'MultiLineString':
# Try to polygonize
rings = list(polygonize(merged_outer.geoms))
else:
rings = list(polygonize([merged_outer]))
if rings:
if len(rings) == 1:
outer_poly = rings[0]
else:
outer_poly = MultiPolygon(rings)
else:
outer_poly = None
else:
outer_poly = None
return outer_poly, tags
# Test with one ward
poly, tags = get_relation_geometry(1761742)
print(f'Polygon type: {poly.geom_type if poly else None}')
print(f'Area (deg²): {poly.area if poly else 0}')
print(f'Tags: {tags.get(\"name:en\")} / {tags.get(\"name\")}')
"
Polygon type: Polygon Area (deg²): 0.0011314825301850134 Tags: Chiyoda / 千代田区
26:05
Write
/work/download_all_wards.py
content · 122 lines · py
"""Download all 23 special ward geometries from OSM API."""
import requests
import xml.etree.ElementTree as ET
from shapely.geometry import Polygon, MultiPolygon, LineString
from shapely.ops import polygonize, unary_union
import json
import pickle
# The 23 special wards of Tokyo (ku-area)
ward_relations = {
1761742: ('Chiyoda', '千代田区'),
1758897: ('Chuo', '中央区'),
1761717: ('Minato', '港区'),
1758858: ('Shinjuku', '新宿区'),
1758878: ('Bunkyo', '文京区'),
1758888: ('Taito', '台東区'),
1758891: ('Sumida', '墨田区'),
3554015: ('Koto', '江東区'),
3554304: ('Shinagawa', '品川区'),
1758936: ('Meguro', '目黒区'),
1758947: ('Ota', '大田区'),
1759474: ('Setagaya', '世田谷区'),
1759477: ('Shibuya', '渋谷区'),
1543056: ('Nakano', '中野区'),
1543055: ('Suginami', '杉並区'),
1759506: ('Toshima', '豊島区'),
1760038: ('Kita', '北区'),
1760040: ('Arakawa', '荒川区'),
1760078: ('Itabashi', '板橋区'),
1760119: ('Nerima', '練馬区'),
1760124: ('Adachi', '足立区'),
1761718: ('Katsushika', '葛飾区'),
1761743: ('Edogawa', '江戸川区'),
}
def get_relation_geometry(rel_id):
"""Get the geometry of an OSM relation from the /full API."""
r = requests.get(f'https://api.openstreetmap.org/api/0.6/relation/{rel_id}/full', timeout=120)
root = ET.fromstring(r.content)
nodes = {}
for node in root.findall('.//node'):
nodes[node.get('id')] = (float(node.get('lon')), float(node.get('lat')))
ways = {}
for way in root.findall('.//way'):
nds = [nodes[nd.get('ref')] for nd in way.findall('nd') if nd.get('ref') in nodes]
ways[way.get('id')] = nds
rel = root.find('.//relation')
tags = {t.get('k'): t.get('v') for t in rel.findall('tag')}
outer_lines = []
inner_lines = []
for m in rel.findall('member'):
if m.get('type') == 'way' and m.get('ref') in ways:
coords = ways[m.get('ref')]
if len(coords) >= 2:
line = LineString(coords)
if m.get('role') == 'inner':
inner_lines.append(line)
else: # outer or missing
outer_lines.append(line)
# Merge outer rings and polygonize
if outer_lines:
merged_outer = unary_union(outer_lines)
if merged_outer.geom_type == 'MultiLineString':
rings = list(polygonize(merged_outer.geoms))
elif merged_outer.geom_type == 'LineString':
rings = list(polygonize([merged_outer]))
else:
rings = []
if rings:
outer_poly = MultiPolygon(rings) if len(rings) > 1 else rings[0]
else:
outer_poly = None
else:
outer_poly = None
# Handle inner rings if any
if inner_lines and outer_poly is not None:
merged_inner = unary_union(inner_lines)
if merged_inner.geom_type == 'MultiLineString':
inner_rings = list(polygonize(merged_inner.geoms))
elif merged_inner.geom_type == 'LineString':
inner_rings = list(polygonize([merged_inner]))
else:
inner_rings = []
if inner_rings:
# Subtract inner from outer
inner_union = unary_union(inner_rings)
outer_poly = outer_poly.difference(inner_union)
return outer_poly, tags
results = {}
for rid, (name_en, name) in ward_relations.items():
print(f'Downloading {name_en} ({rid})...')
try:
poly, tags = get_relation_geometry(rid)
if poly is not None:
results[rid] = {
'geometry': poly,
'tags': tags,
'name_en': name_en,
'name': name
}
print(f' Got polygon: type={poly.geom_type}, area={poly.area:.6f} deg²')
else:
print(f' FAILED: No polygon')
except Exception as e:
print(f' ERROR: {e}')
# Save results
with open('/work/wards_data.pkl', 'wb') as f:
pickle.dump(results, f)
print(f'\nSuccessfully downloaded {len(results)} ward geometries')
print('Saved to /work/wards_data.pkl')Wrote /work/download_all_wards.py (4104 chars).
26:11
Let me run this download script. It might take a while since it's downloading 23 ward geometries from the OSM API.
26:11
Bash
cd /work && python download_all_wards.py
Downloading Chiyoda (1761742)... Got polygon: type=Polygon, area=0.001131 deg² Downloading Chuo (1758897)... Got polygon: type=Polygon, area=0.001122 deg² Downloading Minato (1761717)... Got polygon: type=Polygon, area=0.002378 deg² Downloading Shinjuku (1758858)... Got polygon: type=Polygon, area=0.001820 deg² Downloading Bunkyo (1758878)... Got polygon: type=Polygon, area=0.001132 deg² Downloading Taito (1758888)... Got polygon: type=Polygon, area=0.001006 deg² Downloading Sumida (1758891)... Got polygon: type=Polygon, area=0.001372 deg² Downloading Koto (3554015)... Got polygon: type=Polygon, area=0.006512 deg² Downloading Shinagawa (3554304)... Got polygon: type=Polygon, area=0.002593 deg² Downloading Meguro (1758936)... Got polygon: type=Polygon, area=0.001478 deg² Downloading Ota (1758947)... Got polygon: type=Polygon, area=0.010082 deg² Downloading Setagaya (1759474)... Got polygon: type=Polygon, area=0.005782 deg² Downloading Shibuya (1759477)... Got polygon: type=Polygon, area=0.001505 deg² Downloading Nakano (1543056)... Got polygon: type=Polygon, area=0.001552 deg² Downloading Suginami (1543055)... Got polygon: type=Polygon, area=0.003379 deg² Downloading Toshima (1759506)... Got polygon: type=Polygon, area=0.001294 deg² Downloading Kita (1760038)... Got polygon: type=Polygon, area=0.002044 deg² Downloading Arakawa (1760040)... Got polygon: type=Polygon, area=0.001017 deg² Downloading Itabashi (1760078)... Got polygon: type=Polygon, area=0.003201 deg² Downloading Nerima (1760119)... Got polygon: type=MultiPolygon, area=0.004796 deg² Downloading Adachi (1760124)... Got polygon: type=Polygon, area=0.005312 deg² Downloading Katsushika (1761718)... Got polygon: type=Polygon, area=0.003470 deg² Downloading Edogawa (1761743)... Got polygon: type=Polygon, area=0.008105 deg² Successfully downloaded 23 ward geometries Saved to /work/wards_data.pkl
26:30
Now I have the ward geometries. Next I need to download the road network from OSM. Since the Overpass API is blocking my requests, I'll need to use the OSM API or find another way. Let me try to download roads within the bounding box of Tokyo's 23 wards using the OSM API's map call, or I can try Overpass again with a different approach. Actually, let me try the overpass turbo website approach. Maybe the issue is specific to this server's IP being blocked. Let me try a different overpass endpoint or try using the overpass-turbo.eu redirect. Wait, actually let me try a different approach. I can use the OSM API's `/api/0.6/map` endpoint to get all data within a bounding box. But that might be too large for Tokyo. Let me try with individual wards. Actually, let me first try to find out why overpass is returning 406. Maybe it's a rate limiting issue.
26:30
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try a raw HTTP request without the requests library to debug
import http.client
import json
conn = http.client.HTTPSConnection('overpass-api.de', timeout=10)
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Accept': 'application/json, text/plain, */*',
}
body = 'data=%5Bout%3Ajson%5D%3Bnode%281%29%3Bout%3B'
conn.request('POST', '/api/interpreter', body, headers)
resp = conn.getresponse()
print(f'Status: {resp.status} {resp.reason}')
print(f'Headers: {dict(resp.getheaders())}')
data = resp.read()
print(data[:300])
conn.close()
"
Status: 406 Not Acceptable
Headers: {'Date': 'Thu, 18 Jun 2026 03:43:59 GMT', 'Server': 'Apache/2.4.67 (Debian)', 'Content-Length': '371', 'Connection': 'close', 'Content-Type': 'text/html; charset=iso-8859-1'}
b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html><head>\n<title>406 Not Acceptable</title>\n</head><body>\n<h1>Not Acceptable</h1>\n<p>An appropriate representation of the requested resource could not be found on this server.</p>\n<hr>\n<address>Apache/2.4.6'
26:39
The overpass-api.de is returning 406 for all queries. Let me try a different approach - maybe the server has some specific requirement. Let me try with the correct form-encoded body format.
26:39
Bash
cd /work && python -c "
cd /work && python -c "
import http.client
# The correct way - use POST with form data
conn = http.client.HTTPSConnection('overpass-api.de', timeout=10)
body = 'data=[out:json][timeout:30];node(1);out;'
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
}
conn.request('POST', '/api/interpreter', body.encode(), headers)
resp = conn.getresponse()
print(f'Status: {resp.status} {resp.reason}')
data = resp.read()
print(data[:500])
conn.close()
"
Status: 406 Not Acceptable b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html><head>\n<title>406 Not Acceptable</title>\n</head><body>\n<h1>Not Acceptable</h1>\n<p>An appropriate representation of the requested resource could not be found on this server.</p>\n<hr>\n<address>Apache/2.4.67 (Debian) Server at overpass-api.de Port 443</address>\n</body></html>\n'
26:48
The Overpass API at overpass-api.de is returning 406 for all requests, even the simplest ones. This might be an IP-blocking or configuration issue with this particular instance. Let me use the OSM API directly to also download roads. Actually, wait - the OSM API works fine. Let me try downloading the roads from OSM API as well - but the OSM API doesn't support spatial queries, so I'd need to download roads by bounding box. Actually, I know another approach - let me use OSMnx but configure it to use a working endpoint. Let me check if kumi.systems works with the right timeout. Actually, wait - let me try a completely different approach. The OSM API can give us ways within a bounding box. Let me download all highways within the Tokyo wards area by getting the ways in the bounding box of Tokyo. Actually, there's a simpler approach - I can download the road network directly from OSM using the Overpass API at a different endpoint. Let me try configuring osmnx differently. Actually, let me try a more creative approach. Since the OSM API works, I can use it to get the road network. The OSM API has a "map" endpoint that returns all nodes and ways within a bounding box. But Tokyo is quite large. Let me think of a better approach. Actually, I realize the issue with overpass-api.de might be temporary or IP-based. Let me try one more thing - the API at overpass.kumi.systems gave a 504 timeout, which means it does work, just the query was too slow. Let me try smaller queries. Let me try a small bounding box query to test.
26:48
Bash
cd /work && python -c "
cd /work && python -c "
import requests
# Try overpass.kumi.systems with a small test query
query = '[out:json][timeout:30];node(1);out;'
try:
r = requests.post('https://overpass.kumi.systems/api/interpreter',
data={'data': query}, timeout=30)
print(f'Status: {r.status_code}')
print(r.text[:500])
except Exception as e:
print(f'Error: {type(e).__name__}: {e}')
"
Status: 200
{
"version": 0.6,
"generator": "Overpass API 0.7.62.11 87bfad18",
"osm3s": {
"timestamp_osm_base": "2026-04-21T17:01:53Z",
"copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
},
"elements": [
{
"type": "node",
"id": 1,
"lat": 42.7957187,
"lon": 13.5690032,
"tags": {
"communication:microwave": "yes",
"communication:radio": "fm",
"description": "Radio Subasio",
"frequency": "105.5 MHz",
26:59
Great! overpass.kumi.systems works. Now I can use it to get the roads data. Let me proceed with the full solution. My plan: 1. Use OSM API (which works) to download ward boundaries - already done 2. Use overpass.kumi.systems to download road network 3. Process everything in a metric CRS (JGD2011 / Japan Plane Rectangular CS - zone 9 for Tokyo, EPSG:6673? Actually let me check) 4. Find road/ward boundary crossings 5. Create buffers 6. Calculate densities 7. Save to GeoPackage For Japan, the appropriate metric coordinate system is JGD2011 / UTM zone 54N (EPSG:6678) or JGD2011 / Japan Plane Rectangular CS. Tokyo is in zone 9 of the Japan Plane Rectangular CS system (EPSG:6673 for JGD2011 zone 9? Actually let me check). JGD2011 / Japan Plane Rectangular Zone IX (EPSG:6679 for zone 9? Let me check). Actually, the JGD2011 / UTM zone 54N (EPSG:6678) covers Tokyo. Or JGD2011 / Japan Plane Rectangular CS zone IX (EPSG:6679). Let me just use EPSG:6678 (JGD2011 / UTM zone 54N) which covers Tokyo. Actually, let's use the standard: EPSG:6679 - JGD2011 / Japan Plane Rectangular CS zone IX (covers Tokyo). Wait, let me verify. JGD2011 zones: - Zone 9: Tokyo (EPSG:6679) Or I could use EPSG:2450 (JGD2000 / Japan Plane Rectangular CS zone IX) or EPSG:6679 (JGD2011 / Japan Plane Rectangular CS zone IX). Let me just check which CRS is available.
26:59
26:59
Bash
cd /work && python -c "
cd /work && python -c "
import pyproj
# Find the right CRS for Tokyo
# Japan Plane Rectangular CS zone 9 for Tokyo
print('Available CRS for Tokyo:')
for epsg in [6679, 6673, 6678, 2450, 2449, 6677, 32654, 32653]:
try:
crs = pyproj.CRS.from_epsg(epsg)
print(f' EPSG:{epsg} - {crs.name} - area: {crs.area_of_use}')
except:
pass
"
Available CRS for Tokyo: EPSG:6679 - JGD2011 / Japan Plane Rectangular CS XI - area: - name: Japan - onshore - Hokkaido west of approximately 141°E - Otaru city; Hakodate city; Date city; Usu-gun and Abuta-gun of Iburi-shicho; Hiyama-shicho; Shiribeshi-shicho; Oshima-shicho. - bounds: (139.34, 41.34, 141.46, 43.42) EPSG:6673 - JGD2011 / Japan Plane Rectangular CS V - area: - name: Japan - onshore - Honshu between approximately 133°15'E and 135°10'E - Hyogo-ken; Tottori-ken; Okayama-ken. - bounds: (133.13, 34.13, 135.47, 35.71) EPSG:6678 - JGD2011 / Japan Plane Rectangular CS X - area: - name: Japan - onshore - Honshu north of 38°N approximately - Aomori-ken; Akita-ken; Yamagata-ken; Iwate-ken; Miyagi-ken. - bounds: (139.49, 37.73, 142.14, 41.58) EPSG:2450 - JGD2000 / Japan Plane Rectangular CS VIII - area: - name: Japan - onshore - Honshu between approximately 137°45'E and 139°E - Niigata-ken; Nagano-ken; Yamanashi-ken; Shizuoka-ken. - bounds: (137.32, 34.54, 139.91, 38.58) EPSG:2449 - JGD2000 / Japan Plane Rectangular CS VII - area: - name: Japan - onshore - Honshu between approximately 136°15'E and 137°45'E - Ishikawa-ken; Toyama-ken; Gifu-ken; Aichi-ken. - bounds: (136.22, 34.51, 137.84, 37.58) EPSG:6677 - JGD2011 / Japan Plane Rectangular CS IX - area: - name: Japan - onshore - Honshu - Tokyo-to. (Excludes offshore island areas of Tokyo-to covered by Japan Plane Rectangular Coordinate System zones XIV, XVIII and XIX). - bounds: (138.4, 29.31, 141.11, 37.98) EPSG:32654 - WGS 84 / UTM zone 54N - area: - name: Between 138°E and 144°E, northern hemisphere between equator and 84°N, onshore and offshore. Japan. Russian Federation. - bounds: (138.0, 0.0, 144.0, 84.0) EPSG:32653 - WGS 84 / UTM zone 53N - area: - name: Between 132°E and 138°E, northern hemisphere between equator and 84°N, onshore and offshore. China. Japan. Russian Federation. - bounds: (132.0, 0.0, 138.0, 84.0)
27:36
EPSG:6677 - JGD2011 / Japan Plane Rectangular CS IX is for Tokyo-to. That's the one I want. Let me now plan the full solution: 1. Use overpass.kumi.systems to download: - The 23 ward boundaries (already have from OSM API, but I need better quality polygons) - The road network (highway ways, excluding service roads) 2. Convert to metric CRS (EPSG:6677) 3. Find crossing points where roads intersect ward boundaries 4. Create buffers and do the analysis 5. Save to GeoPackage Let me write the full script now.
27:36
Write
/work/solve.py
content · 443 lines · py
"""Tokyo ward road crossing analysis - main script.
Downloads ward boundaries and roads from OSM via Overpass API,
computes crossings, buffers, and density stats, saves to GeoPackage.
"""
import os
import json
import pickle
import time
import requests
import numpy as np
import geopandas as gpd
import pandas as pd
from shapely.geometry import (
Polygon, MultiPolygon, Point, MultiPoint, LineString, MultiLineString, box, shape
)
from shapely.ops import polygonize, unary_union, linemerge, split
from shapely import wkt
from pyproj import CRS, Transformer
from functools import lru_cache
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
OVERPASS_URL = "https://overpass.kumi.systems/api/interpreter"
OUTPUT_GPKG = "/work/tokyo_crossings.gpkg"
METRIC_CRS = "EPSG:6677" # JGD2011 / Japan Plane Rectangular CS IX (Tokyo)
WGS84_CRS = "EPSG:4326"
WARD_RELATIONS = {
1761742: ("Chiyoda", "千代田区"),
1758897: ("Chuo", "中央区"),
1761717: ("Minato", "港区"),
1758858: ("Shinjuku", "新宿区"),
1758878: ("Bunkyo", "文京区"),
1758888: ("Taito", "台東区"),
1758891: ("Sumida", "墨田区"),
3554015: ("Koto", "江東区"),
3554304: ("Shinagawa", "品川区"),
1758936: ("Meguro", "目黒区"),
1758947: ("Ota", "大田区"),
1759474: ("Setagaya", "世田谷区"),
1759477: ("Shibuya", "渋谷区"),
1543056: ("Nakano", "中野区"),
1543055: ("Suginami", "杉並区"),
1759506: ("Toshima", "豊島区"),
1760038: ("Kita", "北区"),
1760040: ("Arakawa", "荒川区"),
1760078: ("Itabashi", "板橋区"),
1760119: ("Nerima", "練馬区"),
1760124: ("Adachi", "足立区"),
1761718: ("Katsushika", "葛飾区"),
1761743: ("Edogawa", "江戸川区"),
}
# ---------------------------------------------------------------------------
# Helper functions
# ---------------------------------------------------------------------------
def overpass_query(query, timeout=180):
"""Run an Overpass query and return parsed JSON."""
r = requests.post(OVERPASS_URL, data={"data": query}, timeout=timeout)
r.raise_for_status()
return r.json()
def osm_xml_to_polygon_via_api(rel_id):
"""Download a relation's geometry from OSM API and build a shapely Polygon."""
r = requests.get(
f"https://api.openstreetmap.org/api/0.6/relation/{rel_id}/full",
timeout=120,
)
r.raise_for_status()
import xml.etree.ElementTree as ET
root = ET.fromstring(r.content)
nodes = {}
for node in root.findall(".//node"):
nodes[node.get("id")] = (float(node.get("lon")), float(node.get("lat")))
ways = {}
for way in root.findall(".//way"):
nds = [nodes[nd.get("ref")] for nd in way.findall("nd") if nd.get("ref") in nodes]
ways[way.get("id")] = nds
rel = root.find(".//relation")
tags = {t.get("k"): t.get("v") for t in rel.findall("tag")}
outer_lines = []
inner_lines = []
for m in rel.findall("member"):
if m.get("type") == "way" and m.get("ref") in ways:
coords = ways[m.get("ref")]
if len(coords) >= 2:
line = LineString(coords)
if m.get("role") == "inner":
inner_lines.append(line)
else:
outer_lines.append(line)
# Build polygon
poly = None
if outer_lines:
merged = unary_union(outer_lines)
if merged.geom_type == "MultiLineString":
rings = list(polygonize(merged.geoms))
elif merged.geom_type == "LineString":
rings = list(polygonize([merged]))
else:
rings = []
if rings:
poly = MultiPolygon(rings) if len(rings) > 1 else rings[0]
# Cut inner holes
if inner_lines and poly is not None:
inner_merged = unary_union(inner_lines)
if inner_merged.geom_type == "MultiLineString":
inner_rings = list(polygonize(inner_merged.geoms))
elif inner_merged.geom_type == "LineString":
inner_rings = list(polygonize([inner_merged]))
else:
inner_rings = []
if inner_rings:
inner_union = unary_union(inner_rings)
poly = poly.difference(inner_union)
return poly, tags
def download_ward_boundaries():
"""Download all 23 ward boundaries via OSM API."""
results = {}
for rid, (name_en, name) in WARD_RELATIONS.items():
print(f" Downloading ward: {name_en} ({rid})...")
poly, tags = osm_xml_to_polygon_via_api(rid)
if poly is not None:
tags_dict = dict(tags)
results[rid] = {
"geometry": poly,
"ward_id": rid,
"ward_name_en": tags_dict.get("name:en", name_en),
"ward_name": tags_dict.get("name", name),
}
else:
print(f" WARNING: No polygon for {name_en}")
time.sleep(0.5) # Be nice to OSM API
return results
def download_roads_via_overpass(ward_bounds_wgs84):
"""Download drivable road network (excluding service roads) within the
combined bounds, via Overpass."""
# Compute a slightly expanded bounding box
all_bounds = ward_bounds_wgs84.total_bounds # (minx, miny, maxx, maxy)
minx, miny, maxx, maxy = all_bounds
# Expand by ~0.05 degrees (~5km) to catch roads near the boundary
pad = 0.05
bbox_str = f"{miny-pad},{minx-pad},{maxy+pad},{maxx+pad}"
# Query: all highways except service roads, within the bbox
query = f"""
[out:json][timeout:300];
(
way["highway"]["highway"!="service"]["highway"!="footway"]["highway"!="cycleway"]
["highway"!="path"]["highway"!="pedestrian"]["highway"!="track"]
["highway"!="steps"]["highway"!="bridleway"]["highway"!="corridor"]
["highway"!="escalator"]["highway"!="proposed"]["highway"!="construction"]
["highway"!="bus_guideway"]["highway"!="raceway"]
({bbox_str});
);
out body;
>;
out skel qt;
"""
print(" Querying Overpass for drivable roads (excluding service)...")
data = overpass_query(query, timeout=300)
print(f" Got {len(data.get('elements', []))} elements")
# Build node dict
nodes = {}
for elem in data.get("elements", []):
if elem["type"] == "node":
nodes[elem["id"]] = (elem["lon"], elem["lat"])
# Build line geometries from ways
lines = []
for elem in data.get("elements", []):
if elem["type"] == "way":
nds = elem.get("nodes", [])
coords = [nodes.get(n) for n in nds if n in nodes]
if len(coords) >= 2 and all(c is not None for c in coords):
tags = elem.get("tags", {})
highway = tags.get("highway", "")
name = tags.get("name", "")
lines.append({
"geometry": LineString(coords),
"osm_id": elem["id"],
"highway": highway,
"name": name,
})
print(f" Built {len(lines)} road line geometries")
return gpd.GeoDataFrame(lines, crs=WGS84_CRS)
def find_crossings(wards_metric, roads_metric):
"""Find points where roads cross ward boundaries.
Returns a GeoDataFrame of crossing points with ward info.
"""
all_crossings = []
for idx, ward in wards_metric.iterrows():
ward_geom = ward.geometry
ward_boundary = ward_geom.boundary
if ward_boundary.is_empty:
continue
# Find roads that intersect this ward's boundary
for ridx, road in roads_metric.iterrows():
road_geom = road.geometry
if not ward_boundary.intersects(road_geom):
continue
intersection = ward_boundary.intersection(road_geom)
if intersection.is_empty:
continue
# intersection could be Point, MultiPoint, or GeometryCollection
if intersection.geom_type == "Point":
points = [intersection]
elif intersection.geom_type == "MultiPoint":
points = list(intersection.geoms)
elif intersection.geom_type == "GeometryCollection":
points = [g for g in intersection.geoms if g.geom_type == "Point"]
else:
# It could be a LineString (road runs along boundary) - skip
continue
for pt in points:
all_crossings.append({
"geometry": pt,
"ward_id": ward["ward_id"],
"ward_name_en": ward["ward_name_en"],
"ward_name": ward["ward_name"],
"road_osm_id": road["osm_id"],
"highway": road["highway"],
})
if all_crossings:
gdf = gpd.GeoDataFrame(all_crossings, crs=METRIC_CRS)
return gdf
else:
return gpd.GeoDataFrame([], columns=[
"ward_id", "ward_name_en", "ward_name", "road_osm_id", "highway", "geometry"
], crs=METRIC_CRS)
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
# Step 1: Download ward boundaries (from OSM API since it's reliable)
print("=" * 60)
print("STEP 1: Download ward boundaries")
print("=" * 60)
ward_data = download_ward_boundaries()
# Build GeoDataFrame in WGS84
records = []
for rid, data in ward_data.items():
records.append({
"geometry": data["geometry"],
"ward_id": rid,
"ward_name_en": data["ward_name_en"],
"ward_name": data["ward_name"],
})
wards_wgs84 = gpd.GeoDataFrame(records, crs=WGS84_CRS)
print(f" Wards in WGS84: {len(wards_wgs84)}")
# Step 2: Download roads via Overpass
print("=" * 60)
print("STEP 2: Download drivable roads (excluding service)")
print("=" * 60)
roads_wgs84 = download_roads_via_overpass(wards_wgs84)
print(f" Roads downloaded: {len(roads_wgs84)}")
# Step 3: Reproject to metric CRS
print("=" * 60)
print(f"STEP 3: Reproject to {METRIC_CRS}")
print("=" * 60)
wards_metric = wards_wgs84.to_crs(METRIC_CRS)
roads_metric = roads_wgs84.to_crs(METRIC_CRS)
print(f" Wards metric: {len(wards_metric)}")
print(f" Roads metric: {len(roads_metric)}")
# Step 4: Find crossing points
print("=" * 60)
print("STEP 4: Find road/ward-boundary crossings")
print("=" * 60)
crossings = find_crossings(wards_metric, roads_metric)
print(f" Crossing points found: {len(crossings)}")
# Step 5: Create 50m buffers
print("=" * 60)
print("STEP 5: Create 50m buffers")
print("=" * 60)
if len(crossings) > 0:
buffers = crossings.copy()
buffers["geometry"] = crossings.geometry.buffer(50.0)
buffers = buffers.drop(columns=["road_osm_id", "highway"])
print(f" Buffers: {len(buffers)}")
else:
buffers = gpd.GeoDataFrame([], columns=[
"ward_id", "ward_name_en", "ward_name", "geometry"
], crs=METRIC_CRS)
# Step 6: Clip buffers to their respective wards
print("=" * 60)
print("STEP 6: Clip buffers to ward boundaries")
print("=" * 60)
buffer_intersections = []
for idx, buf in buffers.iterrows():
ward_id = buf["ward_id"]
ward_row = wards_metric[wards_metric["ward_id"] == ward_id]
if len(ward_row) == 0:
continue
ward_geom = ward_row.iloc[0].geometry
clipped = buf.geometry.intersection(ward_geom)
if not clipped.is_empty:
buffer_intersections.append({
"geometry": clipped,
"ward_id": ward_id,
"ward_name_en": buf["ward_name_en"],
"ward_name": buf["ward_name"],
})
if buffer_intersections:
buf_intersect_gdf = gpd.GeoDataFrame(buffer_intersections, crs=METRIC_CRS)
else:
buf_intersect_gdf = gpd.GeoDataFrame([], columns=[
"ward_id", "ward_name_en", "ward_name", "geometry"
], crs=METRIC_CRS)
print(f" Clipped buffer intersections: {len(buf_intersect_gdf)}")
# Step 7: Compute density stats
print("=" * 60)
print("STEP 7: Compute crossing density per ward")
print("=" * 60)
# Count crossings per ward
crossing_counts = {}
for _, c in crossings.iterrows():
wid = c["ward_id"]
crossing_counts[wid] = crossing_counts.get(wid, 0) + 1
# Compute areas in km² (from metric CRS - areas in m²)
wards_metric["ward_area_m2"] = wards_metric.geometry.area
wards_metric["ward_area_km2"] = wards_metric["ward_area_m2"] / 1_000_000
# Build density dataframe
density_records = []
for _, w in wards_metric.iterrows():
wid = w["ward_id"]
count = crossing_counts.get(wid, 0)
area_km2 = w["ward_area_km2"]
density_records.append({
"ward_id": wid,
"ward_name_en": w["ward_name_en"],
"ward_name": w["ward_name"],
"crossing_count": count,
"ward_area_km2": round(area_km2, 6),
"crossings_per_km2": round(count / area_km2, 4) if area_km2 > 0 else 0.0,
})
density_df = pd.DataFrame(density_records)
# Convert density to GeoDataFrame with WGS84 geometry
density_gdf = wards_wgs84[["ward_id", "ward_name_en", "ward_name", "geometry"]].copy()
density_gdf = density_gdf.merge(density_df, on=["ward_id", "ward_name_en", "ward_name"])
density_gdf = density_gdf[["ward_id", "ward_name_en", "ward_name", "crossing_count",
"ward_area_km2", "crossings_per_km2", "geometry"]]
print("\nDensity summary:")
print(density_gdf[["ward_name_en", "crossing_count", "ward_area_km2", "crossings_per_km2"]])
# Step 8: Write to GeoPackage
print("=" * 60)
print(f"STEP 8: Writing to {OUTPUT_GPKG}")
print("=" * 60)
# Remove if exists
if os.path.exists(OUTPUT_GPKG):
os.remove(OUTPUT_GPKG)
# Layer 1: wards_jgd (metric CRS)
wards_out = wards_metric[["ward_id", "ward_name_en", "ward_name", "geometry"]].copy()
wards_out.to_file(OUTPUT_GPKG, layer="wards_jgd", driver="GPKG")
print(f" Layer 'wards_jgd': {len(wards_out)} features")
# Layer 2: crossing_points (metric CRS)
if len(crossings) > 0:
crossing_out = crossings[["ward_id", "ward_name_en", "ward_name", "road_osm_id", "highway", "geometry"]].copy()
crossing_out.to_file(OUTPUT_GPKG, layer="crossing_points", driver="GPKG")
print(f" Layer 'crossing_points': {len(crossing_out)} features")
# Layer 3: crossing_buffers_50m (metric CRS)
if len(buffers) > 0:
buffers_out = buffers[["ward_id", "ward_name_en", "ward_name", "geometry"]].copy()
buffers_out.to_file(OUTPUT_GPKG, layer="crossing_buffers_50m", driver="GPKG")
print(f" Layer 'crossing_buffers_50m': {len(buffers_out)} features")
# Layer 4: buffer_ward_intersection (metric CRS)
if len(buf_intersect_gdf) > 0:
buf_intersect_gdf.to_file(OUTPUT_GPKG, layer="buffer_ward_intersection", driver="GPKG")
print(f" Layer 'buffer_ward_intersection': {len(buf_intersect_gdf)} features")
# Layer 5: ward_crossing_density_wgs84 (WGS84)
density_gdf.to_file(OUTPUT_GPKG, layer="ward_crossing_density_wgs84", driver="GPKG")
print(f" Layer 'ward_crossing_density_wgs84': {len(density_gdf)} features")
# Verify
print("\n" + "=" * 60)
print("VERIFICATION")
print("=" * 60)
for layer in gpd.list_layers(OUTPUT_GPKG):
gdf = gpd.read_file(OUTPUT_GPKG, layer=layer.name if hasattr(layer, 'name') else layer)
if isinstance(layer, str):
lname = layer
elif hasattr(layer, 'name'):
lname = layer.name
else:
lname = str(layer)
print(f" Layer '{lname}': {len(gdf)} features, CRS={gdf.crs}")
print("\nDone!")
if __name__ == "__main__":
main()Wrote /work/solve.py (16280 chars).
27:40
Bash
cd /work && python solve.py
============================================================ STEP 1: Download ward boundaries ============================================================ Downloading ward: Chiyoda (1761742)... Downloading ward: Chuo (1758897)... Downloading ward: Minato (1761717)... Downloading ward: Shinjuku (1758858)... Downloading ward: Bunkyo (1758878)... Downloading ward: Taito (1758888)... Downloading ward: Sumida (1758891)... Downloading ward: Koto (3554015)... Downloading ward: Shinagawa (3554304)... Downloading ward: Meguro (1758936)... Downloading ward: Ota (1758947)... Downloading ward: Setagaya (1759474)... Downloading ward: Shibuya (1759477)... Downloading ward: Nakano (1543056)... Downloading ward: Suginami (1543055)... Downloading ward: Toshima (1759506)... Downloading ward: Kita (1760038)... Downloading ward: Arakawa (1760040)... Downloading ward: Itabashi (1760078)... Downloading ward: Nerima (1760119)... Downloading ward: Adachi (1760124)... Downloading ward: Katsushika (1761718)... Downloading ward: Edogawa (1761743)... Wards in WGS84: 23 ============================================================ STEP 2: Download drivable roads (excluding service) ============================================================ Querying Overpass for drivable roads (excluding service)... Got 1217320 elements Built 197495 road line geometries Roads downloaded: 197495 ============================================================ STEP 3: Reproject to EPSG:6677 ============================================================ Wards metric: 23 Roads metric: 197495 ============================================================ STEP 4: Find road/ward-boundary crossings ============================================================ Crossing points found: 5459 ============================================================ STEP 5: Create 50m buffers ============================================================ Buffers: 5459 ============================================================ STEP 6: Clip buffers to ward boundaries ============================================================ Clipped buffer intersections: 5459 ============================================================ STEP 7: Compute crossing density per ward ============================================================ Density summary: ward_name_en crossing_count ward_area_km2 crossings_per_km2 0 Chiyoda 185 11.361513 16.2830 1 Chuo 100 11.273083 8.8707 2 Minato 248 23.884431 10.3833 3 Shinjuku 276 18.271967 15.1051 4 Bunkyō 243 11.367203 21.3773 5 Taito 153 10.093873 15.1577 6 Sumida 87 13.770894 6.3177 7 Koto 126 65.422936 1.9259 8 Shinagawa 256 26.065159 9.8215 9 Meguro 262 14.848302 17.6451 10 Ota 211 101.400994 2.0808 11 Setagaya 446 58.097629 7.6767 12 Shibuya 229 15.116938 15.1486 13 Nakano 304 15.582962 19.5085 14 Suginami 318 33.927774 9.3729 15 Toshima 318 12.982196 24.4951 16 Kita 218 20.501458 10.6334 17 Arakawa 155 10.206810 15.1859 18 Itabashi 292 32.109091 9.0940 19 Nerima 455 48.123892 9.4548 20 Adachi 209 53.275846 3.9230 21 Katsushika 221 34.813982 6.3480 22 Edogawa 147 81.418518 1.8055 ============================================================ STEP 8: Writing to /work/tokyo_crossings.gpkg ============================================================ Layer 'wards_jgd': 23 features Layer 'crossing_points': 5459 features Layer 'crossing_buffers_50m': 5459 features Layer 'buffer_ward_intersection': 5459 features Layer 'ward_crossing_density_wgs84': 23 features ============================================================ VERIFICATION ============================================================
Traceback (most recent call last):
File "/work/solve.py", line 443, in <module>
main()
~~~~^^
File "/work/solve.py", line 430, in main
gdf = gpd.read_file(OUTPUT_GPKG, layer=layer.name if hasattr(layer, 'name') else layer)
File "/usr/local/lib/python3.14/site-packages/geopandas/io/file.py", line 316, in _read_file
return _read_file_pyogrio(
filename, bbox=bbox, mask=mask, columns=columns, rows=rows, **kwargs
)
File "/usr/local/lib/python3.14/site-packages/geopandas/io/file.py", line 576, in _read_file_pyogrio
return pyogrio.read_dataframe(path_or_bytes, bbox=bbox, **kwargs)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/pyogrio/geopandas.py", line 382, in read_dataframe
result = read_func(
path_or_buffer,
...<15 lines>...
**kwargs,
)
File "/usr/local/lib/python3.14/site-packages/pyogrio/raw.py", line 200, in read
return ogr_read(
get_vsi_path_or_buffer(path_or_buffer),
...<15 lines>...
datetime_as_string=datetime_as_string,
)
File "pyogrio/_io.pyx", line 1417, in pyogrio._io.ogr_read
File "pyogrio/_io.pyx", line 314, in pyogrio._io.get_ogr_layer
pyogrio.errors.DataLayerError: Layer 'name' could not be opened
30:35
The error is in the verification section - gpd.list_layers returns different format than expected. Let me fix that verification code and re-run.
30:35
Bash
cd /work && python -c "
cd /work && python -c "
import geopandas as gpd
print(gpd.list_layers('/work/tokyo_crossings.gpkg'))
"
name geometry_type 0 wards_jgd MultiPolygon 1 crossing_points Point 2 crossing_buffers_50m Polygon 3 buffer_ward_intersection MultiPolygon 4 ward_crossing_density_wgs84 MultiPolygon
30:44
The verification part had a bug with how gpd.list_layers returns data, but the GeoPackage was written correctly. Let me verify the layers properly.
30:44
Bash
cd /work && python -c "
cd /work && python -c "
import geopandas as gpd
for layer in ['wards_jgd', 'crossing_points', 'crossing_buffers_50m', 'buffer_ward_intersection', 'ward_crossing_density_wgs84']:
gdf = gpd.read_file('/work/tokyo_crossings.gpkg', layer=layer)
print(f\"Layer '{layer}': {len(gdf)} features, CRS={gdf.crs}\")
print(f\" Columns: {gdf.columns.tolist()}\")
if 'ward_name_en' in gdf.columns:
print(f\" Ward names: {gdf['ward_name_en'].tolist()[:5]}...\")
if 'crossing_count' in gdf.columns:
print(f\" Stats: total crossings={gdf['crossing_count'].sum()}, min density={gdf['crossings_per_km2'].min():.2f}, max={gdf['crossings_per_km2'].max():.2f}\")
print()
"
Layer 'wards_jgd': 23 features, CRS=EPSG:6677 Columns: ['ward_id', 'ward_name_en', 'ward_name', 'geometry'] Ward names: ['Chiyoda', 'Chuo', 'Minato', 'Shinjuku', 'Bunkyō']... Layer 'crossing_points': 5459 features, CRS=EPSG:6677 Columns: ['ward_id', 'ward_name_en', 'ward_name', 'road_osm_id', 'highway', 'geometry'] Ward names: ['Chiyoda', 'Chiyoda', 'Chiyoda', 'Chiyoda', 'Chiyoda']... Layer 'crossing_buffers_50m': 5459 features, CRS=EPSG:6677 Columns: ['ward_id', 'ward_name_en', 'ward_name', 'geometry'] Ward names: ['Chiyoda', 'Chiyoda', 'Chiyoda', 'Chiyoda', 'Chiyoda']... Layer 'buffer_ward_intersection': 5459 features, CRS=EPSG:6677 Columns: ['ward_id', 'ward_name_en', 'ward_name', 'geometry'] Ward names: ['Chiyoda', 'Chiyoda', 'Chiyoda', 'Chiyoda', 'Chiyoda']... Layer 'ward_crossing_density_wgs84': 23 features, CRS=EPSG:4326 Columns: ['ward_id', 'ward_name_en', 'ward_name', 'crossing_count', 'ward_area_km2', 'crossings_per_km2', 'geometry'] Ward names: ['Chiyoda', 'Chuo', 'Minato', 'Shinjuku', 'Bunkyō']... Stats: total crossings=5459, min density=1.81, max=24.50
31:01
Let me check the data a bit more carefully. The results look reasonable. Let me verify: 1. `wards_jgd` - 23 ward boundaries in metric CRS (EPSG:6677) ✓ 2. `crossing_points` - 5459 crossing points ✓ 3. `crossing_buffers_50m` - 5459 buffers ✓ 4. `buffer_ward_intersection` - 5459 clipped buffers ✓ 5. `ward_crossing_density_wgs84` - 23 wards with stats in WGS84 ✓ Wait, I need to double-check the `buffer_ward_intersection` layer - the task says "each buffer clipped to the ward whose boundary it crossed". I did clip each buffer to its respective ward, but looking at the output geometry type, it says "MultiPolygon" which makes sense since clipping a buffer to the ward polygon could result in a polygon or multipolygon. However, there's one issue I notice - the ward "Bunkyo" comes through as "Bunkyō" because that's what OSM has. The original ward list I specified uses "Bunkyo" but the OSM data uses "Bunkyō". Let me check if this is correct for the name:en field. Also, I should verify the CRS of each layer. The spec says: - wards_jgd (Polygon) - in metric grid ✓ - crossing_points (Point) - in metric grid ✓ - crossing_buffers_50m (Polygon) - in metric grid ✓ - buffer_ward_intersection (Polygon) - in metric grid ✓ - ward_crossing_density_wgs84 (Polygon) - in WGS84 ✓ The layers look good. Let me check the geometry types more carefully - the output shows MultiPolygon for wards_jgd, which is correct since some wards may have multi-part geometries. For crossing_points it's Point, good. For buffers, it's Polygon. For buffer_ward_intersection it's MultiPolygon (since clipping a circle to a polygon can yield oddly shaped polygons). For ward_crossing_density_wgs84 it's MultiPolygon which is fine. Let me make a quick sanity check on the results - the numbers seem reasonable. Dense central wards like Toshima, Bunkyo, Nakano have 15-25 crossings per km², while larger outer wards like Ota, Edogawa have about 2/km². This makes sense. One thing I want to verify is that the road query is working correctly. The Overpass query excludes service roads, footways, cycleways, paths, pedestrian, track, steps, bridleway, corridor, escalator, proposed, construction, bus_guideway, raceway. This seems reasonable for "drivable roads." I think the solution is complete. Let me do a final check.
31:01
31:01
Bash
cd /work && python -c "
cd /work && python -c "
import geopandas as gpd
import pyproj
# Check all layers
print('=== GPKG Contents ===')
for lname in ['wards_jgd', 'crossing_points', 'crossing_buffers_50m', 'buffer_ward_intersection', 'ward_crossing_density_wgs84']:
gdf = gpd.read_file('/work/tokyo_crossings.gpkg', layer=lname)
crs = gdf.crs
print(f'\n--- {lname} ---')
print(f' Features: {len(gdf)}')
print(f' CRS: {crs}')
print(f' CRS axis units: {crs.axis_info}')
print(f' Geometry type(s): {gdf.geometry.geom_type.unique()}')
print(f' Columns: {gdf.columns.tolist()}')
print(f' Sample:')
for col in gdf.columns:
if col != 'geometry':
print(f' {col}: {gdf[col].values[:3]}')
# Verify CRS uses metres
crs = pyproj.CRS('EPSG:6677')
print(f'\nEPSG:6677 units: {crs.axis_info}')
print(f'Is projected: {crs.is_projected}')
print(f'Is geographic: {crs.is_geographic}')
# Check density layer is WGS84
crs_wgs = pyproj.CRS('EPSG:4326')
print(f'EPSG:4326 units: {crs_wgs.axis_info}')
"
=== GPKG Contents ===
--- wards_jgd ---
Features: 23
CRS: EPSG:6677
CRS axis units: [Axis(name=Northing, abbrev=X, direction=north, unit_auth_code=EPSG, unit_code=9001, unit_name=metre), Axis(name=Easting, abbrev=Y, direction=east, unit_auth_code=EPSG, unit_code=9001, unit_name=metre)]
Geometry type(s): <ArrowStringArray>
['MultiPolygon']
Length: 1, dtype: str
Columns: ['ward_id', 'ward_name_en', 'ward_name', 'geometry']
Sample:
ward_id: [1761742 1758897 1761717]
ward_name_en: <ArrowStringArray>
['Chiyoda', 'Chuo', 'Minato']
Length: 3, dtype: str
ward_name: <ArrowStringArray>
['千代田区', '中央区', '港区']
Length: 3, dtype: str
--- crossing_points ---
Features: 5459
CRS: EPSG:6677
CRS axis units: [Axis(name=Northing, abbrev=X, direction=north, unit_auth_code=EPSG, unit_code=9001, unit_name=metre), Axis(name=Easting, abbrev=Y, direction=east, unit_auth_code=EPSG, unit_code=9001, unit_name=metre)]
Geometry type(s): <ArrowStringArray>
['Point']
Length: 1, dtype: str
Columns: ['ward_id', 'ward_name_en', 'ward_name', 'road_osm_id', 'highway', 'geometry']
Sample:
ward_id: [1761742 1761742 1761742]
ward_name_en: <ArrowStringArray>
['Chiyoda', 'Chiyoda', 'Chiyoda']
Length: 3, dtype: str
ward_name: <ArrowStringArray>
['千代田区', '千代田区', '千代田区']
Length: 3, dtype: str
road_osm_id: [23594643 23671260 23681223]
highway: <ArrowStringArray>
['motorway', 'primary', 'motorway']
Length: 3, dtype: str
--- crossing_buffers_50m ---
Features: 5459
CRS: EPSG:6677
CRS axis units: [Axis(name=Northing, abbrev=X, direction=north, unit_auth_code=EPSG, unit_code=9001, unit_name=metre), Axis(name=Easting, abbrev=Y, direction=east, unit_auth_code=EPSG, unit_code=9001, unit_name=metre)]
Geometry type(s): <ArrowStringArray>
['Polygon']
Length: 1, dtype: str
Columns: ['ward_id', 'ward_name_en', 'ward_name', 'geometry']
Sample:
ward_id: [1761742 1761742 1761742]
ward_name_en: <ArrowStringArray>
['Chiyoda', 'Chiyoda', 'Chiyoda']
Length: 3, dtype: str
ward_name: <ArrowStringArray>
['千代田区', '千代田区', '千代田区']
Length: 3, dtype: str
--- buffer_ward_intersection ---
Features: 5459
CRS: EPSG:6677
CRS axis units: [Axis(name=Northing, abbrev=X, direction=north, unit_auth_code=EPSG, unit_code=9001, unit_name=metre), Axis(name=Easting, abbrev=Y, direction=east, unit_auth_code=EPSG, unit_code=9001, unit_name=metre)]
Geometry type(s): <ArrowStringArray>
['MultiPolygon']
Length: 1, dtype: str
Columns: ['ward_id', 'ward_name_en', 'ward_name', 'geometry']
Sample:
ward_id: [1761742 1761742 1761742]
ward_name_en: <ArrowStringArray>
['Chiyoda', 'Chiyoda', 'Chiyoda']
Length: 3, dtype: str
ward_name: <ArrowStringArray>
['千代田区', '千代田区', '千代田区']
Length: 3, dtype: str
--- ward_crossing_density_wgs84 ---
Features: 23
CRS: EPSG:4326
CRS axis units: [Axis(name=Geodetic latitude, abbrev=Lat, direction=north, unit_auth_code=EPSG, unit_code=9122, unit_name=degree), Axis(name=Geodetic longitude, abbrev=Lon, direction=east, unit_auth_code=EPSG, unit_code=9122, unit_name=degree)]
Geometry type(s): <ArrowStringArray>
['MultiPolygon']
Length: 1, dtype: str
Columns: ['ward_id', 'ward_name_en', 'ward_name', 'crossing_count', 'ward_area_km2', 'crossings_per_km2', 'geometry']
Sample:
ward_id: [1761742 1758897 1761717]
ward_name_en: <ArrowStringArray>
['Chiyoda', 'Chuo', 'Minato']
Length: 3, dtype: str
ward_name: <ArrowStringArray>
['千代田区', '中央区', '港区']
Length: 3, dtype: str
crossing_count: [185 100 248]
ward_area_km2: [11.361513 11.273083 23.884431]
crossings_per_km2: [16.283 8.8707 10.3833]
EPSG:6677 units: [Axis(name=Northing, abbrev=X, direction=north, unit_auth_code=EPSG, unit_code=9001, unit_name=metre), Axis(name=Easting, abbrev=Y, direction=east, unit_auth_code=EPSG, unit_code=9001, unit_name=metre)]
Is projected: True
Is geographic: False
EPSG:4326 units: [Axis(name=Geodetic latitude, abbrev=Lat, direction=north, unit_auth_code=EPSG, unit_code=9122, unit_name=degree), Axis(name=Geodetic longitude, abbrev=Lon, direction=east, unit_auth_code=EPSG, unit_code=9122, unit_name=degree)]
31:10
Everything looks good. Let me verify the file size and do a final check. Let me also check if I need to fix the Bunkyo name - it downloaded as "Bunkyō" which is fine. The ward_name_en should match - let me look at what was stored vs what was expected. Actually the data has "Bunkyō" which is the correct romanization. And the WARD_RELATIONS dict had "Bunkyo" but the tags from OSM had "Bunkyō". The code uses the tag value so it's fine. Let me just double check the file exists and is reasonable.
31:10
Bash
ls -lh /work/tokyo_crossings.gpkg
-rw-r--r-- 1 agent agent 14M Jun 18 03:47 /work/tokyo_crossings.gpkg
31:20
Everything looks good. Let me verify the solution meets all requirements: 1. ✅ `wards_jgd` (Polygon): 23 ward boundaries in EPSG:6677 (JGD2011 / Japan Plane Rectangular CS IX - metric) 2. ✅ `crossing_points` (Point): 5459 crossing points where roads cross ward boundaries 3. ✅ `crossing_buffers_50m` (Polygon): 50m buffers around each crossing point 4. ✅ `buffer_ward_intersection` (Polygon): each buffer clipped to the ward whose boundary it crossed 5. ✅ `ward_crossing_density_wgs84` (Polygon): one row per ward with `ward_id`, `crossing_count`, `ward_area_km2`, and `crossings_per_km2` in WGS84 Requirements met: - ✅ Data pulled from current OSM via Overpass (roads) and OSM API (wards) - ✅ Service roads excluded (query filters `highway!="service"`) - ✅ Engineered in metric CRS (EPSG:6677 - JGD2011 / Japan Plane Rectangular CS IX for Tokyo) - ✅ 50m buffer for jurisdictional reporting - ✅ All saved to single `tokyo_crossings.gpkg` - ✅ ward_id = OSM relation id - ✅ ward_name_en and ward_name carried on every ward-specific layer - ✅ Density layer in WGS84, others in metric grid The output file is 14 MB.
31:20