Hi,
I am currently replacing the old hardware (a Hetzner EX42 aka buegelfalte.openrailwaymap.org) by a new server at Hetzner (aka knallfrosch.openrailwaymap.org).
My plan was to first prerender all tiles on zoom levels 0 to 12 as it is good practice for raster tile serving. Afterwards I planned to make the new server serve all IPv6 clients (28 % of all tile requests [1]) in order to get important tiles on larger zoom levels rendered before I put full load (the remaining IPv4 traffic) on the new server.
Everything worked fine when I changed the DNS entries on Saturday evening but I did not notice broken rendering until Sunday morning. :-| Random ways were missing. After short investigation I found that those ways were missing in the rendering database because Osm2pgsql failed to generated geometries for them. The initial import was made using a node location cache in the database (the planet_osm_nodes table) but the updates used a flatnodes file (cache on NVMe storage). This was caused by a bug in the shell script. It was not noticeable on the old server because that used a cache in the database.
https://github.com/OpenRailwayMap/server-admin/commit/35c9c84737003092644d13...
The database import was quick but rerendering zoom levels 0 to 12 takes more than six hours while serving IPv6 clients. I changed the DNS entries again to point to the old server. It will very likely need the whole night (UTC+2) to complete rerendering.
API requests have not been affected because they have been served by the old server for all clients.
I will continue moving IPv6 traffic followed by IPv4 traffic to the new server after rerendering is completed.
Mailing lists are not affected at all, I migrated them to a new virtual machine (wal.openrailwaymap.org) in June.
Best regards
Michael
[1] 30,072,801 requests in total per day, measured on Thursday, 18 September