[openrailwaymap] Speeding up file-based rendering ("Offline" so we can use a static web server for the tiles)

[Niklas B]

Hi,

I'm just beginning my usage of Open Railway Map - thank you so much :) As a thank you for all the hard work you do I donated 100 EUR as well.

I've successfully setup everything on my machine and I'm now trying to generate complete file-based maps[1] up to zoom level 15 for Sweden. I've written a "offline renderer" using the node-tileserver that accepts a rectangular range of co-ordinates (i.e only Sweden) and iteratively generates the tiles. Generating the 715 378 unique tiles (x4 for all bitmap variants and json) takes roughly a week using 8 cores and 52GB of ram. I'm guessing I'm doing something super inefficient. 

- Machine: Google Cloud VM, 8 CPU with 52GB ram plus 200 GB of SSD disk
- Only loaded Europe, not entire planet.
- Postgresql:
	- 12GB shared buffer
	- 1.5GB work_mem
	- autovacuum = Off

The node worker is using Redis and Bull (https://github.com/OptimalBits/bull <https://github.com/OptimalBits/bull>) and is multi-core. It's a fairly simple version[2] (see source code), but I'm lacking quite a bit understanding of which parts of node-tileserver I need to use so it's a bit of a hack. 

My best guess is that it's the database queries that are slow I need to do additional tuning. Since the DB is only 6GB for Europe I'm thinking about moving Postgresql to ramdisk. 

Any helpful pointers? I've looked into the Postgresql tuning guides (both general and PostGIS). So far I'm only getting a few tiles per second. 

If you'r interested in the code itself it's available on https://gist.github.com/bivald/975e56a0850408255cbb6fc9e921df63 <https://gist.github.com/bivald/975e56a0850408255cbb6fc9e921df63>

I'm fluent in JS and Python if that helps. 

Regards,
Niklas

[1] The reason is that since we only want to cover small parts of the world, the hosting for generated maps is so much simpler then hosting API:s and TileServers - not to mention faster. 
[2] https://gist.github.com/bivald/975e56a0850408255cbb6fc9e921df63 <https://gist.github.com/bivald/975e56a0850408255cbb6fc9e921df63>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openrailwaymap.org/pipermail/openrailwaymap/attachments/20170925/fdd5b2b5/attachment.html>