From osm-ml at michreichert.de  Tue Oct 17 21:05:29 2017
From: osm-ml at michreichert.de (Michael Reichert)
Date: Tue, 17 Oct 2017 21:05:29 +0200
Subject: [openrailwaymap] Speeding up file-based rendering ("Offline" so
 we can use a static web server for the tiles)
In-Reply-To: <2F808CCB-65CE-461F-BE3C-C6B575F94809@enplore.com>
References: <2F808CCB-65CE-461F-BE3C-C6B575F94809@enplore.com>
Message-ID: <e4e97879-7b61-7e9b-6629-2041f7b8e04e@michreichert.de>

Hi Niklas,

Am 25.09.2017 um 11:16 schrieb Niklas B:
> I've successfully setup everything on my machine and I'm now trying to generate complete file-based maps[1] up to zoom level 15 for Sweden. I've written a "offline renderer" using the node-tileserver that accepts a rectangular range of co-ordinates (i.e only Sweden) and iteratively generates the tiles. Generating the 715 378 unique tiles (x4 for all bitmap variants and json) takes roughly a week using 8 cores and 52GB of ram. I'm guessing I'm doing something super inefficient. 
> 
> - Machine: Google Cloud VM, 8 CPU with 52GB ram plus 200 GB of SSD disk
> - Only loaded Europe, not entire planet.
> - Postgresql:
> 	- 12GB shared buffer
> 	- 1.5GB work_mem
> 	- autovacuum = Off
> 
> The node worker is using Redis and Bull (https://github.com/OptimalBits/bull <https://github.com/OptimalBits/bull>) and is multi-core. It's a fairly simple version[2] (see source code), but I'm lacking quite a bit understanding of which parts of node-tileserver I need to use so it's a bit of a hack. 
> 
> My best guess is that it's the database queries that are slow I need to do additional tuning. Since the DB is only 6GB for Europe I'm thinking about moving Postgresql to ramdisk. 

I did not try it but partial indexes might help (in addition to the
spatial index which is created by osm2pgsql):
CREAETE INDEX <name> ON <table> USING gist(way) WHERE <where clause of
expensive query>;

You can find the SQL queries used for the generation of the vector tiles
in the config.json file.

That's the only advice I am able to give based on my knowledge (mainly
the classic Mapnik toolchain).

Best regards

Michael


-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschl?sselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openrailwaymap.org/pipermail/openrailwaymap/attachments/20171017/c21d07f7/attachment.sig>