How We Reduced our Mapping costs by 90% using Protomaps and Cloudflare

One of our internal goals in Sakay is reducing its operating expenses. One cost center in particular is mapping since as a transport app, we display maps pretty much everywhere in our application, as well as web pages.

We initially used Google Maps and with its generous free tier, the costs were manageable. However when Google hiked the price of Google Maps back in 2018 the costs of maps increased by tenfold so we moved to alternative map platforms like Mapbox and Maptiler to keep the costs down. During that time, all self-hosted map solutions either require populating the server with thousands of files or having a fewer number of files that needs running your own application server to host them. In either case the complexity and server resources required are pretty hefty so relying on a Map SaaS made sense.

Recently, two developments made self-hosted maps simple and economical:

1. Protomaps: a file format that stores all the map tiles in one basemap file, making it manageable to host. More importantly the format arranges the data in such a way that getting a tile in the basemap simply requires getting a subrange of the file directly as opposed to having to load the entire file and doing complex file traversal. This makes it possible to host the basemap in an object storage services that supports Byte-Range Fetches, which almost all have support, without the need of a database or non-trivial application to host the map. So the map tile can be served either directly from the object storage or through a serverless function like Cloudflare Workers, AWS Lambda, or Google Cloud Functions.
2. Cloudflare R2: An object storage service by Cloudflare that’s compatible with Amazon’s S3. Its zero egress cost makes hosting large files like maps very economical compared to offerings by Amazon and Google. The downside with their offering is that R2 by itself cannot publish public URLs to files; serving files to the public internet has to be done over Cloudflare workers.

So the plan is pretty clear: generate a Protomap basemap and upload the data to Cloudflare R2 in order to serve the map tiles through Cloudflare Workers.

Generating Protomap Basemap

Pregenerated Protomap basemaps can be downloaded from https://docs.protomaps.com/basemaps/downloads which may work for most cases. However, these basemaps have their own layer schema (see https://docs.protomaps.com/basemaps/layers) that is different from Maptiler or Mapbox so simply using these basemaps as a drop-in replacement while using existing map styles will not work. We either have to generate the basemap with the schema like Mapbox, or modify the map styles to target the schema of Protomap. We chose the former option since this way, we only need to convert the schema of one basemap as opposed to having to convert multiple styles to fit the schema.

So to generate the Protomap basemap (ending with .pmtiles) from OpenStreetMap extracts (in osm.pbf format downloadable from https://download.geofabrik.de), we need two tools:

• Tilemaker to convert osm.pbf files to .mbtiles format, and
• go-pmtiles to convert the .mbtiles file to .pmtiles

While go-pmtiles, which is written in Go, installs and runs smoothly on Windows (the prebuilt binaries in the project page just works), Tilemaker is trickier to compile on Windows, so I just resorted to compiling and running it on WSL2. Using the Pengwin Distro, the following commands are run to install Tilemaker:

$ sudo apt install libboost1.74-all-dev rapidjson-dev\
  libprotobuf-dev protobuf-compiler libsqlite3-dev libshp-dev libluajit-5.1-dev
$ make -e CONFIG=-DFAT_TILE_INDEX #FAT_TILE_INDEX enables generation of zoom levels >17
$ sudo make install

The conversion from OSM PBF to mbtiles is not a simple translation. It is at this stage where OSM elements, which is comprised of Nodes, Ways, and Relations are processed into semantic layers like “roads”, “highways”, “administrative boundaries” using Lua scripts that Tilemaker runs. For many existing styles, such as those used by Maptiler, the OpenMapTiles processing scripts in https://github.com/systemed/tilemaker/tree/master/resources will do the processing to layers compatible with those styles.

$ tilemaker --input map.osm.pbf  --output map.mbtiles --config tilemaker/resources/config-openmaptiles.json  --process tilemaker/resources/process-openmaptiles.lu

Once the MVT is produced from Tilemaker, it can now be converted to pmtiles using go-pmtiles convert subcommand.

$ pmtiles convert map.mbtiles map.pmtiles

Hosting

We then provisioned a Cloudflare R2 bucket and uploaded the resulting pmtiles to it using rclone.

$ rclone copy map.pmtiles "bucket-name:path/"

Protomaps have provided a Cloudflare Worker script which does most of the basic task of serving the map tile give an HTTP request: https://github.com/protomaps/PMTiles/tree/main/serverless/cloudflare, the configuration of the script, such as specifying the bucket name, is done via environment variables.

All in all, the time it took to host the basemap serving the map tiles took less than an hour. This is way faster than other alternatives which require setting up a database server or a NodeJS application server (together with a reverse proxy).

For the styles such as the classic OSM Bright style (style source is hosted on https://github.com/openmaptiles/osm-bright-gl-style), we opted to self-host them using Cloudflare Pages after modifying the source field in the style.json to point to our map tile server.

To make the most out of the cost advantage, we have to make sure that the caching is configured correctly.

Costs

Protomaps has provided a cost calculator: https://docs.protomaps.com/deploy/cost and in our experience, it’s quite accurate and reflects the operating costs we encountered.

Our map cost shrunk to a tenth of the original cost despite also having to self-host the styles. Since we were already paying for Cloudflare Pro, the Worker allocation that comes with it also absorbed a huge chunk of our costs and by limiting the map to just cover the Philippines, we fall under R2’s free tier, virtually eliminating the storage costs for the basemap.

Conclusion

Migrating to Protomaps do require some more work compared to using existing map services but when those map services start to cost a lost, it may be worth it to self-host the maps on Cloudflare using Protomaps to drastically cut costs.

However, additional factors should be considered such as map updates, workflow integration, and geocoding needs. We automated the map conversion process to a cron job to resolve the map update issue and reworked our tooling to use the Protomap basemaps. The engineering needed is small, but still non-trivial, still, we were willing to invest in such efforts as the tradeoff is worth it.

Location data is present in many data systems and leveraging Mapping and Geographic Information Systems (GIS) unlocks valuable insight to what is otherwise hidden in the sea of numbers.

Whether you are looking to reduce costs from mapping, develop location-based apps, or analyze geographic data, our team of experts can help. Reach out to us and discover how geospatial technology can transform your business!