How to make MBTiles valid on Mapbox Android SDK #mbtiles #android #mapbox

README.md

MBTilesSource.kt + MbtilesServer.kt

What it does?

It enables a MapboxMap (from Mapbox Android SDK) to use MBTiles as Source.

Inspired from similar Swift approach in iOS SDK, made by @namannik.

How it works?

In you APP, when a MBTilesSource instance activates, it starts a localhost MBTilesServer and works like a common Mapbox Raster/Vector Source.

MBTilesServer is a Singleton. It can hosts multiple MBTilesSource. The URL for each MBTilesSource is http://localhost:8888/[Source ID]/{z}/{x}/{y}.[jpg|png|pbf|mvt]

Since MBTilesSource only works as common Mapbox Raster/Vector Source, you always need to add layer or set style for MapboxMap by yourself. Mapbox Style is a bit more complex. If your MBTiles contains vector tile (.mvt), make sure your specify correct value for source-layer. See spec here.

Also, you may check an example by MapTiler, an OSM vector tiles provider.

Installation

• Add MBTilesSource.kt and MBTilesServer.kt (from this gist) into your Android project.

Usage

Prepare network configuration

To make sure MBTilesServer works without network connection above Android 9 (API level 28), you have to allow HTTP traffic to localhost.

For more information, see Android Developers.

1. Add network configuration into AndroidManifest.xml

   android:networkSecurityConfig="@xml/network_security_config" >

2. A resource file named in AndroidManifest.xml

   <?xml version="1.0" encoding="utf-8"?>
   <network-security-config>
       <domain-config cleartextTrafficPermitted="true">
           <domain includeSubdomains="false">localhost</domain>
       </domain-config>
   </network-security-config>

Create a MBTilesSource to read your MBTiles file

// Connect to localhost even when device is not connected to internet
Mapbox.setConnected(true)

// Create MBTilesSource
val path = "/path/to/source.mbtiles" // file path
val sourceId = "source1" // used as Mapbox Source ID
val mbSource = try {
    MBTilesSource(path, sourceId).apply { activate() }
} catch (e: MBTilesSourceException.CouldNotReadFileException){
    // Deal with error if fail to read MBTiles
}

Now, a XYZ Tile Service is working on http://localhost:8888/source1/{z}/{x}/{y}.[mvt|pbf|jpg|png].

The file extension of each tile is judged by the format in MBTiles. (Defined in table metadata, only mvt, pbf, jpg, png are allowed)

Also, if your MBTiles is at asset folder, you may use companion object in MBTilesSource to copy it into internal storage, and get the path directly.

val path = MBTilesSource.readAsset(context, "FILENAME.mbtiles")

Remove a source from MBTilesServer

 mbSource.deactivate()

Work with Mapbox Style

// In callback of Style.OnStyleLoaded
style.addSource(mbSource.instance)

// If mbSource contains raster tiles
val rasterLayer = RasterLayer("raster_layer_id", mbSource.id)
style.addLayer(rasterLayer)

// If mbSource contains vector tiles
val vectorLayer = LineLayer("vector_layer_id", mbSource.id)
style.addLayer(vectorLayer)

How it looks like?

I created this for my app, Five More Minutes. This demo use vector MBTiles and style from OpenMapTiles.

Alternatives

• If you want to create an APP with map which really reads MBTiles directly, consider use Tangram SDK instead of Mapbox SDK. It natively support use a MBTiles as source offline, see Documentation here.

  Do not challenge why Mapbox doesn't support a format they created :D

• Or, use Maplibre-gl-native, a mapbox-gl-native fork which embraces open source!

• If it won't bother you to store tiles with folders, you can use asset:// and file:// protocol with source. See related thread in StackOverflow here.

CHANGELOG

• 2021-01-26
  • Add Alternatives section
• 2020-11-25
  • Add network configuration for newer Android version, thanks @Cacaonut !
  • Remove Anko from dependencies
  • Code refactor
  • Exmaple refactor for latest Mapbox API
• 2020-05-26
  • Fix connection issue about localhost, thanks @grzesiek2010 !

MBTilesServer.kt

package com.example.sample.offline

import android.util.Log
import java.io.BufferedReader
import java.io.ByteArrayOutputStream
import java.io.FileNotFoundException
import java.io.PrintStream
import java.net.ServerSocket
import java.net.Socket
import kotlin.math.pow

/*
 * Localhost tile server with MBTilesSource
 */

object MBTilesServer : Runnable {

    const val port = 8888
    private val serverSocket: ServerSocket = ServerSocket(port)
    var isRunning = false
    val sources: MutableMap<String, MBTilesSource> = mutableMapOf()

    fun start() {
        isRunning = true
        Thread(this).start()
    }

    fun stop() {
        isRunning = false
        serverSocket.close()
    }

    override fun run() {
        try {
            while (isRunning) {
                serverSocket.accept().use { socket ->
                    Log.d(javaClass.simpleName, "Handling request")
                    handle(socket)
                    Log.d(javaClass.simpleName, "Request handled")
                }
            }
        } catch (e: Exception) {
            Log.d(
                javaClass.simpleName,
                e.localizedMessage ?: "Exception while running MBTilesServer"
            )
        } finally {
            Log.d(javaClass.simpleName, "request handled")
        }
    }

    @Throws
    private fun handle(socket: Socket) {

        val reader: BufferedReader = socket.getInputStream().reader().buffered()
        // Output stream that we send the response to
        val output = PrintStream(socket.getOutputStream())

        try {
            var route: String? = null

            // Read HTTP headers and parse out the route.
            do {
                val line = reader.readLine() ?: ""
                if (line.startsWith("GET")) {
                    // the format for route should be {source}/{z}/{x}/{y}
                    route = line.substringAfter("GET /").substringBefore(".")
                    break
                }
            } while (line.isNotEmpty())

            // the source which this request target to
            val source = sources[route?.substringBefore("/")] ?: return

            // Prepare the content to send.
            if (null == route) {
                writeServerError(output)
                return
            }

            val bytes = loadContent(source, route) ?: run {
                writeServerError(output)
                return
            }

            // Send out the content.
            with(output) {
                println("HTTP/1.0 200 OK")
                println("Content-Type: " + detectMimeType(source.format))
                println("Content-Length: " + bytes.size)
                if (source.isVector) println("Content-Encoding: gzip")
                println()
                write(bytes)
                flush()
            }
        } finally {
            reader.close()
            output.close()
        }
    }

    @Throws
    private fun loadContent(source: MBTilesSource, route: String): ByteArray? = try {
        val (z, x, y) = route.split("/").subList(1, 4).map { it.toInt() }
        source.getTile(z, x, (2.0.pow(z)).toInt() - 1 - y)
    } catch (e: FileNotFoundException) {
        e.printStackTrace()
        null
    }

    private fun writeServerError(output: PrintStream) {
        output.println("HTTP/1.0 500 Internal Server Error")
        output.flush()
    }

    private fun detectMimeType(format: String): String? = when (format) {
        "jpg" -> "image/jpeg"
        "png" -> "image/png"
        "mvt" -> "application/x-protobuf"
        "pbf" -> "application/x-protobuf"
        else -> "application/octet-stream"
    }
}

MBTilesSource.kt

package com.example.sample.offline

import android.content.Context
import android.database.sqlite.SQLiteDatabase
import com.mapbox.mapboxsdk.style.sources.RasterSource
import com.mapbox.mapboxsdk.style.sources.Source
import com.mapbox.mapboxsdk.style.sources.TileSet
import com.mapbox.mapboxsdk.style.sources.VectorSource
import java.io.File
import java.io.FileOutputStream
import kotlin.properties.Delegates

/*
 *  Mapbox Source backend by localhost tile server
 */

sealed class MBTilesSourceException : Exception() {
    class CouldNotReadFileException : MBTilesSourceException()
    class UnsupportedFormatException : MBTilesSourceException()
}

class MBTilesSource(filePath: String, sourceId: String? = null) {

    val id = sourceId ?: filePath.substringAfterLast("/").substringBefore(".")
    val url get() = "http://localhost:${MBTilesServer.port}/$id/{z}/{x}/{y}.$format"
    private val db: SQLiteDatabase = try {
        SQLiteDatabase.openOrCreateDatabase(filePath, null)
    } catch (e: RuntimeException) {
        throw MBTilesSourceException.CouldNotReadFileException()
    }
    val instance: Source by lazy {
        if (isVector) VectorSource(id, TileSet(null, url))
        else RasterSource(id, TileSet(null, url))
    }

    var isVector by Delegates.notNull<Boolean>()
    lateinit var format: String
//    var tileSize: Int? = null
//    var layersJson: String? = ""
//    var attributions: String? = ""
//    var minZoom: Float? = null
//    var maxZoom: Float? = null
//    var bounds: LatLngBounds? = null

    init {
        try {
            format = db.query(
                "metadata", null, "name = ?",
                arrayOf("format"), null, null, null
            ).use { cursor ->
                cursor.moveToFirst()
                val index = cursor.getColumnIndex("value")
                cursor.getString(index)
            }

            isVector = when (format) {
                in validVectorFormats -> true
                in validRasterFormats -> false
                else -> throw MBTilesSourceException.UnsupportedFormatException()
            }

        } catch (error: MBTilesSourceException) {
            print(error.localizedMessage)
        }
    }

    fun getTile(z: Int, x: Int, y: Int): ByteArray? {
        return db.query(
            "tiles", null, "zoom_level = ? AND tile_column = ? AND tile_row = ?",
            arrayOf("$z", "$x", "$y"), null, null, null
        ).use { cursor ->
            if (cursor.count == 0) return null

            cursor.moveToFirst()
            val index = cursor.getColumnIndex("tile_data")
            cursor.getBlob(index)
        }
    }

    fun activate() = with(MBTilesServer) {
        sources[id] = this@MBTilesSource
        if (!isRunning) start()
    }

    fun deactivate() = with(MBTilesServer) {
        sources.remove(id)
        if (isRunning && sources.isEmpty()) stop()
    }

    companion object {
        val validRasterFormats = listOf("jpg", "png")
        val validVectorFormats = listOf("pbf", "mvt")

        fun readAsset(context: Context, asset: String): String =
            context.assets.open(asset).use { inputStream ->
                val path = context.getDatabasePath(asset).path
                val outputFile = File(path)
                FileOutputStream(outputFile).use { outputStream ->
                    inputStream.copyTo(outputStream)
                    outputStream.flush()
                }
                return path
            }
    }
}

How can I find what source layers I need to add? Does the mbtiles file tell me?

1. Ask the provider of this MBTiles. For example, openmaptiles list their schema at https://openmaptiles.org/schema/
2. According to MBTiles spec, the layers are described as json object only after 1.3 (as you mentioned

Also, how do I know what type of layer (line, circle, fill, etc...) to create and add to the style?

It depends on your usecase. For example, a source layer with geometry=LineString can be used with SymbolLayer or LineLayer.
If you are not familiar with these Mapbox styling spec, better to use Maputnik or Mapbox studio to do some practice/design.

Hi @typebrook. I'm using Mapbox version 10.0.0-rc.6, I can load mbtiles but I can add any Annotation or GeoJsonSource after load style uri from asset. How can I fix that issue. Thank you so much

Hi @hungtrn75

I can load mbtiles but I cannot add any Annotation or GeoJsonSource after load style uri from asset

Because you said you can load mbtiles, then I guess MBTilesSource works in your case.
To be clear, a style contains several sources as source of map data, and MBTilesSource is just one of them.

So if you cannot add any new source into current style, then I guess it's because you didn't call Mapbox APIs well.
But anyway, I think your issue is not related to this gist. Maybe make a new stackoverflow post would helps.

If I remove the code to load style json from asset, it works fine. I realized that OnStyleLoaded couldn't be called when passing styleUri as an asset so I thought it was a Mapbox issue. Thanks for quick response @typebrook

Hello! I have a tiles server, its url is as follows http: // xxxxx / tiles / {x} / {y} / {z} .pbf what I need instead of taking, the files from the cell phone are from that url of my server ... is that possible in this implementation?

@alenigrelli
You have your own tile server? Then of course you don't need this gist.

If you are using Mapbox SDK or any other tools (Like MapLibre) which support Mapbox style spec on your mobile APP, just add endpoint of your tile server as one of Vector Source in style.

Hi, will this see an update with the Mapbox SDK V10 changes? Adding the MBTilesSource using addSource() like before:

style.addSource(mbSource.instance)

Is giving:

com.mapbox.maps.MapboxStyleException: Add source failed: source must have tiles

This is with a sample mbtiles file (countries.mbtiles from this). Will report back if I find a solution.

Woo~ It's been a long time I review this gist.

I guess the problem is here:
https://gist.github.com/typebrook/7d25be326f0e9afd58e0bbc333d2a175#file-mbtilessource-kt-L31-L34

Depends on the content of MBTiles, instance either return a RasterSource or VectorSource
Looks like from SDK V10, The argument for their constructor changes:

// Before V9
// https://docs.mapbox.com/android/maps/api/9.7.1/
fun VectorSource(id: String id, uri: Uri)

// V10
// https://docs.mapbox.com/android/maps/api/10.10.1/mapbox-maps-android/com.mapbox.maps.extension.style.sources.generated/-vector-source/
fun VectorSource(builder: VectorSource.Builder)

@ahmedmansour3548
Would you mind fix this by yourself and give me a PR?

Although I can directly change code by document. But to test the code, it took me a while to prepare Android development environment on my PC. (I don't code for Android recently)

PR could be done by the following steps:

1. Fork this gist
2. Make a new commit (Either on gist.github.com or push from local)
3. Comment here with your gist URL (Or git remote URL), or mail me patch files from the following command:

   git format-patch 39fca96