Use cases to Export Data

In some cases, you may find that you need to export data from Countly. This can happen in one of two cases:

1. Archiving older data and keeping short term data in Countly for faster server experience or reducing server costs, etc.
2. Putting data that Countly collects in some data lake, which combines data from multiple other platforms so data scientists can connect all the data in one place

This article talks about how you can undertake such an export.

With DB Viewer

Before we dive into DB Viewer API, please note that there is also UI for DB Viewer in Countly dashboard, which allows easy data export of specific collections and queries in JSON, CSV, and Excel formats.

Get Information About All Apps

This can be done through a standard API call available in the core; it will provide you with info about apps that exist and are accessible for user with provided api_key.

For example, this would be useful for getting a list of app_ids if you are writing script to iterate through all apps.

Example call (replacing domain.com and {api_key} for user)
https://domain.com/o/apps/all?api_key={api_key}

Get more info about the API call at https://api.count.ly/reference#oappsall

Get User Information About a Specific App

This API call involves DB Viewer API and will list all documents of all users and their properties that are available for the specific app.

Example (replacing domain.com and {app_id} for the app you want to get info and {api_key} for user)

http://domain.com/o/db?dbs=countly&collection=app_users{app_id}&limit=1000000000000&filter={"_id":{"$ne":"uid-sequence"}}&api_key={api_key}

In the response object, the collections array should have all the user objects with their properties.

If there are too many users to read in single request, you can paginate result by using limit and skip parameters. As example:

http://domain.com/o/db?dbs=countly&collection=app_users{app_id}&limit=100&skip=200&filter={"_id":{"$ne":"uid-sequence"}}&api_key={api_key}

More information on pagination and filtering: https://api.count.ly/reference#odbdbsdbcollectioncollection)

More information on possible user properties that you might get: https://support.count.ly/hc/en-us/articles/360037681532-countly-app-users-appID-

If you want to monitor only new users, then you can adjust the filter part as:

http://domain.com/o/db?dbs=countly&collection=app_users{app_id}&limit=1000000000000&filter={"_id":{"$ne":"uid-sequence"},"fs":{"$gt":1470009600}}&api_key={api_key}

replacing 1470009600 with timestamp of the last data sync. You can also apply other filtering options or perform any kind of MongoDB query using the filter clause.

Get All Event Data

Getting all events' data would require 2 steps. What you need to do is get all event keys for a specific app, iterate through those event key and create sha1 hashes to be able to request data for each event specifically through DB Viewer API.

Step 1

The first step would be getting a list of all event keys that are available for the provided app.

Example call (replacing domain.com and {app_id} for the app you want to get info and {api_key} for user)

http://domain.com/o?api_key={api_key}&app_id={app_id}&method=get_events

Get more information on this API call at: https://api.count.ly/reference#omethodget_events

There is a property named “list” on the response object, which is an array of all event keys.

So you got all event keys that you had used in events in your app using Countly SDK.

Additionally there are some “built in Countly events” that are stored as events but are internal events, such as:

• [CLY]_session to get session data
• [CLY]_crash to get crash data
• [CLY]_push_open to get push opened data
• [CLY]_push_action to get push action data
• [CLY]_view to get view data

If you need any information on that data from Drill, then include them in your script’s event key array (by adding them to the array of event keys you retrieved from the API).

Step 2

So now we need to create sha1 hashes for each event and use them in REST API calls to get information about any specific event.

Hash is formed using sha1 hash for event key together with app ID.

For example, for event with key “Buy” and for app ID “542e95d747f0be510c000004” the hash would be sha1(“Buy542e95d747f0be510c000004”) and could look like: “57c2b7d4eeae912495088c3754d0c6cfe9949f06”

And the request to get data about specific event is (replacing domain.com and {hash} for the event hash you want to get info and {api_key} for user)

http://domain.com/o/db?dbs=countly_drill&collection=drill_events{hash}&limit=1000000000000&filter={"_id":{"$ne":"meta"}}&api_key={api_key}

Again, if there are too many events for single request you can paginate using limit and skip parameters as discussed when getting users. This is applicable because it is the same DB Viewer API call with same parameters, only for another collection.

You can then recreate event timeline using 'ts' property, which is the timestamp that the event occurred.

Another example: if you want to monitor only latest data, then you can adjust filter with timestamp in milliseconds (replacing 1470009600000 with timestamp in milliseconds of the last data sync)

http://domain.com/o/db?dbs=countly_drill&collection=drill_events{hash}&limit=1000000000000&filter={"_id":{"$ne":"meta"},"ts":{"$gt":1470009600000}}&api_key={api_key}

Similarly, you can modify filter parameter to perform any kind of MongoDB queries through filter parameter to filter out event data.

Get more info about possible data and properties that you can retrieve using this request at:
https://support.count.ly/hc/en-us/articles/360038263211-countly-drill-drill-events-ID-

Without DB Viewer

Directly from MongoDB

You can also take a direct and complete data dump from MongoDB using MongoDB tools.

Using mongodump (https://docs.mongodb.com/database-tools/mongodump/) command to dump the whole database in unreadable BSON format and then restore it using mongorestore(https://docs.mongodb.com/database-tools/mongorestore/https://docs.mongodb.com/database-tools/mongorestore/) command.

Or alternatively, you can use mongoexport (https://docs.mongodb.com/database-tools/mongoexport/) which can also export specific collections from specific databases (or even results of queries) in JSONL format, which can be imported respectively using mongoimport (https://docs.mongodb.com/database-tools/mongoimport/) command.