Use cases to Export Data
In some cases, you may find that you need to export data from Countly. This can happen in one of two cases:
- Archiving older data and keeping short-term data in Countly for faster server experience or reducing server costs, etc.
- Putting data that Countly collects in some data lake, which combines data from multiple other platforms so data scientists can connect all the data in one place
This article talks about how you can undertake such an export.
With DB Viewer
Before we dive into the DB Viewer API, please note that there is also a UI for DB Viewer in the Countly dashboard, which allows easy data export of specific collections and queries in JSON, CSV, and Excel formats.
Getting Information About All Apps
This can be done through a standard API call available in the core; it will provide you with info about apps that exist and are accessible for the user with a provided api_key.
For example, this would be useful for getting a list of app_ids if you are writing script to iterate through all apps.
Example call (replacing domain.com and {api_key} for user)
https://domain.com/o/apps/all?api_key={api_key}
Get more info about the API call at https://api.count.ly/reference#oappsall.
Getting User Information About a Specific App
This API call involves a DB Viewer API and will list all documents of all users and their properties that are available for the specific app.
Example (replacing domain.com and {app_id} for the app you want to get info and {api_key} for user)
http://domain.com/o/db?dbs=countly&collection=app_users{app_id}&limit=1000000000000&filter={"_id":{"$ne":"uid-sequence"}}&api_key={api_key}
In the response object, the collections array should have all the user objects with their properties.
If there are too many users to read in single request, you can paginate the result by using the limit and skip parameters. For example:
http://domain.com/o/db?dbs=countly&collection=app_users{app_id}&limit=100&skip=200&filter={"_id":{"$ne":"uid-sequence"}}&api_key={api_key}
More information on pagination and filtering: https://api.count.ly/reference#odbdbsdbcollectioncollection)
More information on possible user properties that you might get: https://support.count.ly/hc/en-us/articles/360037681532-countly-app-users-appID-
If you want to monitor only new users, then you can adjust the filter part as:
http://domain.com/o/db?dbs=countly&collection=app_users{app_id}&limit=1000000000000&filter={"_id":{"$ne":"uid-sequence"},"fs":{"$gt":1470009600}}&api_key={api_key}
replacing 1470009600 with the timestamp of the last data sync. You can also apply other filtering options or perform any kind of MongoDB query using the filter clause.
Getting All Event Data
Getting all events' data would require 2 steps. What you need to do is get all event keys for a specific app, iterate through those event keys and create shard hashes to be able to request data for each event specifically through DB Viewer API.
Step 1
The first step would be to get a list of all the event keys available for the provided app.
Example call (replacing domain.com and {app_id} for the app you want to get info and {api_key} for user)
http://domain.com/o?api_key={api_key}&app_id={app_id}&method=get_events
Get more information on this API call at: https://api.count.ly/reference#omethodget_events
There is a property named “list” on the response object, which is an array of all event keys.
So you got all event keys that you had used in events in your app using Countly SDK.
Additionally, there are some “built-in Countly events” that are stored as events but are internal events, such as:
- [CLY]_session to get session data
- [CLY]_crash to get crash data
- [CLY]_push_open to get push opened data
- [CLY]_push_action to get push action data
- [CLY]_view to get view data
- [CLY]_star_rating to get ratings
- [CLY]_nps to get NPS survey results
- [CLY]_survey to get survey results
- [CLY]_action to get click actions from websites
- [CLY]_apm_device to get apm device metrics
- [CLY]_apm_network to get apm network metrics
If you need any information on that data from Drill, then include them in your script’s event key array (by adding them to the array of event keys you retrieved from the API).
Step 2
So now we need to create sha1 hashes for each event and use them in REST API calls to get information about any specific event.
Hash is formed using sha1 hash for event key together with app ID.
For example, for event with key “Buy” and for app ID “542e95d747f0be510c000004” the hash would be sha1(“Buy542e95d747f0be510c000004”) and could look like: “57c2b7d4eeae912495088c3754d0c6cfe9949f06”
And the request to get data about a specific event is (replacing domain.com and {hash} for the event hash you want to get info and {api_key} for user)
http://domain.com/o/db?dbs=countly_drill&collection=drill_events{hash}&limit=1000000000000&filter={"_id":{"$ne":"meta"}}&api_key={api_key}
Again, if there are too many events for a single request, you can paginate using limit and skip parameters as discussed when getting users. This is applicable because it is the same DB Viewer API call with the same parameters, only for another collection.
You can then recreate the event timeline using the ts property, which is the timestamp that the event occurred.
Another example: if you want to monitor only the latest data, then you can adjust the filter with a timestamp in milliseconds (replacing 1470009600000 with a timestamp in milliseconds of the last data sync)
http://domain.com/o/db?dbs=countly_drill&collection=drill_events{hash}&limit=1000000000000&filter={"_id":{"$ne":"meta"},"ts":{"$gt":1470009600000}}&api_key={api_key}
Similarly, you can modify the filter parameters to perform any kind of MongoDB queries to filter out event data.
Get more info about possible data and properties that you can retrieve using this request at:
https://support.count.ly/hc/en-us/articles/360038263211-countly-drill-drill-events-ID-
Without DB Viewer
Directly from MongoDB
You can also take a direct and complete data dump from MongoDB using MongoDB tools.
Using the mongodump (https://docs.mongodb.com/database-tools/mongodump/) command to dump the whole database in unreadable BSON format and then restore it using the mongorestore(https://docs.mongodb.com/database-tools/mongorestore/https://docs.mongodb.com/database-tools/mongorestore/) command.
Or alternatively, you can use mongoexport (https://docs.mongodb.com/database-tools/mongoexport/) which can also export specific collections from specific databases (or even results of queries) in JSON or CSV formats, which can be imported respectively using the mongoimport (https://docs.mongodb.com/database-tools/mongoimport/) command.
Using the Data Migration Plugin
Countly-hosted clients do not have direct access to their MongoDB, impending them from running commands like mongodump. Therefore, using the Data Migration plugin will allow Countly-hosted clients to complete the DB export. You can find more information in the Data Migration documentation.
Incremental Exports
There might be cases when you need to do incremental backups for the new data since the last backup. For these cases, it is possible to use the Data Migration, but it is not recommended because it cannot have multiple migration exports for the same app. Since Data Migration assists for exports/imports between different servers, every time you do an incremental backup, you will need to find a way to store them in a different server.
For incremental exports, we suggest using database snapshots, which will also make the process faster.