Is Countly suitable for large scale, general purpose data collection?
Hello Everybody,
I am wondering: Is Countly suitable for large scale, general purpose data collection? For example, long-time collection of custom data of several 100.000 of IoT devices? Each device firing at least a dozen events (each up to 5kb) per hour and the use case would require that (complete) individual events of a device as well as event history per device can be fetched for external processing of data. UI and analytics would not be used, only API and Mongo DB.
Is this a feasible and pragmatically realistic Countly usage or would it rather be a misuse?
Comments
Hi Marc,
Yes, Countly supports that. The only thing we strongly recommend is to delete old data; not keep it in the database for too long, or transfer it to a data lake if you want to keep it longer. The main reason for this, is that MongoDB requires keeping a working set of data and all indexes in memory, so with larger data sets, it means it will require larger and larger amount of memory; once memory is not enough and it starts paging memory to disk, MongoDB queries start to become really slow.
Please sign in to leave a comment.