Symbols and HTML entities

Countly Server: 22.03

Countly iOS SDK: 21.11.2

Something very strange and frustrating is happening with all of my event names and metrics submitted into Countly: all metrics data and event names that contain symbols are now showing up as HTML encoded entities in the Countly dashboard.

Upon investigation, I noticed that this information is ALSO encoded as HTML entities in the data submitted into the countly endpoints by the Countly SDK on the device.

For instance, an event is logged like this:

{
  "app_key": "087052fde5212020965a8babf8e219f1696fccc5",
  "device_id": "096DEA23-D8E5-4B58-A628-4147D896224A",
  "timestamp": "1655232675997",
  "hour": "14",
  "dow": "2",
  "tz": "-240",
  "sdk_version": "21.11.0",
  "sdk_name": "objc-native-ios",
  "events": [
    {
      "dur": 0,
      "hour": 14,
      "segmentation": {
        "app_runCount": "741",
        "src_line": "52",
        "id_device": ""52FDD2FB-875D-4C58-A01E-EA7464453F58"",
        "src_function": ""application(_:willFinishLaunchingWithOptions:)"",
        "id_vendor": ""48F63BC8-24CC-405E-8874-1C0C48AE2B2E"",
        "app_theme": "".deep"",
        "app_version": ""3.1.7"",
        "app_reviewed": "2021-11-07 20:49:34 +0000",
        "app_premium": "true",
        "app_build": ""54"",
        "app_environment": "private",
        "app_appIcon": "Spectre.AppIcon.dark",
        "app_rating": "5",
        "app_container": ""app"",
        "id_owner": ""096DEA23-D8E5-4B58-A628-4147D896224A"",
        "app_colorfulSites": "true",
        "src_file": ""AppDelegate.swift"",
        "app_masterPasswordCustomer": "true",
        "app_allowHandoff": "true"
      },
      "count": 1,
      "key": "app >startup",
      "timestamp": 1655232653833,
      "dow": 2,
      "sum": 0
    },

You'll notice the metrics are getting their data HTML encoded, and even the event name is. Other elements in the request are not, such as the device_id at the top.  This HTML encoding makes no sense whatsoever in the context of a JSON-encoded request. We ALSO notice that strings are getting an extra layer of " pre- & suffixed to them, further messing up the true data in these values.

It is interesting to compare this event request with the session request that came before it, where custom metrics were set:

{
  "app_key": "087052fde5212020965a8babf8e219f1696fccc5",
  "device_id": "096DEA23-D8E5-4B58-A628-4147D896224A",
  "timestamp": "1655657712164",
  "hour": "12",
  "dow": "0",
  "tz": "-240",
  "sdk_version": "21.11.2",
  "sdk_name": "objc-native-ios",
  "begin_session": "1",
  "metrics": {
    "app_environment": "private",
    "_device": "iPhone13,2",
    "_carrier": "Freedom Mobile",
    "_os": "iOS",
    "app_allowHandoff": "true",
    "_app_version": "3.1.7",
    "app_version": "3.1.7",
    "app_colorfulSites": "true",
    "app_theme": ".deep",
    "app_runCount": "744",
    "app_rating": "5",
    "_os_version": "15.5",
    "id_vendor": "48F63BC8-24CC-405E-8874-1C0C48AE2B2E",
    "app_reviewed": "true",
    "_locale": "en_CA",
    "id_device": "52FDD2FB-875D-4C58-A01E-EA7464453F58",
    "app_premium": "true",
    "app_build": "54",
    "_device_type": "mobile",
    "app_appIcon": "dark",
    "id_owner": "096DEA23-D8E5-4B58-A628-4147D896224A",
    "_resolution": "1170x2532",
    "_density": "@3x",
    "app_masterPasswordCustomer": "true",
    "_store": "iOS",
    "_source_channel": "Direct",
    "_browser": "WebKit",
    "_browser_version": "[webkit]_605.1.15"
  },
  "location": "",
  "checksum256": "1cf4cba831dc10a3aad45365d0c9515a220cba7823913d679ea474beddfc56af"
}

We do NOT see these extra " elements in this request.

Note that on the device this data is being passed into the iOS Countly SDK as a Dictionary of NSString: NSString, and both the keys and values are clean (ie. non-encoded and non-encumbered with excess "" or " values).

I was under the impression that this was not a problem in the Countly Dashboard before the redesign, but perhaps the redesign has merely surfaced a problem that exists in the Countly Cocoa SDK.

Either way, please look into what is going on here - I need clean and reliable data, obviously.

This is a mess:

 

0

Comments

6 comments
  • I am unsure if it is related, but I am also no longer seeing event details show up even though the Event Overview reports many events being recorded and the Request Log clearly sees them coming in.

    0
    Comment actions Permalink
  • Hi Ihunath,

    Thank you for reaching out to us.

    We're investigating it closely with our engineering team.


    Could you please confirm if you are getting any errors on the browser console?

    0
    Comment actions Permalink
  • 0
    Comment actions Permalink
  • Hi Ihunath,

    Thank you for your patience.

    We weren't able to reproduce either of the issues (the main one is about the SDK), and the second one:

    I am unsure if it is related, but I am also no longer seeing event details show up even though the Event Overview reports many events being recorded and the Request Log clearly sees them coming in.

    Is regarding the server.

    To better investigate it, could you please enable logging and then send the SDK side logs?

    Regards,

    0
    Comment actions Permalink
  • (Prefixing this with a note: I discovered that the additional `"` characters at the leading and trailing end of my values were my fault, this has been fixed now)

    As an example, from the Countly cocoa SDK logs on an iPhone device:

    2022-07-03 11:58:09.164551-0400 Spectre[667:68855] [Countly] [Debug] Proceeding on queue...
    2022-07-03 11:58:09.166490-0400 Spectre[667:68855] [Countly] [Debug] Request <0x28091bc40> started:
    [POST] https://countly.spectre.app/i 
    app_key=---&device_id=096DEA23-D8E5-4B58-A628-4147D896224A&timestamp=1656863888776&hour=11&dow=0&tz=-240&sdk_version=21.11.2&sdk_name=objc-native-ios&events=%5B%7B%22dur%22%3A0%2C%22hour%22%3A11%2C%22segmentation%22%3A%7B%22app_container%22%3A%22app%22%2C%22src_line%22%3A%22117%22%2C%22src_file%22%3A%22BaseUsersViewController.swift%22%2C%22items%22%3A%228%22%2C%22value%22%3A%221%22%2C%22src_function%22%3A%22collectionView%28_%3AdidSelectItemAt%3A%29%22%7D%2C%22count%22%3A1%2C%22key%22%3A%22users%20%3Euser%22%2C%22timestamp%22%3A1656863861262%2C%22dow%22%3A0%2C%22sum%22%3A0%7D%2C%7B%22dur%22%3A0.73727691173553467%2C%22hour%22%3A11%2C%22segmentation%22%3A%7B%22app_container%22%3A%22app%22%2C%22src_line%22%3A%22563%22%2C%22src_file%22%3A%22BaseUsersViewController.swift%22%2C%22src_function%22%3A%22attemptBiometrics%28%29%22%2C%22type%22%3A%22biometric%22%2C%22result%22%3A%22success%22%7D%2C%22count%22%3A1%2C%22key%22%3A%22users%20%23user%22%2C%22timestamp%22%3A1656863862000%2C%22dow%22%3A0%2C%22sum%22%3A0%7D%2C%7B%22dur%22%3A0%2C%22hour%22%3A11%2C%22segmentation%22%3A%7B%22app_container%22%3A%22app%22%2C%22biometricLock%22%3A%22true%22%2C%22defaultType%22%3A%22long%22%2C%22algorithm%22%3A%22v3%22%2C%22avatar%22%3A%22avatar-6%22%2C%22src_line%22%3A%22225%22%2C%22sites%22%3A%22138%22%2C%22src_file%22%3A%22Tracker.swift%22%2C%22src_function%22%3A%22login%28user%3A%29%22%2C%22maskPasswords%22%3A%22false%22%2C%22loginType%22%3A%22personal%22%7D%2C%22count%22%3A1%2C%22key%22%3A%22user%20%3Esigned_in%22%2C%22timestamp%22%3A1656863862002%2C%22dow%22%3A0%2C%22sum%22%3A0%7D%2C%7B%22dur%22%3A72.369281053543091%2C%22hour%22%3A11%2C%22segmentation%22%3A%7B%22name%22%3A%22Spectre.MainUsersViewController%22%2C%22segment%22%3A%22iOS%22%7D%2C%22count%22%3A1%2C%22key%22%3A%22%5BCLY%5D_view%22%2C%22timestamp%22%3A1656863789708%2C%22dow%22%3A0%2C%22sum%22%3A0%7D%2C%7B%22dur%22%3A0%2C%22hour%22%3A11%2C%22segmentation%22%3A%7B%22function%22%3A%22viewWillAppear%28_%3A%29%22%2C%22segment%22%3A%22iOS%22%2C%22visit%22%3A1%2C%22name%22%3A%22Spectre.MainSitesViewController%22%2C%22line%22%3A%22103%22%2C%22file%22%3A%22BaseViewController.swift%22%7D%2C%22count%22%3A1%2C%22key%22%3A%22%5BCLY%5D_view%22%2C%22timestamp%22%3A1656863862077%2C%22dow%22%3A0%2C%22sum%22%3A0%7D%5D&checksum256=---
    2022-07-03 11:58:09.174088-0400 Spectre[667:69013] [Countly] [Debug] Result of writing data to file: 1
    2022-07-03 11:58:09.344550-0400 Spectre[667:68855] [Countly] [Debug] Request <0x28091bc40> successfully completed.

    Which URL-decodes into the following value for the `events` query item:

    [{"dur":0,"hour":11,"segmentation":{"app_container":"app","src_line":"117","src_file":"BaseUsersViewController.swift","items":"8","value":"1","src_function":"collectionView(_:didSelectItemAt:)"},"count":1,"key":"users >user","timestamp":1656863861262,"dow":0,"sum":0},{"dur":0.73727691173553467,"hour":11,"segmentation":{"app_container":"app","src_line":"563","src_file":"BaseUsersViewController.swift","src_function":"attemptBiometrics()","type":"biometric","result":"success"},"count":1,"key":"users #user","timestamp":1656863862000,"dow":0,"sum":0},{"dur":0,"hour":11,"segmentation":{"app_container":"app","biometricLock":"true","defaultType":"long","algorithm":"v3","avatar":"avatar-6","src_line":"225","sites":"138","src_file":"Tracker.swift","src_function":"login(user:)","maskPasswords":"false","loginType":"personal"},"count":1,"key":"user >signed_in","timestamp":1656863862002,"dow":0,"sum":0},{"dur":72.369281053543091,"hour":11,"segmentation":{"name":"Spectre.MainUsersViewController","segment":"iOS"},"count":1,"key":"[CLY]_view","timestamp":1656863789708,"dow":0,"sum":0},{"dur":0,"hour":11,"segmentation":{"function":"viewWillAppear(_:)","segment":"iOS","visit":1,"name":"Spectre.MainSitesViewController","line":"103","file":"BaseViewController.swift"},"count":1,"key":"[CLY]_view","timestamp":1656863862077,"dow":0,"sum":0}]

    Here, we can for instance see:

    "key": "user >signed_in"

    On the server side, under `Request Logs`, we find this event shown like this:

    {
    "app_key": "---",
      "device_id": "096DEA23-D8E5-4B58-A628-4147D896224A",
      "timestamp": "1656863888776",
      "hour": "11",
      "dow": "0",
      "tz": "-240",
      "sdk_version": "21.11.2",
      "sdk_name": "objc-native-ios",
      "events": [
        {
          "dur": 0,
          "hour": 11,
          "segmentation": {
            "app_container": "app",
            "src_line": "117",
            "src_file": "BaseUsersViewController.swift",
            "items": "8",
            "value": "1",
            "src_function": "collectionView(_:didSelectItemAt:)"
          },
          "count": 1,
          "key": "users &gt;user",
          "timestamp": 1656863861262,
          "dow": 0,
          "sum": 0
        },
        {
          "dur": 0.7372769117355347,
          "hour": 11,
          "segmentation": {
            "app_container": "app",
            "src_line": "563",
            "src_file": "BaseUsersViewController.swift",
            "src_function": "attemptBiometrics()",
            "type": "biometric",
            "result": "success"
          },
          "count": 1,
          "key": "users #user",
          "timestamp": 1656863862000,
          "dow": 0,
          "sum": 0
        },
        {
          "dur": 0,
          "hour": 11,
          "segmentation": {
            "app_container": "app",
            "biometricLock": "true",
            "defaultType": "long",
            "algorithm": "v3",
            "avatar": "avatar-6",
            "src_line": "225",
            "sites": "138",
            "src_file": "Tracker.swift",
            "src_function": "login(user:)",
            "maskPasswords": "false",
            "loginType": "personal"
          },
          "count": 1,
          "key": "user &gt;signed_in",
          "timestamp": 1656863862002,
          "dow": 0,
          "sum": 0
        },
        {
          "dur": 72.36928105354309,
          "hour": 11,
          "segmentation": {
            "name": "Spectre.MainUsersViewController",
            "segment": "iOS"
          },
          "count": 1,
          "key": "[CLY]_view",
          "timestamp": 1656863789708,
          "dow": 0,
          "sum": 0
        },
        {
          "dur": 0,
          "hour": 11,
          "segmentation": {
            "function": "viewWillAppear(_:)",
            "segment": "iOS",
            "visit": 1,
            "name": "Spectre.MainSitesViewController",
            "line": "103",
            "file": "BaseViewController.swift"
          },
          "count": 1,
          "key": "[CLY]_view",
          "timestamp": 1656863862077,
          "dow": 0,
          "sum": 0
        }
      ],
    "checksum256": "---"
    }

    Where we can see:

    "key": "user &gt;signed_in",

     

    0
    Comment actions Permalink
  • Should I expect a follow-up?

    0
    Comment actions Permalink

Please sign in to leave a comment.