502 Gateway error in control panel after install and reboot
I installed on a fresh Ubuntu 18.04 install, and successfully installed countly. And works fine until I reboot the server.
Then I get a 502 gateway on the control panel, I have checked system resources and all seems fine there, also no major issues
Comments
Hello to who got the same issue. :)
There is an issue with latest release on Ubuntu 18 & 20 which will be fixed with next minor release, please run these commands after fresh installation:
cd `countly dir`
curl https://raw.githubusercontent.com/Countly/countly-server/master/bin/commands/systemd/install.sh > bin/commands/systemd/install.sh
rm -f /etc/systemd/system/mongod.service
bash bin/scripts/detect.init.sh
Hello,
HTTP 502 means NGINX is running but it can't reach to Countly API. Can you share the output of `countly status`?
I ran into this issue twice now! First time I thought I made a mistake and reinstalled. A few days later it's the same situation. It got so bad that the error.log file from nginx was north of 100GB.
nginx always logs these errors:
"http://127.0.0.1:3001/i?app_key=my-app-key×tamp=1607626035129&hour=19&dow=4&tz=60&sdk_version=20.04.4&sdk_name=java-native-android&begin_session=1&metrics=%7B%22_device%22%3A%22SM-G930F%22%2C%22_os%22%3A%22Android%22%2C%22_os_version%22%3A%228.0.0%22%2C%22_carrier%22%3A%22MEDIONmobile%22%2C%22_resolution%22%3A%221080x1920%22%2C%22_density%22%3A%22XXHDPI%22%2C%22_locale%22%3A%22de_DE%22%2C%22_app_version%22%3A%229.10.3%22%2C%22_store%22%3A%22com.android.vending%22%7D&aid=%7B%22adid%22%3A%22f3c6c3d9-d5dc-4748-8480-646479f09e01%22%7D&device_id=85f7721d8bf90715&checksum=15323f91ba991180d1e21a6d2b988a67881ce300", host: "backend.domain.com"
Countly status
● countly.service - countly-supervisor
Loaded: loaded (/etc/systemd/system/countly.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-12-10 20:19:18 CET; 4min 46s ago
Docs: http://count.ly
Main PID: 958 (supervisord)
Tasks: 23 (limit: 4915)
CGroup: /system.slice/countly.service
├─ 958 /usr/bin/python /usr/bin/supervisord --nodaemon --configuration /root/countly/bin/config/supervisord.conf
├─1463 countly: dashboard node /root/countly/frontend/express/app.
└─1464 countly: api master node /root/countly/api/api.js
Dec 10 20:19:18 v220201139069133430 systemd[1]: Started countly-supervisor.
Dec 10 20:19:19 v220201139069133430 supervisord[958]: 2020-12-10 20:19:19,666 CRIT Set uid to user 0
Dec 10 20:19:19 v220201139069133430 supervisord[958]: 2020-12-10 20:19:19,694 CRIT Server 'unix_http_server' running without any HTTP authentication checking
nginx config:
server {
server_name backend.domain.com;
access_log off;
location = /i {
if ($http_content_type = "text/ping") {
return 404;
}
proxy_pass http://127.0.0.1:3001;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
location ^~ /i/ {
if ($http_content_type = "text/ping") {
return 404;
}
proxy_pass http://127.0.0.1:3001;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
location = /o {
if ($http_content_type = "text/ping") {
return 404;
}
proxy_pass http://127.0.0.1:3001;
}
location ^~ /o/ {
if ($http_content_type = "text/ping") {
return 404;
}
proxy_pass http://127.0.0.1:3001;
}
location / {
if ($http_content_type = "text/ping") {
return 404;
}
proxy_pass http://127.0.0.1:6001;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/backend.domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/backend.domain.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = backend.domain.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
listen [::]:80 ipv6only=on;
server_name backend.domain.com;
return 404; # managed by Certbot
}
Trying to access the dashboard throws a 502 Bad Gateway response and no user data gets logged as it seems. Any idea? There was literally no change this I accessed the dashboard the last time. Maybe a day or two ago.
Thanks!
Hello,
Yes, error.log of NGINX may be filled up by upstream errors when the server get enormous requests while Countly isn't working. You need to disable error logging of NGINX or set log rotate for it.
I can't see the worker process of Countly in the output of `countly status` so can you run `countly restart` command, wait 2 minutes and share the content of `countly-api.log` which is located under `log/` directory in Countly's root directory?
Here's the last 100 lines of countly's log:
tail -n 100 countly-api.log
hourly: '2020.12.10.16',
weekly: 50,
month: '12',
day: '10',
hour: '16'
},
app_user: {
_id: '90d991a044426c0f37fd0c56d0663713a0f06dcf',
uid: '1Ar',
did: '3fc383685f730daf',
cc: 'DE',
cty: 'Sibbesse',
last_req: 'dfaafcaf35cd3722efdddd2767d3183c84242bc21607613261425',
loc: { gps: false, geo: [Object], date: 1606382436692 },
rgn: 'NI',
tz: 60,
av: '9:10:3',
c: 'Congstar',
d: 'SM-G973F',
dnst: 'aXXHDPI',
fs: 1584959895,
la: 'de',
lac: 1607613261,
lo: 'de_DE',
ls: 1607613245,
mt: false,
p: 'Android',
pv: 'a10',
r: '1080x2042',
sc: 463,
src: 'com.android.vending',
sd: 16,
tsd: 31370,
lest: 1607600851,
lbst: 1607613245,
fac: 1584959895000,
hadAnyFatalCrash: 1607600851,
hadAnyNonfatalCrash: 1607600851,
last_sync: 1607613271,
sdk: { name: 'java-native-android', version: '20.04.4' },
lv: 'Catalogs',
lvt: 1603562434,
vc: 0,
dt: 'mobile',
ingested: false,
lsid: '51a70accfdf4141662141f42254773a60769b6801607613245785_1Ar_1607613245785',
data: { events: 2 },
hos: true
},
request_hash: 'dfaafcaf35cd3722efdddd2767d3183c84242bc21607613261425',
preservedEvents: '[{"key":"CB_Stats","count":1,"timestamp":1607600851694,"hour":12,"dow":4,"segmentation":{"Compaction_duration":"5ms"},"sum":0},{"key":"my key","count":1,"timestamp":1607613247979,"hour":16,"dow":4,"segmentation":{"other key":"false"},"sum":0}]',
logging_is_allowed: true,
log_processed: true,
previous_session: '51a70accfdf4141662141f42254773a60769b6801607613245785_1Ar_1607613245785',
previous_session_start: 1607613245,
request_id: 'dfaafcaf35cd3722efdddd2767d3183c84242bc21607613261425_1Ar_1607613261425',
session_duration: 16,
views: [],
viewsNamingMap: {},
response: { code: 200, body: '{"result":"Success"}' }
}
2020-12-10T15:14:32.017Z: ERROR [batcher] Error updating documents BulkWriteError: The dollar ($) prefixed field '$ - Grand fouet#' in 'meta_v2.entered item.$ - Grand fouet#' is not valid for storage.
at UnorderedBulkOperation.handleWriteError (/root/countly/node_modules/mongodb/lib/bulk/common.js:1257:9)
at UnorderedBulkOperation.handleWriteError (/root/countly/node_modules/mongodb/lib/bulk/unordered.js:117:18)
at resultHandler (/root/countly/node_modules/mongodb/lib/bulk/common.js:521:23)
at handler (/root/countly/node_modules/mongodb/lib/core/sdam/topology.js:942:24)
at /root/countly/node_modules/mongodb/lib/cmap/connection_pool.js:350:13
at handleOperationResult (/root/countly/node_modules/mongodb/lib/core/sdam/server.js:558:5)
at MessageStream.messageHandler (/root/countly/node_modules/mongodb/lib/cmap/connection.js:277:5)
at MessageStream.emit (events.js:315:20)
at processIncomingData (/root/countly/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
at MessageStream._write (/root/countly/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
at writeOrBuffer (_stream_writable.js:352:12)
at MessageStream.Writable.write (_stream_writable.js:303:10)
at Socket.ondata (_stream_readable.js:719:22)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:309:12)
at readableAddChunk (_stream_readable.js:284:9) {
driver: true,
code: 52,
writeErrors: [ WriteError { err: [Object] }, WriteError { err: [Object] } ],
result: BulkWriteResult {
result: {
ok: 1,
writeErrors: [Array],
writeConcernErrors: [],
insertedIds: [],
nInserted: 0,
nUpserted: 0,
nMatched: 0,
nModified: 0,
nRemoved: 0,
upserted: []
}
}
}
2020-12-10T15:30:50.378Z: INFO [jobs:manager] Starting job manager in 1596
2020-12-10T19:08:42.142Z: INFO [jobs:manager] Starting job manager in 1551
2020-12-10T19:18:08.629Z: INFO [jobs:manager] Starting job manager in 2047
2020-12-10T19:19:21.922Z: INFO [jobs:manager] Starting job manager in 1464
And nginx config is fine:
sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
I hope this info helps in solving this issue! I don't feel like reinstalling just to run into the issue in a few days again...
Hello,
I don't need last n line of the log file but all it since the last restart as I've mentioned before. By the way, is it a standalone MongoDB server or a sharded cluster? Can you provide `log/countly-api.log` & `api/config.js` file, please?
Feel free to contact me via our Countly Community Slack or by email(kk@count.ly) to send files.
Please sign in to leave a comment.