LS adapter gets stops automatically get function only after restart the LS

[rvkvino]

Hello,
I have already added the log configuration as your guidance. And I have downloaded log and searched like Current delay for tasks scheduled on thread pool SERVER but couldn't found anything like this. IS there any other log I need to enable in config.

[giuseppe.corti]

So, with subscriptions loggers at DEBUG level you should be able to follow the entire flow relating to each Item, starting from the subscription request to the adapter and all the updates received by the server.

Message like this refer to the subscription request forwarded to the Data Adapter: "Subscribing to item <Item> on <AdapterSet.DataAdapter>"

Then you should looking for the replies of the adapter with messages like these:
"INCOMING SNAPSHOT for <Item> on <AdapterSet.DataAdapter> --> ... "
"INCOMING END OF SNAPSHOT NOTIFY for <Item> on <AdapterSet.DataAdapter>"
"INCOMING DATA for <Item> on <AdapterSet.DataAdapter> ... "

With that information you can be sure whether the updates are flowing regularly from the adapter to the server. If so, we'll need to focus on the client side.

Regards,
Giuseppe

[rvkvino]

Hello,
As per your instruction I have checked the log, now my system didn't work got stopped the pushing rates so suddenly checked log. Please check this log I have attached,
07-Dec-20 04:37:21,331|INFO |L.requests |PUMP POOLED THREAD 1 |Closed session S91a4ff89e6ee459fM656T2306451 with internal cause code: 38/2 (Connection interrupted by client).
07-Dec-20 04:37:23,715|INFO |streamerMonitorText|Timer-1 |Total threads = 92, Total heap = 62914560 (free = 43428880), Sessions = 3 (max = 13, refused = 0), New sessions = [+2, -2], Connections = 3 (max = 18), New connections = [+4, -4], In-pool threads = 32, Active threads = 0, Available threads = 32, Queued tasks = 0, Pool queue wait = 0, NIO write queue = 0, NIO write queue wait = 0, NIO write selectors = 4, NIO total selectors = 16, Subscribed items = 12 (for 36 subscriptions), Inbound throughput = 0 updates/s (pre-filtered = 0), Outbound throughput = 0 updates/s (0.73 kbit/s, max = 50.97), Lost updates = 0 (total = 0), Total bytes sent = 26829363, Client messages throughput = 0 msgs/s (0 kbit/s, max = 0), Total messages handled = 0, Extra sleep = 0, Notify delay = 0
07-Dec-20 04:37:25,159|INFO |L.requests |SERVER POOLED THREAD 8 |Serving request: /lightstreamer/create_session.js --> LS_op2=create&LS_phase=8801&LS_cause=new.api&LS_po lling=true&LS_polling_millis=0&LS_idle_millis=0&LS _cid=pcYgxn8m8 feOojyA1T661j3g2.pz479h7m&LS_adapter_set=WLSTOCKLI ST_REMOTE&LS_container=lsc& on "Lightstreamer HTTP Server" from 1.22.25.136:58569
07-Dec-20 04:37:25,160|INFO |L.requests |SERVER POOLED THREAD 8 |Starting new session: S3e7c8f24186ea3cdM656T3725160 (javascript_client 7.0.4 build 1721) on "Lightstreamer HTTP Server" from 1.22.25.136:58569
07-Dec-20 04:37:25,450|INFO |L.requests |SERVER POOLED THREAD 8 |Serving request: bind_session -> LS_session=S3e7c8f24186ea3cdM656T3725160&LS_phase= 8803&LS_cause=loop1&LS_container=lsc& on "Lightstreamer HTTP Server" from 1.22.25.136:58570
07-Dec-20 04:37:25,450|INFO |L.requests |SERVER POOLED THREAD 8 |Attaching session: S3e7c8f24186ea3cdM656T3725160 on "Lightstreamer HTTP Server" from 1.22.25.136:58570
07-Dec-20 04:37:25,455|INFO |L.requests |SERVER POOLED THREAD 8 |Serving request: control -> LS_mode=COMMAND&LS_id=kgnbull&LS_schema=key%20comm and%20desc%20bid%20ask%20high%20low%20order%20deli very&LS_data_adapter=WLQUOTE_ADAPTER&LS_snapshot=t rue&LS_table=1&LS_req_phase=2&LS_win_phase=1&LS_op =add&LS_session=S3e7c8f24186ea3cdM656T3725160& on "Lightstreamer HTTP Server" from 1.22.25.136:58570
07-Dec-20 04:37:25,455|INFO |L.requests |SERVER POOLED THREAD 8 |Controlling session: S3e7c8f24186ea3cdM656T3725160 on "Lightstreamer HTTP Server" from 1.22.25.136:58570
07-Dec-20 04:37:25,457|INFO |L.requests |SERVER POOLED THREAD 4 |Controlling session: S3e7c8f24186ea3cdM656T3725160 on "Lightstreamer HTTP Server" from 1.22.25.136:58570
07-Dec-20 04:37:46,992|INFO |L.requests |PUMP POOLED THREAD 4 |Closed session S3e7c8f24186ea3cdM656T3725160 with internal cause code: 38/2 (Connection interrupted by client).
07-Dec-20 04:37:51,030|INFO |L.requests |SERVER POOLED THREAD 1 |Starting new session: S2c036881423e770bM656T3751030 (javascript_client 7.0.4 build 1721) on "Lightstreamer HTTP Server" from 1.22.25.136:58610
07-Dec-20 04:37:51,992|DEBUG|L.subscriptions |SERVER POOLED THREAD 4 |Unsubscribing from item kgnbull on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:37:51,992|DEBUG|L.subscriptions |SERVER POOLED THREAD 1 |Unsubscribing from item Marketstatus-kgnbull on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:37:56,468|INFO |L.requests |SERVER POOLED THREAD 1 |Serving request: /lightstreamer/bind_session.js --> LS_session=S2c036881423e770bM656T3751030&LS_phase= 6104&LS_cause=ws.error.failure.wsc.unknown&LS_cont ent_length=4000000&LS_container=lsc& on "Lightstreamer HTTP Server" from 1.22.25.136:58610
07-Dec-20 04:37:56,468|INFO |L.requests |SERVER POOLED THREAD 1 |Attaching session: S2c036881423e770bM656T3751030 on "Lightstreamer HTTP Server" from 1.22.25.136:58610
07-Dec-20 04:37:56,796|INFO |L.requests |SERVER POOLED THREAD 4 |Serving request: /lightstreamer/control.js --> LS_mode=COMMAND&LS_id=kgnbull&LS_schema=key%20comm and%20desc%20bid%20ask%20high%20low%20order%20deli very&LS_data_adapter=WLQUOTE_ADAPTER&LS_snapshot=t rue&LS_table=1&LS_req_phase=2&LS_win_phase=1&LS_op =add&LS_unique=1&LS_session=S2c036881423e770bM656T 3751030& on "Lightstreamer HTTP Server" from 1.22.25.136:58615
07-Dec-20 04:37:56,797|INFO |L.requests |SERVER POOLED THREAD 4 |Controlling session: S2c036881423e770bM656T3751030 on "Lightstreamer HTTP Server" from 1.22.25.136:58615
07-Dec-20 04:37:56,798|DEBUG|L.subscriptions |SERVER POOLED THREAD 1 |Subscribing to item kgnbull on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:37:57,105|DEBUG|L.subscriptions |SERVER POOLED THREAD 4 |Subscribing to item Marketstatus-kgnbull on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:38:02,614|INFO |L.requests |SERVER POOLED THREAD 1 |Serving request: /lightstreamer/control.js --> LS_op=destroy&LS_session=S84a38836513b0243M656T275 6818&LS_cause=new.timeout.10.1&LS_unique=84& on "Lightstreamer HTTP Server" from 169.149.227.120:35294
07-Dec-20 04:38:02,618|INFO |L.requests |SERVER POOLED THREAD 1 |Serving request: /lightstreamer/create_session.js --> LS_op2=create&LS_phase=42506&LS_cause=create.timeo ut&LS_polling=true&LS_polling_millis=0&LS_idle_mil lis=0&LS_cid=pcYgxn8m8 feOojyA1T661k3g2.pz479h8o&LS_adapter_set=WLSTOCKLI ST_REMOTE&LS_container=lsc& on "Lightstreamer HTTP Server" from 169.149.203.46:35296
07-Dec-20 04:38:02,622|INFO |L.requests |SERVER POOLED THREAD 1 |Starting new session: S10bbf993d030c7feM656T3802622 (javascript_client 7.0.5 build 1722) on "Lightstreamer HTTP Server" from 169.149.203.46:35296
07-Dec-20 04:38:03,273|INFO |L.requests |SERVER POOLED THREAD 1 |Serving request: /lightstreamer/bind_session.js --> LS_session=S10bbf993d030c7feM656T3802622&LS_phase= 42508&LS_cause=loop1&LS_keepalive_millis=5000&LS_c ontent_length=4000000&LS_container=lsc& on "Lightstreamer HTTP Server" from 169.149.203.46:35296
07-Dec-20 04:38:03,273|INFO |L.requests |SERVER POOLED THREAD 1 |Attaching session: S10bbf993d030c7feM656T3802622 on "Lightstreamer HTTP Server" from 169.149.203.46:35296
07-Dec-20 04:38:03,279|INFO |L.requests |SERVER POOLED THREAD 1 |Serving request: /lightstreamer/control.js --> LS_mode=MERGE&LS_id=GOLDFEB%20MGOLD-C%20MGOLD-F%20SILVERMAR%20SILVER-F%20MSILVER-C%20MSILVER-F%20SPOT-GOLD%20SPOT-SILVER%20SPOT-INR&LS_schema=desc%20bid%20ask%20high%20low%20ltp& LS_data_adapter=WLQUOTE_ADAPTER&LS_snapshot=true&L S_table=56&LS_req_phase=57&LS_win_phase=57&LS_op=a dd&LS_unique=85&LS_session=S10bbf993d030c7feM656T3 802622& on "Lightstreamer HTTP Server" from 169.149.227.120:35294
07-Dec-20 04:38:03,279|INFO |L.requests |SERVER POOLED THREAD 1 |Controlling session: S10bbf993d030c7feM656T3802622 on "Lightstreamer HTTP Server" from 169.149.227.120:35294
07-Dec-20 04:38:03,280|DEBUG|L.subscriptions |SERVER POOLED THREAD 3 |Subscribing to item MGOLD-F on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:38:03,280|DEBUG|L.subscriptions |SERVER POOLED THREAD 8 |Subscribing to item MSILVER-C on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:38:03,280|DEBUG|L.subscriptions |SERVER POOLED THREAD 4 |Subscribing to item SILVER-F on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:38:03,659|INFO |L.requests |SERVER POOLED THREAD 4 |Serving request: /lightstreamer/control.js --> LS_mode=MERGE&LS_id=GOLDFEB%20MGOLD-C%20MGOLD-F%20SILVERMAR%20SILVER-F%20MSILVER-C%20MSILVER-F%20SPOT-GOLD%20SPOT-SILVER%20SPOT-INR&LS_schema=desc%20bid%20ask%20high%20low%20ltp& LS_data_adapter=WLQUOTE_ADAPTER&LS_snapshot=true&L S_table=57&LS_req_phase=58&LS_win_phase=57&LS_op=a dd&LS_unique=86&LS_session=S10bbf993d030c7feM656T3 802622& on "Lightstreamer HTTP Server" from 169.149.227.120:35294
07-Dec-20 04:38:03,660|INFO |L.requests |SERVER POOLED THREAD 4 |Controlling session: S10bbf993d030c7feM656T3802622 on "Lightstreamer HTTP Server" from 169.149.227.120:35294
07-Dec-20 04:38:11,243|INFO |L.requests |PUMP POOLED THREAD 4 |Closed session S10bbf993d030c7feM656T3802622 with internal cause code: 38/5 (Connection interrupted by client).
07-Dec-20 04:38:16,243|DEBUG|L.subscriptions |SERVER POOLED THREAD 3 |Unsubscribing from item MGOLD-F on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:38:16,244|DEBUG|L.subscriptions |SERVER POOLED THREAD 3 |Unsubscribing from item SILVER-F on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:38:16,244|DEBUG|L.subscriptions |SERVER POOLED THREAD 5 |Unsubscribing from item MSILVER-C on WLSTOCKLIST_REMOTE.WLQUOTE_ADAPTER
07-Dec-20 04:39:23,762|INFO |streamerMonitorText|Timer-1 |Total threads = 93, Total heap = 62914560 (free = 41810672), Sessions = 4 (max = 13, refused = 0), New sessions = [+3, -2], Connections = 4 (max = 18), New connections = [+10, -9], In-pool threads = 32, Active threads = 0, Available threads = 32, Queued tasks = 0, Pool queue wait = 0, NIO write queue = 0, NIO write queue wait = 0, NIO write selectors = 4, NIO total selectors = 16, Subscribed items = 12 (for 36 subscriptions), Inbound throughput = 0 updates/s (pre-filtered = 0), Outbound throughput = 0 updates/s (0.98 kbit/s, max = 50.97), Lost updates = 0 (total = 0), Total bytes sent = 26839721, Client messages throughput = 0 msgs/s (0 kbit/s, max = 0), Total messages handled = 0, Extra sleep = 0, Notify delay = 0

I have uploaded full log file in this link http://post.nammauzhavan.com/logs/Lightstreamer.log

[giuseppe.corti]

As you can see from the log, the messages reporting received updates from the Data Adapter ("INCOMING DATA for ... ") disappear completely from 03:50:01,925; but the rest of the server activity continues regularly.
New client sessions are processed properly, as well as subscriptions and unsubscription requests; just no new data from the Data Adapter were received.

Please you should double check your Data Adapter code and your back-end source of the data to try to figure out where could be addressed the stoppage.

Regards,
Giuseppe

[rvkvino]

But when I restart the LS and adapter this getting run normally. This is too hard to find out the issue.

[giuseppe.corti]

Indeed, in this case I would start the investigation from the initialization operations performed in the Adapters, since they seem to fix the issue.

Maybe you have a forever loop that once interrupted is not re-started?
Or a connection to an external service that as well is not restored following a failure?

[rvkvino]

Hello,
I have attached my full source code in this link http://post.nammauzhavan.com/logs/src.7z Could you please check and help me on this.