R Client for lightstreamer

[toolbox]

Below summary is only valid based on following environment:
Server:
- Lightstreamer server positioned at IG premises.
- Server version less then 6.1
Client:
- old text protocol (not TLCP)
- OS - Linux
- Programming environment - "R".

################################################
# Summary
################################################
I am working on setting up an "automated trading system" completely
in the programming environment "R". As first step I created a download
environment based on REST API. During the test phase it was obviously that
the robustness was not acceptable, missing 5%-30% downloaded OHLC lines.
The resolution was set to 5 minutes candlesticks. The 5 min resolution means that the REST API
sends out "historical data" that is not meant change dynamically.

The reason for the missing lines was that IG sends out values "NULL" in their datalines without
any explanation of how to handle those "NULL" values. Due to that my client is
programmed to delete any line that contains minimum one "NULL".

To continue with my project I initiated the task to replace REST API with Streaming API.
(same bank though). To benchmark increase in quality of downloaded lines, the aim was
to set streaming API to 5 minutes candlesticks. You should be able to subscribe on "5 min candlestick" but I encountered 2 problems:

1) The streaming API from IG sends out all ticks, despite if you subscribe on "5 min candlesticks",
you do get the timestamps per each 5 minutes though.

2) Since you need to work with the stream as source, you need to let each line,
accumulate the changes until you "freeze" the line to be historical and will never change.
This is of course not valid for the last timestamp (since it will keep changing).

...further on tried to set the clients "LS_requested_max_frequency" in order to reduce the
stream intensity but the change did not have any impact on the stream at all.

################################################## ####################
# The basics of building the client in "R", without technical details.
################################################## ####################
1) Open up a stream connection using create_session.txt, that continuously saves to file (e.g. stream1.txt)
2) Request for a subscription using control.txt
3) Poll data from stream connection using bind_session.txt
4) Script to transform the stream1.txt to 5 minutes OHLC data.
5) Compare result between Streaming API <-> REST API.
6) Develop function to steer the stream (start, stop, etc).

################################################## ####################
# Result and needed improvements
################################################## ####################
I have managed to reduce missing OHLC data lines from REST API (5% - 30%) to Streaming API (roughly <5%). These <5% includes potential bottlenecks, such as programming language, OS, network, scheduler, modeling, architecture, etc. Benchmarking between REST API and Streaming API is currently not possible since Streaming API is "self-healing". If a line is missing, next production of line will include all historical lines, resulting in total success of 100%. From a "automated trading system" -persective, if the line is not there at a specific time, the loss for that line is 100%, and next decision is to be taken next successful line. To stresstest the system i now run 1 minute resolution.

The biggest back-draws right now is:
1) If the stream stops (hangs or breaks), the system stops to have updated download data.
2) Currently the transformation function uses a screen-shot of complete accumulated data,
at 5 p.m. it screen-shots the whole day.