Could you explain more about _stockGenerators, _snapshotQueue, _snapshotSender?

[tuongkha]

Dear Dario,
Let's me explain : Stock Exchange send data file (binary file) to the Broker company, this file has information for all stocks (stock symbol, stock name, price, volume, ceiling, floor, percent change,...). If some stock has information changed, they send this file again to the Broker company. So every 5-10 seconds, I need read this file again to get values for all stocks, because i don't know which stock modified.

Exactly I need to read from file :
- Get values for all stocks : stock symbol, stock name, stock type (bond or common stock) ceiling price, floor price, open price, split (means stock has divident or not), benifit (means stock has action coporation), prior Close price, last price, last volume, last values, percent change, highest price, lowest price, Best 1 bid, Best 1 bid volume, Best 2 bid, Best 2 bid volume, Best 3 bid, Best 3 bid volume, Best 1 Offer, Best 1 Offer volume, Best 2 Offer, Best 2 Offer volume, Best 3 Offer, Best 3 Offer volume.

for each stock read, I want to get values for all fields :
-------
last price, last volume, last values, percent change, highest price, lowest price, Best 1 bid, Best 1 bid volume, Best 2 bid, Best 2 bid volume, Best 3 bid, Best 3 bid volume, Best 1 Offer, Best 1 Offer volume, Best 2 Offer, Best 2 Offer volume, Best 3 Offer, Best 3 Offer volume

Here struct binary file, it has contain 504 stocks :
--------
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Struct_Security
{
//221: SECURITY.DAT
//Ma CK dang so (2 bytes)
public short StockNo;

//Ma CK dang chuoi (8 bytes)
[MarshalAs(UnmanagedType.ByValTStr, SizeConst =8)]
public string StockSymbol;

//Loai CK (S:Co phieu, D:Trai phieu, U:Chung chi quy) (1 byte)
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string StockType;

//Gia tran (4 bytes)
public int Ceiling;
//Gia san (4 bytes)
public int Floor;

//
public double BigLotValue;


//Ten day du cua CK
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 25)]
public string SecurityName;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string SectorNo;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
//
public string Designated;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
//CK bi ngung GD (Null: GD binh thuong, S: Bi ngung GD)
public string Suspension;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string Delist;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string HaltResumeFlag;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string Split;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string Benefit;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string Meeting;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string Notice;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string ClientIDRequired;
public short CouponRate;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 6)]
public string IssueDate;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 6)]
public string MatureDate;

public int AvrPrice;

public short ParValue;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
public string SDCFlag;

public int PriorClosePrice;

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 6)]
public string PriorCloseDate;

public int ProjectOpen;

public int OpenPrice;
public int Last;
public int LastVol;
public double LastVal;
public int Highest;
public int Lowest;

public double Totalshares;
public double TotalValue;
public short AccumulateDeal;
public short BigDeal;
public int BigVol;
public double BigVal;
public short OddDeal;
public int OddVol;
public double OddVal;

//Single
public int Best1Bid;
public int Best1BidVolume;
//Single
public int Best2Bid;
public int Best2BidVolume;
//Single
public int Best3Bid;
public int Best3BidVolume;

//Single
public int Best1Offer;
public int Best1OfferVolume;
//Single
public int Best2Offer;
public int Best2OfferVolume;
//Single
public int Best3Offer;
public int Best3OfferVolume;

public short BoardLost;

public static System.Collections.ArrayList ReaderBlock2Array(BinaryReader br)
{
System.Collections.ArrayList arr = new System.Collections.ArrayList();

br.BaseStream.Position = 0;
while (br.BaseStream.Position < br.BaseStream.Length)
{
byte[] buff = br.ReadBytes(TSSize.Size);
GCHandle handle = GCHandle.Alloc(buff, GCHandleType.Pinned);
Struct_Security s = (Struct_Security)Marshal.PtrToStructure(handle.Add rOfPinnedObject(), typeof(Struct_Security));

handle.Free();
arr.Add(s);
}

return arr;
}
}

internal sealed class TSSize
{
public static int _size;

static TSSize()
{
_size = Marshal.SizeOf(typeof(Struct_Security));

}

public static int Size
{
get
{
return _size;
}
}
}


Can u help me?

[DarioCrivelli]

You should definitely rewrite the ExternalFeedSimulator class and get rid of the included ExternalFeedProducer class.
The Data Adapter should receive all data and detect the changed data in order to send them to Lightstreamer.

However, to keep the Data Adapter simple for demo purpose, the Data Adapter can also forward all data to Lightstreamer, which will be responsible for detecting the changed data and only send them to the client. Please consider that this is not recommended in a production scenario.
The simplest structure might be:

constructor:
   create a map to hold all snapshots
      (let's create an IDictionary and call it _snaps)

method Start:
   start a new thread with the following behaviour:
      every 5 seconds:
         read the file
         for each stock:
            create an IDictionary to store all field name/value pairs
            associate the IDictionary to the stock name in _snaps
               (the previous association, if any, should be garbage collected)
            call onEvent on the listener and send the IDictionary 

method SendCurrentValues:
   get the IDictionary associated to the requested item name in _snap
   if found:
      call onEvent on the listener and send the IDictionary
   else
      call onEvent on the listener with an empty IDictionary

[tuongkha]

Dear Dario,
Thanks for your help, I tried follow by your instruction, but after every 5 second when the data file changed, the Data Adapter didn't receive all data and detect the changed data in order to send them to Lightstreamer. i don't know what are my fault? Can u help me?

Here my changed :
--- "ExternalFeed.cs" file, ExternalFeedSimulator class ----

constructor :
---

private IDictionary _stockGenerators;

public ExternalFeedSimulator {

//create an IDictionary
_stockGenerators= new Hashtable();

ISecurityStructReader reader;
reader = new ReadSecurityStruct("C:\\TEMP\\BACKUP14\\SECURITY.D AT");

nStock = 0;

if (reader.Open())
{
arr = reader.Read();
nStock = arr.Count;

m_stockSymbol = new string[nStock];
m_openprices = new double[nStock];
m_stockNames = new string[nStock];
...

for (int i=0; i< nStock ;i++)
{
Struct_Security item = (Struct_Security)arr[i];
m_openprices[i]= item.OpenPrice;
...
}
}
reader.Close();
}


public void Start()
{
//Start a new thread
lock (this)
{
if (_snapshotSender != null) return;

_snapshotSender = new Thread(new ThreadStart(Run));
_snapshotSender.Start();
}
}


public void Run()
{
do
{
//every 5 seconds
int waitMillis= ComputeNextWaitTime();
Thread.Sleep(waitMillis);

//read the file ReadBinaryFile();

//for each stock for (int i=0; i<= nStock; i++)
{
//create an IDictionary to store all field name/value pairs associate the IDictionary to the stock name in _snaps
string itemName= "item" + (i + 1);
ExternalFeedProducer myProducer = new ExternalFeedProducer(itemName, m_openprices[i], m_refprices[i], m_minprices[i], m_maxprices[i], m_stockSymbol[i], m_stockNames[i], m_ceiling[i], m_floor[i], m_bid1[i], m_bid2[i], m_bid3[i], m_bid1vol[i], m_bid2vol[i], m_bid3vol[i], m_ask1[i], m_ask2[i], m_ask3[i], m_ask1vol[1], m_ask2vol[i], m_ask3vol[i], m_last[i], m_lastVal[i], m_lastVol[i], m_projectOpen[i]);

_stockGenerators[itemName]= myProducer;

//call onEvent on the listener and send the IDictionary
_listener.OnEvent(myProducer.GetItemName(), myProducer.GetCurrentValues(true), true);

}

} while (true);
}


public int ComputeNextWaitTime()
{
lock (this)
{
return 5000;
}
}


public void SendCurrentValues(string itemName)
{
//get the IDictionary associated to the requested item name in _snap
ExternalFeedProducer myProducer= (ExternalFeedProducer) _stockGenerators[itemName];

//call onEvent on the listener with an empty IDictionary, if not found.
if (myProducer == null) return;

//if found, call onEvent on the listener and send the IDictionary
_listener.OnEvent(myProducer.GetItemName(), myProducer.GetCurrentValues(true), true);

}

[DarioCrivelli]

The code seems correct,
provided that "ReadBinaryFile" populates your "m_" arrays in the same way as the constructor does
and that "GetCurrentValues" generates an IDictionary.
Any unwanted behaviour (for instance, the Data Adapter not receiving all read data) should be inspected with a debugger.
Note that only Lightstreamer Server can check for unchanged data. This means that your Remote Adapter would still forward all updates to Lightstreamer Server.
Dario

[tuongkha]

Dear Dario,

Here my ReadBinaryFile() method and GetCurrentValues :

--- "ExternalFeed.cs", ExternalFeedSimulator class---

public class ExternalFeedSimulator
{
private double [] m_refprices = null;
private double [] m_openprices = null;

private double [] m_minprices;
private double [] m_maxprices;
...


public ExternalFeedSimulator()
{

_stockGenerators= new Hashtable();
...
}

//Get data stock from outside, read "security.dat" file
public void ReadBinaryFile()
{
ISecurityStructReader reader;
reader = new ReadSecurityStruct("C:\\TEMP\\BACKUP14\\SECURITY.D AT");

nStock = 0;

if (reader.Open())
{
arr = reader.Read();

nStock = arr.Count;
m_stockSymbol = new string[nStock];
m_openprices = new double[nStock];

for (int i=0; i< nStock ;i++)
{
Struct_Security item = (Struct_Security)arr[i];

m_openprices[i]= item.OpenPrice;
m_stockSymbol[i]= item.StockSymbol;
...
}
}
reader.Close();
}

---- "ExternalFeed.cs" file, ExternalFeedProducer class ---

public void Start() {
lock (this) {
if (_thread != null) return;

_thread = new Thread(new ThreadStart(Run));
_thread.Start();
}
}

private void Run() {
do {
int waitMillis= ComputeNextWaitTime();
Thread.Sleep(waitMillis);

if (_listener != null)
{
_listener.OnEvent(_itemName, GetCurrentValues(false), false);
}

} while (true);
}

//How about this method? Is it correct?
public IDictionary GetCurrentValues(bool fullData) {
lock (this) {
IDictionary eventData = new Hashtable();

eventData["ceiling"] = _ceiling.ToString("#,##0.##");
eventData["floor"] = _floor.ToString("#,##0.##");
...
}
}

How about Start(), Run() method in "ExternalFeedProducer" class? i need remove ? Because it has error in StockListAdapter :

---
Unhandled exception : System.IndexOutOfRangeException : index was outside the bounds of the array at Lightstreamer.Adapters.Data.StockList.ExternalFeed Simulator.Run()

----

Quotes :
"Any unwanted behaviour (for instance, the Data Adapter not receiving all read data) should be inspected with a debugger" --> i don't understand this, u mean if i config in "lightstreamer_log_conf.xml" file and set priority value = "DEBUG" :

<category name="LightstreamerLogger.pump" class="org.apache.log4j.Logger">
<priority value="DEBUG" />
</category>

the Data Adapter not receiving all read data after every 5 seconds?

and How to check or to recognize my Remote Adapter would still forward all updates to Lightstreamer Server?

Can u help me?

[DarioCrivelli]

To my human eye, the "ReadBinaryFile" and "GetCurrentValues" methods seem correct.
I also confirm that "Start" and "Run" of the "ExternalFeedProducer" class should be removed, as now only the main thread in "ExternalFeedSimulator" is needed.

Hence, if you observe that the Data Adapter does not call onEvent as expected, you should try to setup a code debugger and trace it step by step.
Alternatively, if you want to log the data flow sent by your "ExternalFeedSimulator" class, you have to add custom log lines, for instance in the "onEvent" implementation in "StockList.cs".

Note that, in this method, only the events pertaining to stocks that are currently subscribed to are forwarded to Lightstreamer.
You can have the Remote Server log the update events really sent to Lightstreamer by setting as DEBUG the "Lightstreamer.DotNet.Server.RequestReply" logging category. The configuration of the Remote Server log, in the included deployment example, is in the "DotNetServer.exe.config" file beside the "DotNetServer.exe" executable; the configured log file is "DotNetServer.log".

The Server logging configuration file, "lightstreamer_log_conf.xml", can remain unchanged, for the moment.

Dario

[tuongkha]

Thanks Dario very much, now it seem ok. but in html file, javascript code, when data stock changed, function updateItem(item, updateInfo) doesn't work.

Last time, i have a question for this issue, and u have to answer the question, but i followed by your instruction, i didn't have successful. I don't know, what are there steps i wrong? Please help me again.

Here my changed

-----"default.html" file --------

<script>
for (var i = 1; i <= 30; i++)
{
var suff = (i % 2 == 1) ? "A" : "B";

document.write('<tr class="lscold'+suff+'">');
document.write('<td>&nbsp;</td>');
document.write('<td class="stockname'+suff+'"><div source="lightstreamer" table="list" item="'+i+'" field="stock_symbol"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="ref_price"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="ceiling"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="floor"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="open_price"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="max"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="min"> - </div></td>');
document.write('<td>&nbsp;</td>');
document.write('<td>&nbsp;</td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="close_price"> - </div></td>');
document.write(' <td><div source="lightstreamer" table="list" item="'+i+'" field="close_vol"> - </div></td>');
document.write('<td>&nbsp;</td>');
document.write('<td>&nbsp;</td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="bid3"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="bid3vol"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="bid2"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="bid2vol"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="bid1"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="bid1vol"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="last_price"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="lastVol"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="pct_change"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="ask1"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="ask1vol"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="ask2"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="ask2vol"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="ask3"> - </div></td>');
document.write('<td><div source="lightstreamer" table="list" item="'+i+'" field="ask3vol"> - </div></td>');
document.write('</tr>');
}
</script>


//////////////////////////Event handlers

var redColor = "#f8b87a";
var greenColor = "lightgreen";

function updateItem(item, updateInfo) {
if (updateInfo == null)
{
return;
}
if (updateInfo.isValueChanged("pct_change")) {
var val = updateInfo.getNewValue("pct_change"); if (val.indexOf("-") > -1) {
updateInfo.addField(216,imgDown);
} else {
updateInfo.addField(216,imgUp);
}
}

var oldLast = updateInfo.getOldValue("last_price");
var newColor;
if (oldLast == null) { //first update for this item
updateInfo.addField(214,greenColor,true); //no fade for snapshot
if (doFade) {
updateInfo.addField(215,"OFF",true);
}
} else if (updateInfo.isValueChanged("last_price")) {
//at least second update
if (oldLast > updateInfo.getNewValue("last_price")) {
updateInfo.addField(214,redColor,true);
} else {
updateInfo.addField(214,greenColor,true);
}
if (doFade) {
updateInfo.addField(215,"ON",true);
}
}
}


function formatValues(item, itemUpdate) {
if (itemUpdate == null) {
return;
}

if (doFade) {
if (itemUpdate.getServerValue(215) == "ON") {
itemUpdate.setHotToColdTime(300);
}
}
itemUpdate.setHotTime(600);

if (itemUpdate.getServerValue(13) == "inactive") {
carryBackUnchanged(itemUpdate);
itemUpdate.setRowAttribute("#808080","#808080","co lor");
} else {
if (itemUpdate.getFormattedValue(13) != null) {
carryBackUnchanged(itemUpdate);
itemUpdate.setRowAttribute("#000000","#000000","co lor");
itemUpdate.setAttribute(12,"#000080","#000080","co lor");
}

//choose the backgroundColor
var backC = (item % 2 == 1) ? "#eeeeee" : "#ddddee";
var backH = itemUpdate.getServerValue(214); itemUpdate.setRowAttribute(backH,backC,"background Color");

//choose the "change" field stylesheet
var newChng;
if ((newChng = itemUpdate.getFormattedValue(3)) != null) {
var hotTxtCol = (newChng.charAt(0) == '-') ? "#dd0000" : "#009900";
itemUpdate.setAttribute(3,"black",hotTxtCol,"color ");
itemUpdate.setAttribute(3,"bold","bold","fontWeigh t");
}

itemUpdate.setAttribute(12,backC,backC,"background Color");
}

//format the timestamp
var time = itemUpdate.getFormattedValue(2);
if (time != null) {
time = formatTime(time);
itemUpdate.setFormattedValue(2,time);
}

//format the "number" fields
for (var i = 1; i <= 11; i++) {
if (i == 2 || i == 4 || i == 7) {
continue; //"string" fields
}

var newValue = itemUpdate.getFormattedValue(i);
if (newValue == null) continue;

var formattedVal = formatDecimal(newValue, 2, true);

if (i == 3) {
if (formattedVal > 0) {
formattedVal = "+" + formattedVal;
}
formattedVal += "%";
}
itemUpdate.setFormattedValue(i,formattedVal);

}
}


////////////////Global var declaration
var group = ["item1", "item2", "item3", "item4","item5", "item6", "item7", "item8","item9", "item10", "item11", "item12","item13", "item14", "item15", "item16", "item17", "item18", "item19", "item20", "item21", "item22", "item23", "item24", "item25", "item26", "item27", "item28", "item29", "item30"];
var schema = ["last_price", "time", "pct_change","bid1", "bid2", "bid3", "bid1vol", "bid2vol", "bid3vol", "ask1","ask2", "ask3", "ask1vol", "ask2vol", "ask3vol", "min", "max","ref_price", "open_price", "stock_name", "stock_symbol", "ceiling", "floor", "lastVal", "lastVol", "close_price", "close_vol"];
var newTable = new OverwriteTable(group, schema,"MERGE");

----------
Now i want to changed, if all of items :
lastVal, lastVol, last_price, bid1, bid2, bid3, ask1, ask2, ask3, bid1vol, bid2vol, bid3vol, ask1vol, ask2vol, ask3vol, pct_change

has values modified, it will be fill by color (if current value < old_value, fill by red color, else fill by green color)


Please help me again.