Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"IQ feed works very well, does not have all of the normal interruptions I have grown used to on *******" - Comment from Mark
"And by the way, have to say this. I love the IQFeed software. It's rock solid and it has a really nice API." - Comment from Thomas via RT Chat
"DTN has never given me problems. It is incredibly stable. In fact I've occasionally lost the data feed from Interactive Brokers, but still been able to trade because I'm getting good data from DTN." - Comment from Leighton
"Awesome response, as usual. It is a sincere and refreshing pleasure to do business with DTN, compared to your competition." - Comment from Ryan
"I am keeping IQFeed, much better reliabilty than *******. I may refer a few other people in the office to switch as well." - Comment from Don
"There is no doubt that IQFeed is the best data provider. I am very satisfied with your services. And IQFeed is the only one that I would recommend to my friends. Now, most of them are using your product in China." - Comment from Zhezhe
"Thanks for following up with me. You guys do a great job in tech support." - Comment from Phelps
"Just a thank you for the very helpful and prompt assistance and services. You provided me with noticeably superior service in my setup compared to a couple of other options I had looked at." - Comment from John
"Thanks for all of your help. Great customer service deserves to be recognized which one the reasons I've been a customer of DTN for over 10 years!" - Comment from Stuart
"After all the anxiety I had with my previous data provider it is a relief not to have to worry about data speed and integrity." - Comment from Eamonn
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Latency of Q messages
Author Topic: Latency of Q messages (4 messages, Page 1 of 1)

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 11, 2019 08:00 PM          Msg. 1 of 4
Hi,

I've been trying to optimize my .net socket reader. First, I wrote my own server and pushed 1000 "Q," messages per second to help find any bottlenecks in my code. The mock server writes 1000 dummy Q messages in a loop. I've appended a high resolution time stamp to the messages (using interop GetSystemTimePreciseAsFileTime). I compare this time stamp on the reading side to see how fast I process messages. The writes of 1000 messages only takes a few milliseconds. I noticed that I was unable to keep up with this rate on the read side if I processed the message synchronously. I now push the messages to a BlockingCollection(ConcurrentQueue). Once I did this, my socket mostly reads one or two message per read.

I thought this would help reading the live feed. However, I still see many reads with numerous messages (>50). This is averaging 5, but sometime as many as 100+ messages are pulled in one read. Most of the time, these messages only differ in the Bid/Ask time stamp by a few milliseconds but sometimes are > 50ms (as high as 200 ms) from the first message to the last.

Is IQConnect batching the writes to the socket? Or are you batching the push from your servers? No matter what I do I cannot seem to reduce this.

I should note that I am collecting stats in an array and only dumping one line to the console every 10-15 second. I have timed the stat collection and display and I am confident that it is not skewing the results.

I need to sort this out not because I’m concerned about a few milliseconds but because I can get way behind (1-30 seconds!) in a fast market.

I am running on an AWS EC2 T2.Medium machine (2 cores).

My pings to the IQFeed server are constantly between 45-47 ms.

I am only watching CME products (@ES, @NQ, 5 major currencies)

MAC

DTN_Steve_S
-DTN Guru-
Posts: 2090
Joined: Nov 21, 2005


Posted: Feb 19, 2019 06:38 AM          Msg. 2 of 4
Hello, my apologies for the delay responding to this.

Generally speaking, the markets are naturally very "bursty" in terms of how messages are received from the exchanges. I haven't verified with your specific symbols but you are using some of the most active futures contracts as I'm sure you know. I wouldn't be surprised if your few symbols frequently burst well beyond your 1000 message loop test (unless I'm not understanding your test).

A 30s delay is certainly not normal and figuring out where that data is getting queued will help identify what needs to be fixed. If the data is getting to your machine but queued locally within IQConnect waiting to be delivered to your app, then in the diagnostics app, on the client stats tab, you will see the KBQueued column count up for your app's connection. If this number is zero even when you are seeing delays in your app, then the data isn't able to be delivered to your machine and is being queued on the server. Once we know this information it will be easier to identify what is causing the delay.

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 19, 2019 08:17 AM          Msg. 3 of 4
Thanks for the reply.

I have implemented the latest .net core 3.0 System.IO.Pipelines library. It "queues" and "dequeues" on separate threads while minimizing overallocation. With the help of the SequenceReader, you can access the buffer directly without having to copy to a string and then split. I have yet to test it with my mock server. I'll let you know if it performs better.

I also need to monitor KBQueued in code so I can log these infrequent latency events. Due to some logging changes, I was only able to see these events back in 2017. One of which coincided with an ECB announcement.

I will continue to monitor.

MAC

DTN_Steve_S
-DTN Guru-
Posts: 2090
Joined: Nov 21, 2005


Posted: Feb 19, 2019 08:45 AM          Msg. 4 of 4
The KBQueued value in the diagnostics app is just a display frontend for the clientstats messages on the Admin port from IQFeed via the API.

Turning this on/off and message format is documented here http://www.iqfeed.net/dev/api/docs/AdminviaTCPIP.cfm

Not sure if you were aware of that based on your post so wanted to make sure.
Edited by DTN_Steve_S on Feb 19, 2019 at 08:47 AM
 

 

Time: Mon January 20, 2020 8:59 PM CFBB v1.2.0 0 ms.
© AderSoftware 2002-2003