Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)

"I was on the phone with a friend who uses CQG and right after the Fed announcement, CQG was as much as 30 seconds behind DTN.IQ. Some quotes were off by as much as 15-18 cents. Your feed never missed a beat." - Comment from Roger
"There is no doubt that IQFeed is the best data provider. I am very satisfied with your services. And IQFeed is the only one that I would recommend to my friends. Now, most of them are using your product in China." - Comment from Zhezhe
"Awesome response, as usual. It is a sincere and refreshing pleasure to do business with DTN, compared to your competition." - Comment from Ryan
"I like you guys better than *******...much more stable and a whole lot fewer issues." - Comment from Philip
"Interactive Brokers tick data was inconsistent, so I have switched to using DTN exclusively. It is great to no longer have to worry about my datafeed all day long." - Comment from Philippe
"I just wanted to tell you what a fine job you have been doing. While *******, from what I hear, has been down and out, off and on, IQ feed has held like a champ this week." - Comment from Shirin
"I've never had DTN go out on me since switching. ******* would go down a couple times every month when I was using them." - Comment from Bryce in AL.
"And by the way, have to say this. I love the IQFeed software. It's rock solid and it has a really nice API." - Comment from Thomas via RT Chat
"I've been using Neoticker RT with IQFeed for two months, and I'm very happy with both of the products (I've had IQFeed for two years with very few complaints). The service from both companies is exceptional." - Comment from Public Forum
"This is an excellent value, the system is generous (allowing for 500 stocks) and stable (and really is tick-by-tick), and the support is fantastic." - Comment from Shirin via Email
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Latency of Q messages
Author Topic: Latency of Q messages (4 messages, Page 1 of 1)

-Interested User-
Posts: 17
Joined: Apr 6, 2017

Posted: Feb 11, 2019 08:00 PM          Msg. 1 of 4

I've been trying to optimize my .net socket reader. First, I wrote my own server and pushed 1000 "Q," messages per second to help find any bottlenecks in my code. The mock server writes 1000 dummy Q messages in a loop. I've appended a high resolution time stamp to the messages (using interop GetSystemTimePreciseAsFileTime). I compare this time stamp on the reading side to see how fast I process messages. The writes of 1000 messages only takes a few milliseconds. I noticed that I was unable to keep up with this rate on the read side if I processed the message synchronously. I now push the messages to a BlockingCollection(ConcurrentQueue). Once I did this, my socket mostly reads one or two message per read.

I thought this would help reading the live feed. However, I still see many reads with numerous messages (>50). This is averaging 5, but sometime as many as 100+ messages are pulled in one read. Most of the time, these messages only differ in the Bid/Ask time stamp by a few milliseconds but sometimes are > 50ms (as high as 200 ms) from the first message to the last.

Is IQConnect batching the writes to the socket? Or are you batching the push from your servers? No matter what I do I cannot seem to reduce this.

I should note that I am collecting stats in an array and only dumping one line to the console every 10-15 second. I have timed the stat collection and display and I am confident that it is not skewing the results.

I need to sort this out not because I’m concerned about a few milliseconds but because I can get way behind (1-30 seconds!) in a fast market.

I am running on an AWS EC2 T2.Medium machine (2 cores).

My pings to the IQFeed server are constantly between 45-47 ms.

I am only watching CME products (@ES, @NQ, 5 major currencies)


-DTN Guru-
Posts: 2088
Joined: Nov 21, 2005

Posted: Feb 19, 2019 06:38 AM          Msg. 2 of 4
Hello, my apologies for the delay responding to this.

Generally speaking, the markets are naturally very "bursty" in terms of how messages are received from the exchanges. I haven't verified with your specific symbols but you are using some of the most active futures contracts as I'm sure you know. I wouldn't be surprised if your few symbols frequently burst well beyond your 1000 message loop test (unless I'm not understanding your test).

A 30s delay is certainly not normal and figuring out where that data is getting queued will help identify what needs to be fixed. If the data is getting to your machine but queued locally within IQConnect waiting to be delivered to your app, then in the diagnostics app, on the client stats tab, you will see the KBQueued column count up for your app's connection. If this number is zero even when you are seeing delays in your app, then the data isn't able to be delivered to your machine and is being queued on the server. Once we know this information it will be easier to identify what is causing the delay.

-Interested User-
Posts: 17
Joined: Apr 6, 2017

Posted: Feb 19, 2019 08:17 AM          Msg. 3 of 4
Thanks for the reply.

I have implemented the latest .net core 3.0 System.IO.Pipelines library. It "queues" and "dequeues" on separate threads while minimizing overallocation. With the help of the SequenceReader, you can access the buffer directly without having to copy to a string and then split. I have yet to test it with my mock server. I'll let you know if it performs better.

I also need to monitor KBQueued in code so I can log these infrequent latency events. Due to some logging changes, I was only able to see these events back in 2017. One of which coincided with an ECB announcement.

I will continue to monitor.


-DTN Guru-
Posts: 2088
Joined: Nov 21, 2005

Posted: Feb 19, 2019 08:45 AM          Msg. 4 of 4
The KBQueued value in the diagnostics app is just a display frontend for the clientstats messages on the Admin port from IQFeed via the API.

Turning this on/off and message format is documented here

Not sure if you were aware of that based on your post so wanted to make sure.
Edited by DTN_Steve_S on Feb 19, 2019 at 08:47 AM


Time: Sat April 4, 2020 4:13 PM CFBB v1.2.0 16 ms.
© AderSoftware 2002-2003