Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"You are either overstaffed or people just don't have problems with your feed because customer support always answers the phone quickly." - Comment from Jay via Email
"Everything is working amazing now. I'm already impressed with the true-tick feed of IQFeed and it's ability to support my 480 symbol layout." - Comment from Tyler via Email
"Thanks for the great product and support. During this week of high volume trading, my QuoteTracker + IQ Feed setup never missed a beat. Also, thanks for your swiftness in responding to data issues. I was on ******* for a few years before I made the switch over early this year, and wish I had done it a long time ago." - Comment from Ken
"It’s so nice to be working with real professionals!" - Comment from Len
"There is no doubt that IQFeed is the best data provider. I am very satisfied with your services. And IQFeed is the only one that I would recommend to my friends. Now, most of them are using your product in China." - Comment from Zhezhe
"With HUGE volume on AAPL and RIMM for 2 days, everyone in a trading room was whining about freezes, crashes and lag with *******, RealTick, TS and Cyber. InvestorRT with IQFeed was rock solid. I mean SOLID!" - Comment from Public IRC Chat
"I noticed that ******* quotes locked up shortly after the interest rate announcement yesterday while yours stayed stable." - Comment from Ron in Utah
"The service is great, I see a noticeable improvement in my volume profiles over [broker]'s data feed" - Comment from Larry
"I am very happy I changed. I love the product, but more so I am thrilled with Tech Support. You are knowledgeable, polite, pleasant and professional." - Comment from Pat
"Thank you so much - awesome feed, awesome service!" - Comment from Greg via Email
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Keeping up with L2 flow during busy market hours
Author Topic: Keeping up with L2 flow during busy market hours (8 messages, Page 1 of 1)

jmv
-Interested User-
Posts: 19
Joined: May 31, 2011


Posted: Jun 16, 2020 10:18 AM          Msg. 1 of 8
Hi there,

recently, I've spent some time modifying my trading application in order to keep a full real-time visual snapshot or "view" of the DOM landscape. While the visual result is very fancy and all of that (d'oh!), in the process of toying around with IQFeed's L2 stream I've come across a problem which I'd like to share with you.

Thing is, L2 can be at times a really busy stream, even more so if you happen to pick a busy symbol (NQ, ES, YM...). I've come to record incoming L2 message ratios well in excess of 1500 L2 messages per second coming from IQFeed during busy market hours (stock maket open at 09:30EST, close at 16:00EST, etc.. Normal, "quiet" periods tend to average 200-500 messages per sec.

The problem I'm experiencing is that, the moment the L2 message ratio reaches aound the 1000 mps (messages per second) mark, my IQClient app starts losing L2 messages, and it will disconnect / reconnect to your L2 servers. Here's an example from the logs:

------------
STATUS Information 36 0 2020-06-16 16:21:38 No data from Level 2 server in 7s. Reconnecting.
STATUS Information 36 0 2020-06-16 16:21:38 Attempting to reconnect to Level 2 server.
STATUS Information 36 0 2020-06-16 16:21:43 Connected to Level 2 server.

STATUS Information 36 0 2020-06-16 16:40:03 No data from Level 2 server in 43s. Reconnecting.
STATUS Information 36 0 2020-06-16 16:40:03 Attempting to reconnect to Level 2 server.
STATUS Information 36 0 2020-06-16 16:41:09 Connected to Level 2 server.

STATUS Information 36 0 2020-06-16 16:41:48 No data from Level 2 server in 39s. Reconnecting.
STATUS Information 36 0 2020-06-16 16:41:48 Attempting to reconnect to Level 2 server.
STATUS Information 36 0 2020-06-16 16:42:11 Connected to Level 2 server.
----------

Sometimes, it will recconect itself in mere seconds (3-4 seconds), while other times one or two minutes can pass until it will recconect itself to the L2 server.

I know that if my end cannot keep up with IQFeed's server, your host will likely drop my connection, hence these disconnections could be due to my system not being able to cope with one of these "L2 storms" which it's happening during market busy hours.

The problem I see here is that I already tried to make a very simple, stripped-down L2 client just to make a proof-of-concept of how many L2 messages per second I would be able to catch. Even with a simple read-and-discard strategy (no L2 message storage, no ascii parsing, no dynamic structures used, no thread locking, no synchronization locks, etc) and then, I keep getting kicked out of your L2 servers during _Really_ busy market events (opening, close, etc.).

So, if the software isn't blocking your stream flow... could it be just a latency problem ? My servers have a not-so-nice 180-190ms ping round trip to IQFeed servers. With as high a latency as that, am I facing a structural problem here, and no matter how fast my software is, I won't be able to keep up with a fast moving L2 stream ? (unless I opt for leaving Europe and collocating my servers within USA, of course..)

thanks for your time,
Edited by jmv on Jun 16, 2020 at 10:18 AM
Edited by jmv on Jun 16, 2020 at 10:18 AM
Edited by jmv on Jun 16, 2020 at 10:20 AM

DTN_Gary_Stephen
-Interested User-
Posts: 93
Joined: Jul 3, 2019


Posted: Jun 16, 2020 11:15 AM          Msg. 2 of 8
There are no hard-and-fast limits on the number of L2 messages you can receive at once, or anything else that should be causing this behavior. Can you tell me what programming language you are using, and the list of symbols you're requesting that's causing the problem? You may send this information to the IQFeed API support email or support chat if you'd prefer.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

jmv
-Interested User-
Posts: 19
Joined: May 31, 2011


Posted: Jun 16, 2020 11:28 AM          Msg. 3 of 8
The language I'm using is Java (openJDK 14), and I'm asking for seven symbols : ES,NQ,YM,6E,6J,6B, BTC .

I'm just toying a bit with the system's granularity right now... I'm checking the receiver buffer a bit less often (every 25ms instead of 5ms which was before). I'll let you know if this eases the problem.

However, the real test will come when markets close (i.e., 16:00EST ) since this is one of the busiest events in the day, from a L2 perspective. So in three and a half hours we will be able to see whether we still hit this problem again.

For me, it's highly suspicious that the problem only manifests itself in the busiest market moments (i.e., when the L2 message flow is at its peak). Either my receiving software(*) is the bottleneck, or my latency is causing the problem, I'm afraid.

(*)note: as for my software, I've also tried raising the socket buffer size, just to be sure that I wasn't discarding L2 messages due to a network bufer overrun... but that doesn't seem to be the problem, either.
Edited by jmv on Jun 16, 2020 at 11:30 AM
Edited by jmv on Jun 16, 2020 at 11:33 AM
Edited by jmv on Jun 16, 2020 at 11:36 AM
Edited by jmv on Jun 16, 2020 at 11:37 AM

DTN_Gary_Stephen
-Interested User-
Posts: 93
Joined: Jul 3, 2019


Posted: Jun 16, 2020 12:22 PM          Msg. 4 of 8
If you are willing, we can test your app in-house. This will help eliminate network or machine as possible causes of the delay.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

jmv
-Interested User-
Posts: 19
Joined: May 31, 2011


Posted: Jun 17, 2020 08:42 AM          Msg. 5 of 8
Thanks for your kind offer ,

as of yesterday's experiments, I've found out that tinkering with granularity (i.e., thread locking frequency) helps yielding better results. Yesterday, I got only a final (and single) L2 disconnection event, at market's close (16:00EST) which was to be expected, since it's the busiest moment of day (along with the stock market opening at 09:30EST, this is). This is much better than previous days, when I was getting several L2 disconnections *per hour*.

This tells me that the problem is at my end (most probably, excessive thread locking is still my problem). I think I can fix this by toying a bit more with my system's granularity. Should this not be the case, I will send you the source code of my stripped-out as-simple-as-possible test mule barebones L2 receiver, so that you can check it out for yourself.

thanks for you time and great support,

DTN_Gary_Stephen
-Interested User-
Posts: 93
Joined: Jul 3, 2019


Posted: Jun 17, 2020 09:03 AM          Msg. 6 of 8
If you decide you would like us to review it, you can get in touch with me via any of the support channels. Ask to speak to Gary (me), and I'll handle everything from there.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

jmv
-Interested User-
Posts: 19
Joined: May 31, 2011


Posted: Jun 22, 2020 04:31 PM          Msg. 7 of 8
Hi Gary,

just a couple of lines in order to let you know that, by means of optimizing our code (mostly trying to avoid race conditions and time-costly locks), I've been able to keep the L2 disconnections at a minimum (perhaps twice or three times a day, in the busiest day moments).

So the problem is _nearly_ solved. However, there's one thing I forgot to mention in my initial problem description: for years, I've been running iqconnect.exe via wine from linux, which I am quite sure it gives some (if not quite a) performance penalty.

So, after all, I'm quite sure right now that those two or three "spikes" would be handled correctly by iqconnect.exe , should it be natively run from within a Windows machine. Within the next few days, I will set up a Windows Sever 2016 instance in order to try whether this is the case.

Other than that, we're pretty satisfied with our performance right now.

thanks for your time and support !

DTN_Gary_Stephen
-Interested User-
Posts: 93
Joined: Jul 3, 2019


Posted: Jun 24, 2020 06:51 AM          Msg. 8 of 8
Thanks for the update! IQFeed is a native Windows program, and I can't offer a lot of advice on how to run it under Linux or any non-Windows OS. You might also look for any Wine errors or log messages at the times you're having these spikes; in some cases that can clue us in to what's causing it.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist
 

 

Time: Thu July 2, 2020 9:50 AM CFBB v1.2.0 15 ms.
© AderSoftware 2002-2003