Is anyone else experiencing a very high rate of error packets? On Monday while streaming 1300 NASDAQ symbols, I received over 20,000 malformed packets in the first 2 hours. Most had either a missing timestamp or an incorrect number of fields.
IQFeed support informed me of a problem prior to the NADAQ open so it is possible that the problems were ongoing.
At around 11:50am I stopped receiving data. My application automatically disconnected and attempted a reconnect. This failed several times and the application aborted.

I am running IQFeed v2.3.0.2. The router logs and other monitors indicate the Internet connection was fine.

Dennis

Many of us internally run 1300 or more symbols all day long with our DTN.IQ client in order to monitor things. We have never seen malformed packets in our testing. We did have a premarket issues on a few of our servers that was corrected before market open. We also received calls mid/late morning that customers were unable to connect. Our system was fully operational at this time, so we expect the issue was some link between some of the major ISP's and our datacenters. Both of these issues were unrelated. As for the malformed packets, in nearly all cases this has been traced to a 3rd party application that doesn't have a large enough TCP buffer to handle the amount of data we are sending. If you are using your own app, try to increase the TCP buffer and it should take care of the issue.

---

Jay Froscheiser
DTN Market Access, LLC.

Jay,

I have already seen suggestions that the TCP buffer may be the cause. I am running WinXP and as far as I can tell the TCP buffer size in the IP stack is dynamically controlled by the OS.
I am running my own Java application and the buffers for TCP/sockets were increased with no difference to the result. I am running a dual 3.2GHz machine and cpu load is moderate so I'm easily coping with the stream of data.
I might switch on logging for the IQFeed client and compare that output with my raw data received. If the problem is related to the buffers then the IQFeed logs shouldn't be corrupted while my data is. I'll switch it on for Thursdays trading.

cheers,
Dennis

I too have been experiencing currupted records. They seem to be of two kinds.

1. Missing timestamp field. Some days I get just a few, and some days I can get thousands. If there's a lot of them, they usually occur around the open.

For now, I've been simply ignoring these.

2. Currupted layout. Appears as wrong number of comma separated fields, and the string "U,F|" showing up in some field(s).

Once this happens to a particular symbol, it looks like all subsequent records for that symbol are mangled. To get a symbol reporting correctly again, I dop it, wait 1 sec, and resubscribe.

I get between 0 and a dozen of these per day. Most, but not all, happen near the open. I'm watching roughly 475 symbols, all NASDAQ.

My monitoring program is the only thing running on a 3GHz 2MB Win2000 box. It has happened on both of my internet connections (one is ADSL, the other is cable, each with entirely different ISPs).

CPU useage peaks at 30%-40% for a few seconds during the open and otherwise is around 10%.

I have been logging the culprit raw records if you want to see these.

My project is in development and I can help with any additional logging or testing you might want me to try.

Thanks.

The missing timestamps have been around for a while -- I first asked about them a couple of months ago. My experience is similar to those reported above; usually a burst of malformed records, occurring near the open.

Allen

kdresser,

I've found that you're approach is the best way to handle these DTNIQ bugs. ie ignore updates missing a timestamp and unwatch/rewatch symbols that are coming down as corrupted. This bug has been reported as bug 2750 in the API buglist, however if you read through this list there is no indication of what its priority is or when it will be fixed (I would have thought it was critical)

If you browse through the threads in this forum you will see a variety of suggestions that help reduce but not eliminate these problems. eg increase TCP buffer size, use a separate thread to read data from the IQFeed client, etc All of these suggestions try to improve the speed with wich you read updates from the IQFeed client.
No one from the IQFeed development team has confirmed this but it would seem that they have separate buffers for each symbol and if these ever get full that symbol's updates become corrupted. That's why unwatching/rewatching works. We have implemented a very fast method for reading updates from the IQFeed client and it rarely gets more than 100 updates queued. At the start of market with 1300 symbols watched (2000+ updates/sec) that's a max of 50ms delay but we STILL get corruption.
BTW, you also have to be carefull about how fast you send the symbol watch requests!!

I've mentioned in other threads how ridiculous it is for the IQFeed client to pass on corrupt packets under load. If the computer or application is not fast enough then updates should simply be dropped. (IQFeed developers PLEASE FIX THIS BUG in the next release)

I am now actively looking for a second data feed for redundancy and reliability purposes with the intent of combining the two inputs to create one reliable feed.
Of course, as the DTNIQ feed does not include exchange generated time stamps with seconds or any unique exchange generated identifier it will be difficult to do this, but that's another flaming thread :-)

cheers

We are working on fixing the corruption under load.
We are also working on adding second resolution - and considering the addition of exchange time in a new protocol.
And we're working on tick correction.
The bad news is I don't have release dates for any of these upgrades but I currently expect them in first quarter.

---

Lonnie Shumate
Development Manager, IQ Systems
DTN Market ACCESS