Hi,

In the past, from time to time, we have encountered a situation where although the IQConnect client is still connected to the DTN servers, the servers seem to just stop sending data.

This used to be a very infrequent occurrence, but in the last couple of weeks it has become more and more prevalent - last night we had this happen 4 times on 4 different servers:

• 66.112.148.220 at 11:15:18
• 66.112.156.225 at 11:55:18
• 66.112.148.114 at 11:57:52
• 66.112.148.229 at 12:03:21

After 10 seconds of no data we automatically disconnect and reconnect to a new server, so there is no way for us to know how long the outage would have lasted, but in the past we have seen these situations last for up to 10 minutes (we used to only react to server-side disconnects).

Can anyone shed any light on what is happening here, please? Is this a known bug? Is this related to the memory leak problem Jay has mentioned in other posts?

(Oops. Our last night is your yesterday! All times are US EST.)

I have received a private message from another developer who was struggling with this IQFeed bug in December 2005. The staff member he dealt with was Natalie Hannan, who suggested sending a "T\n" to restart the data feed.

So, it appears that not only is this a known problem, but it is nothing like a new one.

Would anyone from DTN care to comment?

The purpose of the "T\n" was to identify a broken socket and cause IQFeed to reconnect to the servers.

The issue is that the older versions of IQFeed just receive data and, should the connection drop, there is no indication in IQFeed that the connection is broken. "T\n" is a timestamp request from the server and the act of transmitting it will fail on a broken socket and cause IQFeed to reconnect. From the user standpoint this could be construed as "restarting the datafeed" but its' actually just reconnecting to the datafeed.

We identified this problem years ago and started monitoring timestamps in IQFeed. To correct it we added a local timeout when a timestamp isn't received and IQFeed will automatically send a "T\n" to force a timestamp.

So the information you are using concerning the "known problem" is out of date, that problem was solved. However, this doesn't solve your problem and the symptom seems similar.

So when you lose the data feed does the IQConnection Manager show reconnections? Do you continue to receive timestamps?

---

Lonnie Shumate
Development Manager, IQ Systems
DTN Market ACCESS

Hi Lonnie,

Using my available networking tools, I can see that the connection has not been broken. We still receive timestamps, and even the occasional update. Your servers quite simply appear to just stop sending 99.9% of the available information.

Your "local timeout" is completely insufficient for our purposes. As I stated in my first post, we use a 10 second timeout for our data feeds. If data from your servers falls below a reasonable threshold during market hours, we drop the connection and reconnect to another server. We are not willing to increase the value of this timeout, as we depend on a timely data feed for our trading information, and hence our livelihoods. As we do not (now) give IQFeed the opportunity to perform its own reconnection, whether or not the IQConnection Manager shows reconnections for timeout reconnects isn't information that we have available to us.

It is not surprising that the information I was given was "out of date", considering it was from another user attempting to help me out. If someone from DTN had themselves responded to my earlier post, there may have been a bit less confusion. That said, I am quite puzzled by both your description of the earlier problem, and the solution you chose to implement. Surely the question should have been, "Why does IQFeed not recognise when the server connection has dropped, and how can we fix this?".

Your note has at least convinced me that we need not attempt to implement sending a "T\n" to restart the connection. We are better off with our current solution. The question however still remains as to why the servers stop sending data to connected clients. And that is where we started.

I am a new user of IQFeed data feed, and not yet actually trading on this data. Here is some more 'evidence' concerning the 'missing data' problem described above:

Today, Wed, 9/19/2007 about 11:15am PST the data for @YM# emni-Dow (continuous contract) stopped completely, until 1:00pm. That is an 1hr-45min time span of no data! I happen to be monitoring the index itself, INDU.X and also the DIA DIAMONDS TRUST data, all at 10 sec compression. Just for your info, my IQFeed version is 4.2.1.4.

The long break in the datastream for the symbol @YM#, was NOT observed in either of the other two symbols, but all are being plotted on the same chart (MultiCharts). I have also, tonight at about 6:30pm, (hours later) made the charting software do a 'reload' which is supposed to 're-quest' all the data on the chart, and then fill in any missing data. That action still resulted in the same missing data to remain "MIA" :>)).

So that would seem to imply to me that the data for that 1hr and 45min is STILL missing on at least the server that I have been connected to. How can this happen??

Doesn't DTN monitor and detect any missing data on server (which according to the IQ Connection Manager, has IP of 66.112.148.222, port 60002]. Am I supposed to somehow 'find' a different server for the re-load of the missing data? Can anybody shed some light on this situation? When will this data be made available, and when will the problem be fixed so that it does not happen in the future?

Just an added note: I exited my charting application, and exited the IQFeed connection, and then reconnected everything a few minutes later. Now the server IP shows as 66.112.156.225, which is different than reported above (a few minutes ago). However, the missing data for @YM# is still missing (missing on all of my different intra-day charts). So, maybe the problem is not related to a single server?

Has anybody else observed that this same emni-dow data is misssing? That would be helpful to know, just to rule out any issue with my computer's database.

ECBOT was down at the exchange today during that time. It was a problem at the exchange affecting all vendors, not only DTN. This would explain why you weren't getting data on the ecbot contract only, and why we aren't able to fill in the missing data.

---

Jay Froscheiser
DTN - Trading Markets

And one more attempt to 'investigate' what is going on, I just requested both the @YMU7 and @YMZ7 futures contracts and charted them (this time at 1min resolution). BOTH of these two contracts are missing data during the same period! That seems logical, since the @YM# contract is defined to be the most 'current' contract, which today I believe would be the DEC (Z) contract, right? In any case that data missing for 1hr and 45 min today, just before 1:00pm (local, PST) time, and that would be 1hr-45min before 4pm EST.

So, again, anybody else see the same problem? What is going on here? :>))

Hi Jay,
guess I missed your post while I was typing the last onejavascript:insertsmilie('');
javascript:insertsmilie(''); Ok, so the ECBOT exchange was down during that time, so does that mean that you get your data via the ECBOT exchange, and thus why the data is STILL not on your servers? The data looks ok AFTER about 1:00pm-PST, until about 4:00pm-PST when I believe the GLOBEX session is closed.... and I now see more data on @YM# beginning 2hr-15min later (4:15pm-PST).

So, will your servers somehow get the missing data replaced by tommorow? Or is it missing forever? Do you have a direct network connection to the source at CBOT, or are you dependent on the GLOBEX or ECBOT "exchange"?

e-cbot is the exchange. The Chicago Board of Trade. We get the data direct from the exchange (we are a tier 1 vendor). It was an outage AT the exchange, so there is no data to fill in. Their system was down, so there was no trading to report.

---

Jay Froscheiser
DTN - Trading Markets

Thanks Jay... now I understand. Not yet fully aware of all of the 'players' involved in making the new 'internet' and 'online' trading a reality. That is kind of scary that the 'exchange' can actually be down for so long. Don't they have provisions for multiple backup-network just so this kind of thing 'should' never happen? The ECBOT is just another ECN right? I thought such networks were designed to be up 99.999% of the time, right? Do you have any insight into how this could have happened that there was no automatic 'switchover', or the equivalent, so components that were still working? This kind of situation can cost a lot of people a LOT of money, so there are no excuses allowed, right?

In my observation E-CBOT has always been less reliable than its counterpart E-CME where the ES & NQ e-mini futures are traded. I recall the two exchanges merged earlier this year, however no plans annouced to move YM to the more stable platform.

For this reason, I've turned my attention away from YM and on the other two. Also more volume and deeper market on ES & NQ.

Thanks, nsolot, for the useful info. I will certainly re-consider my trading 'plans' and look at the other eminis. There might be more down time before they are finished with the merger of CME and CBOT. My experience with the mergers of any two big companies is there are a lot of mishaps before it is altogether and working as one team and one computerized system and networks. I suspect that even the other eminis could be experiencing similar problems sometime in the future during this merger process.

I'm not saying E-CME platform is 100% bulletproof...

just better (more reliable) than E-CBOT.

In either case, you should see almost contant size updates on bid/ask or both. If things go quiet for more than a few seconds, best to assume something has gone wrong.