That would provide a nice way of knowing the connection is up & running.

Seems to me that it would be minimal additional bandwidth usage. Currently my app monitors ETFs (QQQ, SPY, & DIA) as well as futures (NQ, ES, YM) and a number of indices. Often I receive well over 100 messages per second, but a few times a day, I receive zero messages in a one second time slice.

Greetings,

I'm a developer currently working on the DTNIQ client and I've brought this sort of issue to the server team before. They convinced me that it wasn't a very good idea based on the following:

1. Timestamps from the server (if they included second resolution) would be 20 bytes. That means for every 1000 clients we have connected to the server system we are sending out 20k worth of data every second just in time stamps

~20k per 1000 clients for each second
~1.2meg per 1000 clients for each minute
~72meg per 1000 clients for each hour
~468meg per 1000 clients for a 6.5 hour session

This amount of bandwidth for a simple timestamp would be cost prohibitive. Since we have plans to include a second resolution in trade time at some point in the future, second resolution on the timestamps seems to be overkill (and expensive for the clients since this bandwidth cost would have to be passed on to subscribers)

2. If we only sent the time stamp if the client didn't receive any data for the previous second, we would experience the following problems:

a. We would have to track what each client was watching for the previous second.
b. We would have to track if we received / transmitted any fundamental data for that client.
c. We would have to track if any of the symbols they we watching updated in the previous second.
d. We would have to track if they added / removed watches for any symbols in the previous second.
e. We would have to track if they received a news headline or any other message (like a regional update)

After we had figured out that we hadn't sent a message of any type to this client we would then send a timestamp. Again, when you get into a numbers game, only this time it's server processing...

3. TCP/IP is a reliable protocol. It guarantees delivery, unless the connection is lost.. This means if there is a problem with the quote server connection, the client is already notified. So having a second timestamp just isn't that necessary. If you really want to keep a second resolution on the client, synchronize off of the minute timestamp that we send you. This will allow you to generate your own second resolution until we provide the trade time in second resolution.

Sorry if I've been a bit verbose, but I've been through this issue myself and I wanted to explain it as thoroughly as possible.

Thanks,

---

Nathan W. Bartsch - Senior Software Engineer - DTN Market Access

Wow, thanks for complete reply.

Just checked my code, and it I currently watch 16 tickers. By my best guess, you send me about 200MB of data a day, and it seems like an additional 0.468MB shouldn't change the bandwidth metrics all that much.

But I'm more interested in in the lost connection issue. I asked this in another area and was pointed at the thread titled "How to fugure out current connection status ?" which indicates that it takes 75 seconds for lost connection to be reported, and that's way too long for me. Is that info out of date?

Well, the 75 seconds is about the maximum it's supposed to take, based on the fact that the feed should get timestamps once a minute. A failure in getting the timestamps is usually a result of network failure between the customer and our servers, not necessarily a failure in the TCP connection.

I've tried to avoid too much detail in this area up to now, but it looks like I'm not going to be able to do that this time, so, here goes

1. Detecting a connection loss because your internet connection to your computer is disconnected - this is very simple and, if IQFeed doesn't currently have this functionality I'm sure it will in the future.

2. Detecting a connection loss caused by software failure on our end is also easily detected, as the operating system will handle closing out the sockets, causing IQFeed to do whatever it is it does to notify clients and attempt to reconnect to the server way before 75 seconds has past.

3. Now, the most difficult. When a route between an IQFeed instance and our servers goes down temporarily - or becomes congested, or whatever - data may temporarily be unable to be transmitted between client and server. This is the nature of TCP, and it's designed to take these things into account, waiting a set (sometimes very long) amount of time before declaring that the connection is no longer valid. IQFeed attempts to get around this by saying, "After 75 seconds, if I still haven't received a timestamp, attempt a reconnect".

The need to wait a certain amount of time before giving up and attempting to reconnect is necessary, especially if the connection failed due to hardware problems somewhere along the line - attempts to reconnect every second are very likely going to be unsuccessful until the link is no longer congested, or the down hardware is back up, etc.

Also, having arbitrarily short timeouts tends to cause a bunch of false negatives - i.e., "No data received for 1.5 seconds, attempt to reconnect!". You can see where this would be a problem.

Let me recap a bit here:
If a connection fails (server kicks you off, you unplug your cable, software crashes), the remote TCP end should be notified immediately.

When a socket error doesn't occur, it's a very subjective thing to determine whether the TCP connection has 'failed' or is just stalled.

In a nutshell, the 75-seconds is for 'stalled' connections, not necessarily lost connections.

---

Tim Russell
Software Engineer
DTN IQ & FinWin

Let me run this up the flagpole. How about an IQ specific ticker that would send the time (HH:MM:SS) without the date once per second? This way only clients who want this would receive it, rather than the entire client base.

or an alternative, pick some obscure index, and configure it to send an update once per second.

Looking at yesterday (Oct 7) data playback, I see a number of instances (~10) where I received zero messages from IQ, but was receiving messages from my second feed (IB). I'm not so worried about this, since they are a single second interval.

Then something "funky" happened which resolved itself at 2:02:06 EST, which has me scratching my head. For the 6 seconds prior, I received zero messages from IQ, and for the 9 seconds prior I receive zero messages from IB; there was 6 seconds of overlap where I receive zero messages from both feeds. Upon resolution, I received 98 messages from IQ and 120 messages from IB.

Ultimately, I trying to discern between instances where I get a zero message count simply because IQ had no data to send me vs. those where some part of the technology had a "hiccup".