I know Jay mentioned this will be added in future, but I wanted to list it here.

P.S. I'm surprised if not shocked that this was not added from the start.

Add another vote for seconds & milliseconds to time stamp.

Millisecs is important since it's possible that packets over the internet might not arrive in the same order as they are sent (or at least I'm led to believe).

Information transmitted via TCP is guaranteed to be provided in order to the application, as it is a reliable protocol. UDP, on the other hand, is unreliable and data can arrive out of order, or not at all.

IQFeed utilizes TCP for all transmission, so this isn't an issue.

We're working on adding seconds to the data feed currently. I don't foresee adding milliseconds - as far as I know, the exchanges only send times with a precision of seconds.

---

Tim Russell
Software Engineer
DTN IQ & FinWin

This isn't my area of expertise, so I won't give you an arguement, but still curious. I am under the impression that packets over the internet may take different routes between transmitter and receiver and then are "reassembled" in the correct order by the receiver, but this only applies if multiple packets in a single stock/future/option transaction. I'd hope each transaction is small enough to fit in a single packet. If that's true, in theory it's possible for an active issue like QQQ's for IQfeed to send Trade 1 then Trade 2, but be received Trade 2 then Trade 1?

More important to me, and my "wish" would be an additional time stamp placed by IQfeed, with milliseconds, when the trade was received by your system. This would provide me with 2 important peices of info, 1) I could detect if there was some unusual delay between IQfeed and the exchange, and 2) I could detect some unusual delay between your ticker plant and my location.

True, individual datagrams can travel multiple routes through the Internet. However, TCP ensures that they are presented to the application layer in order (based on sequence numbers at the transport layer - not really important). Suffice it to say, the order we send it in is the order that you'll receive it in, guaranteed.

As for milliseconds - they're mostly arbitrary and I really dislike the idea of timestamping them on our end, as our live and EOC systems would then have different (close, but still different) timestamps for the same event. We are adding Tick ID (exchange-generated) to allow you to uniquely identify ticks that occur in the same second.

---

Tim Russell
Software Engineer
DTN IQ & FinWin

So, correct me if I'm wrong, but in the case a packet gets lost or trashed and needs to be resent, then the feed will "stall" until it's sucessfully retransmitted?

Correct - that's the way TCP works. Basically, each TCP packet on the internet will contain sequence information that allows the remote machine to re-sequence and re-request data as needed. When data is missed, the TCP stream will 'stall' until the missed data is re-transmitted. This can happen quickly, or it can take several minutes, depending on the cause of the outage.

This is all transparent to most people, as it's an operating-system level thing.

---

Tim Russell
Software Engineer
DTN IQ & FinWin

Thanks Tim:

This is very enlightening to me, and reinforces my desire for seconds (and perhaps ms) timestamps. One thing my application is watching is patterns of trades. I'll bet you've watched a streaming ticker for an active issue (e.g. e-mini SP's or QQQ's) and you see an period where a bunch of prints occur really fast.

I want to use the time stamps to try an determine that the market has suddenly become very active, however, based on our discussion, it would appear to me that if there is a technological "hiccup", that could appear like a "market event" when in fact it's just the feed getting caught up.

MS timestamps might be helpful in an active market to determine just how "real time" the quotes are being delivered to end users vs. normal market volume. Don't know if you were around in '87 when the tape was running 30-60 minutes slow (drastic example).

I believe someone mentioned the "trade timestamp" in this discussion, but if not, I'd like to ADD it. I know for a fact that the exchanges (at least the CME) send a trade timestamp that at LEAST includes the time down to the second (ins probably a full millisecond TMS) and I have NEVER understood why vendors (DTN isn't the only one) seem to feel compelled to TRUNCATE the seconds off. I am a registered user of the DTN Developer Kit and I was dismayed to discover that the seconds are not even available to a C programmer at the bit level! DTN simply "drops" that information that is passed to it from the CME (unless they are getting the CME feed from ANOTHER intermediate vendor now?).

Not knowing the exact second (or even millisecond) of when a trade was recorded at an exchange is absolutely VITAL imformation to have, especially as we move more and more to electronic exchanges and networked interfaces. If I cannot determine within ONE or TWO seconds at most that a delay has occurred, then I am at a great disadvantage in my trading.

If DTN is confident in it's datafeed, then it should be PROUD to pass on to us, the users, the FULL EXCHANGE GENERATED trade timestamp (not just to the minute) on each and every trade, bid, offer, volume stat, announcemets..... EVERYTHING.

RIght now I can only do a manual time "sync" on a minute-boundary. That is, I have to watch my PC clock (which is constantly synced with the Atomic clock) as it approaches the minute change and then see how quickly the DTN's "minutes-only" version of the timestamp catches up with real time.

I urge DTN to correct this situation globally for all its products.

"and I have NEVER understood why vendors (DTN isn't the only one) seem to feel compelled to TRUNCATE the seconds off. "

Its obvious isn't it? Without the accurate timestamp from the exchange, we the users have no way to quantify the lag through our data supplier.

I stand by my first post in this thread.

We ARE adding seconds into the data feed.

---

Tim Russell
Software Engineer
DTN IQ & FinWin

While you may not be able to determine complete lag from the exchange, our customers are very good and determining if we are lagging other feeds. We are constantly being compared to other feeds. The fact of the matter is you shouldn't care what the lag from the exchange to you is. You should only be concerned that you are getting your data as fast or faster than anyone else.

---

Jay Froscheiser
DTN Market Access, LLC.

in a quick double check, the Nasdaq only sends seconds, the futures ITC spec sends 10th of a second resolution. We will be implementing second resolution for all of our feeds to the IQ product lines since that is what our ticker processes.

---

Jay Froscheiser
DTN Market Access, LLC.

To Tim Russel, in particular.

Do you mean that the seconds are going to APPEAR IN THE TIME COLUMN in the Watchlist and Time and Sales windows AS WELL AS coming through into my Excell spreadsheet via DDE capture of the "trade time" columns?

I have always had TWO reasons for the seconds: One was to compare the time lag represented by the difference between the exchange time stamp and the "instant" I receive the quote. I have done this on a number of occasions exactly for the purpose of determining IF iI'm getting a feed "faster than anybody else." On a number of recent occasions I have determined that I am NOT, indeed, getting the feed fastter throught DTN, but directly through my on-line broker's data feed! So, YES, mr Froscheiser, the seconds are VERY IMPORTANT to be able to do this more easily and accurately!

However, the second, and more important ,. reason is: I NEED the seconds in order to most thoroughly do HISTORICAL TESTING using the data I capture as well as the historical tick data from other vendors, including the exchanges themselves. I do this historical testing to check for ht accuracy of the data I DO get from DTN as well as to test my market trading systems which DEMAND historical data to the second, and need the REAL TIME data to the second as well in order to be able to confidently USE the signals I determine from the historical testing!

I am thrilled to hear the seconds will "soon" be included, but will they be included where I NEED them.... in the Trade Time columns?!?

yes. although it may be phased in as we make the proper modifications to the client applications and server applications. But the answer is yes.

---

Jay Froscheiser
DTN Market Access, LLC.

In response to Scott issue about Historical data testing, I believe Jay and/or Tim has stated they will be implementing both an exchange Tick ID and a DTN Tick ID.

Obviously if the exchange sends a trade or tick timestamp, please send this also - in my case its not to blame DTN for a few milliseconds lag, but rather its for analysis.

But at the very least a sequential DTN Tick identifier is essential, for:

1. Piecing things together from history in the correct order
2. Synchronizing real-time with intra-day ticks - both with crash recovering and in my charts knowing where the real-time ticks start and the previous intra-day ticks end

Jay, can you make me feel better by saying this sequential tick id will be available? I'm pulling my hair out wondering how on earth to do this with absolute precision - its currently impossible.

Yes we will have a tick id which will allow you to easily syncronize between historical and streaming data.

---

Jay Froscheiser
DTN Market Access, LLC.

I would like to request that any answer or response made to this post includes, as I do, the phrase "Originating Exchange Generated" in front of the various tick data attributes under discussion. "Originating Exchange Generated" means the attribute value was determined by the exchange where, and as, the specific trade occurred, and was formatted into the tick record that was then distributed to various subscribers (in this case DTN).

In the past, I have been told by technical people at more than one "feed provider" (including DTN!), that the tick timestamp (only to the minute) was generated by the "feed provider" itself when the tick record was RECEIVED BY the "feed provider" and that is did NOT represent the 'Originating Exchange Generated" timestamp! I was astounded that a "feed provider" would even consider re-selling such a mixed bag of information.

So, I ask again, will the "new" tick timestamps down to the second, and the new tick sequence that Sasha wants (I agree this is very valuable as well) be defined as:

Originating Exchange Generated Tick Sequence ID
Originating Exchange Generated HH:MM:SS Tick Timestamp

Is this true? (Of course, I assume all the other tick information, price, bid, offer, volumes, sizes, etc are also the Originating Exchange Generated data as well.)

I would really appreaciate a definite, specific, "for the record" answer to this. If I have not made this question clear enough or you haves any doubt about what I mean, please respond by asking me to rephrase or clarify.

Thank you,

Jay:

I'm sorry but I have to disagree very strongly with your comment:

"The fact of the matter is you shouldn't care what the lag from the exchange to you is. You should only be concerned that you are getting your data as fast or faster than anyone else."

Let's say someone is looking for abitrage opportunities between the CME e-mini SP future and the SPY (which has some quote issues discussed in another thread). Knowing that there is an exchange related delay is critical to that type of trading model.

Scott,

I am not sure if we store that info in our ticker, but we will check. We may be able to make this available as a field, but it won't be stored in historical/tick data. Most (or, all that I know of) don't provide this field because for storage in internal systems you want a consistent timestamp relatively. None of the exchanges are 100% in sync for time. They are always at least a second or more off from each other. This is why I say looking at an exchange timestamp and ours doesn't give you a good idea of true lag.

---

Jay Froscheiser
DTN Market Access, LLC.

Quote: I would like to request that any answer or response made to this post includes, as I do, the phrase "Originating Exchange Generated" in front of the various tick data attributes under discussion. "Originating Exchange Generated" means the attribute value was determined by the exchange where, and as, the specific trade occurred, and was formatted into the tick record that was then distributed to various subscribers (in this case DTN).

In the past, I have been told by technical people at more than one "feed provider" (including DTN!), that the tick timestamp (only to the minute) was generated by the "feed provider" itself when the tick record was RECEIVED BY the "feed provider" and that is did NOT represent the 'Originating Exchange Generated" timestamp! I was astounded that a "feed provider" would even consider re-selling such a mixed bag of information.

So, I ask again, will the "new" tick timestamps down to the second, and the new tick sequence that Sasha wants (I agree this is very valuable as well) be defined as:

Originating Exchange Generated Tick Sequence ID
Originating Exchange Generated HH:MM:SS Tick Timestamp

Is this true? (Of course, I assume all the other tick information, price, bid, offer, volumes, sizes, etc are also the Originating Exchange Generated data as well.)

I would really appreaciate a definite, specific, "for the record" answer to this. If I have not made this question clear enough or you haves any doubt about what I mean, please respond by asking me to rephrase or clarify.

Thank you,
--- Original message by scottladu on Oct 16, 2004 10:15 AM
Scott,

I am not sure if we store that info in our ticker, but we will check. We may be able to make this available as a field, but it won't be stored in historical/tick data. Most (or, all that I know of) don't provide this field because for storage in internal systems you want a consistent timestamp relatively. None of the exchanges are 100% in sync for time. They are always at least a second or more off from each other. This is why I say looking at an exchange timestamp and ours doesn't give you a good idea of true lag.
Jay Froscheiser
DTN Market Access, LLC.


Mr Froscheiser.

With all due respect, did you carefully read my post? First of all, I discussed TWO fields and you only refer to "...a field." I know which ones I was talking about; which one did you mean (and I would REALLY appreciate it if you humored me and actually used the full descriptive names that I used so that you don't keep confusing the issue).

I will try to make this clear again. What I, and I believe a number of other CUSTOMERS want is, the ORIGINATING EXCHANGE INFORMATION for both timestamps AND sequences (and all the rest); that's what I'm PAYING for and that's what I EXPECT. What I don't want it yours, or any other vendor's, "editorializing" about what information YOU think I need... I KNOW what I need! And if you are not providing the ORIGINATING EXCHANGE INFORMATION for ALL the data fields, then I believe you are violating our agreement.

I am now requesting a full, comprehensive definition, including the source and processing rules, for ALL the fields I am now PAYING TO GET IN REAL TIME from the CME on the Emini contracts. If I have not been getting the full and complete CME ORIGINATED data for the last few years, I may just feel entitled to a refund! Seriously, I am willing to work with you on this, but you've got to 'fess up on your side. Either you're doing it right or you're not; which is it?

Scott, I am not here to humor anyone (especially you). The forum is here to find out what our customers want. They aren't here so you can threaten to cancel and request refunds because you don't get exactly what you need. You cannot get a "Full and completely CME Originated" datafeed via the Internet. Nobody provides this via their retail product, and it isn't stated that we do in any of our "agreements". If that is what you need, pull in a leased line from the CME and pay their re-distributor fees. Its as easy as that. We don't develop our products for any one person. We develop them for a group of people who are looking for high quality data at a reasonable price.

So, in the end, we are doing it right for the majority of our tens of thousands of customers. We can improve, and we will.

---

Jay Froscheiser
DTN Market Access, LLC.

OK, I'm sorry if I offended you, but I grew up in Chicago where a gentleman named Marshal Field came up with the concept that "the customer is always right!" And in my almost 30 years as a programmer, system analyst, database architect, etc. I have had to adhere to that concept in order to provide my users with applications that actually did what they were supposed to do.

Along with the systems themselves, my customers have always demanded good, accurate documentation, including a Data Dictionery with complete and accurate descriptions of all the data element/attributes in the system along with the SOURCE of the the element, it's points of udate, the Owner of the element, etc.

In your retort to me, you still didn't respond to my request, which is to provide this kind of complete description of the data elements I get in the real-time Emini tick feed, including any new timestams (to the second) and sequence IDs that are appently under development.

What this customer wants is to know is if there is some reason you are unable to provide me with this information, or if I haven't made my request clear.