Intraday data for days prior to 18th of March seems to be corrupted for CME futures. The times of data ticks is out of sequence in the DTN Time & Sales program, and the same problem is evident in minutes in the Chart program. I've checked @EUM5, @ESM5, @NQM5, and @BPM5 and they all exhibit the same problem.

Many thanks for investigating,

Brendan

We are currently looking into this issue. More information will be posted when available.

Not sure if I am seeing the same problem you were referencing in this thread, but the data for the index futs (ES, NQ, ER2, and YM) is messed up (thats the best technical description I can offer) for the period 3/14-17.

If this is the same problem, and "marketdataservices" said on 3/21 they were looking into it, are they still "looking into it"???

Seems like a week ought to be plenty of time to look into and remedy a problem of this nature. When can we expect the data issue to be corrected. As it stands, it would appear that I am stuck with bad data in my TICK and Range charts since my charting software only allows me to refresh up to 6 days of history. What can DTN do to help me rectify this?

---

Tradinoncoffee

Yes that is the problem. I agree, seems a long time to replace some bad historical data. DTN emailed me this 45 minutes ago:

"As of yet we have not completed rewriting the bad data.
This is a labor intensive process and will be released once the data is complete.

Ken Heinze
DTN Market Access
Customer Operations Specialist "

They've agreed to credit my account for the period I've been without the futures data I need, so that softens the blow a little. I suggest you also claim compensation.

Regards,

Brendan

Just received this:

"From: IQFeed Customer Support [mailto:support@iqfeed.net]
Sent: 01 April 2005 00:06
Subject: Re: @EUM5 Data Corruption (Thread:441926)


This was caused by corrupt data coming from the exchange in which we have no back up nor source of historical data. We are correcting each symbol by hand and we will continue to correct each symbol until it is completed. @ESM5 should be corrected by Monday.
Thank you and we apologize for the delay
Mike "

Two points spring to mind:

1. Your only function is to supply market data, for which you have "no backup nor [alternative] source of historical data"? Is the robustness of your service not worth one measly monthly subscription to an alternative data provider?

2. On finding some historical data was corrupt, instead of paying your staff $$$$ to manually type in every time and sale (which is what your email implies you are doing), you STILL don't spend $$ on a subcription to an alternative data supplier.

Am I missing something here, or has this exposed DTN's service as an absolute shambles? Does anyone have the correct intraday data for any CME futures between 3/14-17 they can send to support@iqfeed.net? Maybe we'll be better at supplying market data to DTN than DTN are to us.

Brendan

Perhaps one should take that reply with a grain of salt, in light of the date stamp.

Maybe this will help:

http://www.cme.com/trading/dta/hist/ehist2419.html

"2. On finding some historical data was corrupt, instead of paying your staff $$$$ to manually type in every time and sale (which is what your email implies you are doing), you STILL don't spend $$ on a subcription to an alternative data supplier."

I certainly hope they are not wasting their time typing in time and sales, because the tick history files on their servers are automatically deleted after 8 calendar days. So even if they manage to type it all in, us poor sods that pay for this service won't be able to update our databases anyway...

I don't think it 's an April fool, as their previous email to me (see above) stated "This is a labor intensive process and will be released once the data is complete", which suggests they are actually typing the times and sales one by one. I understand the confusion though as DTN's handling of this situation is difficult to distiguish from a joke. Hopefully they'll buy the data from CME using your link rather than continue doing it their way.
Edited by bdholmes on Apr 1, 2005 at 07:53 AM
Edited by bdholmes on Apr 1, 2005 at 07:53 AM

A couple things.

First, Yes, we have to manually correct the data. This is because no other data source has their data in IQFeed format. This wasn't done "By hand" as in someone sitting at a terminal typing in numbers. We had to build a manual import routine to get the data into our server format.

Second, We cannot just subscribe to another service and use their data. This is illegal and against not only the services terms of use but that of the exchanges. We can get the data from the exchange.

"Am I missing something here, or has this exposed DTN's service as an absolute shambles?"
You are missing something....

---

Jay Froscheiser
DTN Market Access, LLC.

...so what are we missing? Delayed data....corrupt and seemingly unfixable data....???

---

Tradinoncoffee
Edited by tradinoncoffee on Apr 2, 2005 at 11:08 PM

Jay,

I now understand why you cannot just subscribe to an alternative supplier, and that you have to convert the replacement data into your own format before importing it.

Please can you confirm that you now have a readily available alternative data source for CME futures, and a system in place so any corrupted data can be corrected MUCH more quickly in future. Also, for any future issues that overrun, the more information you provide your customers explaining the delay, the less likely they will jump to conclusions about the quality of your service.

@EU is the symbol I mostly use, so please can you put this near the top of the list of ones to fix.

Many thanks,

Brendan

Jay,

@EU is still corrupt, three weeks after I first reported it. Please can I have an ETA on a fix. At what point will I be justified in describing DTN's service an absolute shambles again? A month, 6 weeks? If you were a customer of a data provider that supplied corrupt data for this long, how would you describe the service?

Many thanks,

Brendan

not only @EU

5 minute tick history for @EU for 2005-04-10 looks "interesting"

2005-04-10 00:35:00,129490.000000,129490.000000,129490.000000,129490.000000,1,0
2005-04-10 00:50:00,129460.000000,129460.000000,129460.000000,129460.000000,1,0
2005-04-10 18:05:00,129490.000000,129490.000000,129490.000000,129490.000000,1,0
2005-04-10 18:10:00,129450.000000,129430.000000,129450.000000,129430.000000,3,0
2005-04-10 18:25:00,129480.000000,129480.000000,129480.000000,129480.000000,1,0
2005-04-10 18:45:00,129450.000000,129420.000000,129450.000000,129420.000000,1,0
2005-04-10 18:50:00,129410.000000,129410.000000,129410.000000,129410.000000,1,0
2005-04-10 19:55:00,129380.000000,129360.000000,129360.000000,129380.000000,1,0
2005-04-10 21:00:00,129420.000000,129400.000000,129420.000000,129400.000000,1,0
2005-04-10 21:25:00,129310.000000,129310.000000,129310.000000,129310.000000,1,0
2005-04-10 21:30:00,129310.000000,129310.000000,129310.000000,129310.000000,1,0
2005-04-10 21:50:00,129360.000000,129360.000000,129360.000000,129360.000000,2,0
2005-04-10 22:25:00,129410.000000,129410.000000,129410.000000,129410.000000,1,0
2005-04-10 22:30:00,129450.000000,129450.000000,129450.000000,129450.000000,1,0
2005-04-10 23:00:00,129520.000000,129520.000000,129520.000000,129520.000000,1,0
2005-04-10 23:05:00,129530.000000,129530.000000,129530.000000,129530.000000,1,0
2005-04-10 23:15:00,129610.000000,129550.000000,129550.000000,129610.000000,1,0
2005-04-10 23:20:00,129600.000000,129580.000000,129600.000000,129580.000000,1,0
2005-04-10 23:30:00,129560.000000,129560.000000,129560.000000,129560.000000,1,0
2005-04-10 23:55:00,129490.000000,129450.000000,129490.000000,129450.000000,2,0


Notice all the gaps. Notice the initial 2 bars when the market was CLOSED. Notice the obviously wrong volume.
Edited by skunk on Apr 12, 2005 at 07:40 AM

Notice the position of the decimal point.

This should be corrected this afternoon/evening

---

Jay Froscheiser
DTN Market Access, LLC.

Thank you for attending to this problem.