|
|
Home
Search
Register
Login
Recent Posts
»Forums Index
»Archive (2017 and earlier)
»Data and Content Support
» Why getting historical data is slow?
|
|
| Author | Topic: Why getting historical data is slow? (4 messages, Page 1 of 1) |
 cunparis -Interested User- Posts: 29 Joined: Jan 1, 2009 |
Posted: Mar 21, 2009 10:38 AM
Msg. 1 of 4
When getting historical data, I see the number of seconds going up to 27 seconds without an update. Here is an update from my application:
2009-03-14 22:44:30:027 (IQ) Data.Bars.GetBarsNow: instrument='AIPC' from='2009-02-23' to='2009-03-13' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 2009-03-14 22:44:57:403 (IQ) Data.Bars.GetBarsNow: instrument='MYGN' from='2009-02-23' to='2009-03-13' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 I'm noticing it again today with ES. I try to get 1 minute data for 2008 and it takes a very long time. Since the DTN client is reporting 20+ seconds since last update, I think there are two possibilities: 1 - my software is not requesting the data very fast 2 - DTN takes a while to give me the data Any idea how to find out which one? |
|
DTN_Jay_Froscheiser -VP, Product Operations- Posts: 1746 Joined: May 3, 2004 
DTN IQFeed/DTN.IQ/DTN NxCore |
Posted: Mar 21, 2009 10:51 AM
Msg. 2 of 4
The "Seconds since last update" stat isn't impacted by history requests. This is only reset by streaming market data or timestamps within the feed. I pulled @ES# data from Jan 1 2008 through current (417K 1 minute bars) in 58.9 seconds on a slower machine from my house.
I pulled AIPC from Feb 23-March 13 as you have in your request above (4941 1 minute bars) in 1.8 seconds. I pulled MYGN from Feb 23-March 13 as you have in your request above (5810 1 minute bars) in 1.6 seconds. Based on this, you can see that the time it takes to pull down historical data is tightly related to the amount of data you are requesting. Keep in mind that @ES will take longer than stocks for the same time period because it trades nearly 24 hours per day as opposed to stocks which will have only 12-14 hours per day max. Jay Froscheiser DTN - Trading Markets |
|
cunparis -Interested User- Posts: 29 Joined: Jan 1, 2009 |
Posted: Mar 21, 2009 11:28 AM
Msg. 3 of 4
For me @ES# jan 1 2008 to present was taking over 10 minutes and I eventually gave up and started downloading it in smaller chunks. I think the problem is that my application is storing the history in its database, which is Microsoft Access <hic>.
thanks for timing it on your side, that helps to narrow it down. |
|
taa_dtn -DTN Evangelist- Posts: 154 Joined: May 7, 2004 |
Posted: Mar 22, 2009 05:31 PM
Msg. 4 of 4
Probably not the cause of your immediate problem, just something to keep in mind: It is possible for history requests to time out, perhaps due to network issues between you and the history servers, and your code needs to be prepared to handle that.
I normally fetch a large amount of tick history data some hours after the market closes. On Friday there were 5 timeouts during that process, a fairly typical number for me. Allen |
|
|