The forum search functionality hasn't worked for me in a few months, but if I remember correctly, there were some archived posts regarding the handling of dividends on end-of-day equities data. I seem to recall that the policy was not to adjust for dividends except when someone requested it or it looked funny on a chart. Obviously, this is wasn't the most rigorous of approaches. Has this policy been changed to something more methodical?

Here is a post I made a couple days ago about this, but it applies so I am copying it over here for reference. We will look at the search functionality, thanks for the reminder on that. Let me know if there are questions though on the post below.

Tim

***
The original thread = http://forums.dtn.com/index.cfm?page=topic&topicID=3251.
***

Our rule is that we don't process dividends, and we haven't processed any for a number of years, but on a very rare occasion a dividend will come along that causes a chart to become unusable without historical modification. If that happens, and customer demand dictates, then we will make a manual adjustment of the historical data (daily only), these are the only time dividends are processed.

That said, we do not have a place that can be queried against for what dividends were processed when. Though as seldom as we do them, literally years apart, should we do another in the future I will try to make a post to the data and content forums about it. So if you subscribe to those forums, you will be alerted of these changes when they occur going forward, and the thread I create will be available thereafter for historical reference.

Hopefully this helps alleviate some of your concerns, but let me know if you have any questions or if I can expand on any points made.

***

Tim- this isn't directed at you because I'm sure this policy has long been in place, but this untraceable "sometimes" approach is absurd and irresponsible. It makes DTN eod data useless for backtesting. I'm happy to expound upon all the reasons why but I think they're rather obvious. You guys typically go about things in a thoughtful manner but this is a glaring exception. There are two choices in dealing with this properly:

1. Adjust for everything (splits, stock divs, cash divs) in an economically consistent manner (i.e. total returns)

2. Don't adjust at all and keep a database of split and div data that will allow users to make their own adjustments

I suppose there hasn't been much grief given about this in the past since Yahoo data was an easy alternative, but now that their data has fallen into disarray, this is a high priority item. I don't think accurate end of day pricing data is asking too much from a subscription feed.

Will you guys please fix this?

Thank you jonnyb,

I appreciate your post. We will revisit the policy. At this time I can not guarantee a change will be made but I think it deserves our consideration to other alternatives. Feel free to contact me directly if you have questions or comments.

Thank you for your business,

Sincerely,

Mike Driscoll
402.255.8018
mike.driscoll@dtn.com

Thank you Mike, appreciate it. I'll follow up with you end of next week if I don't hear back before then.

Mike- We're coming up on a month so wanted to follow up. Any progress on this front?

Hi jonnyb
Thank you for the follow up. We have revisited the policy, but it is not closed. At this time we decided to provide a location for changes. We will be recording changes going forward. Im including the link to the thread where we will update changes. We will continue to review this thread and topic for improvements. http://forums.dtn.com/index.cfm?page=topic&topicID=3800

While I understand that it does not satisfy your concerns completely. At this time we have not committed to providing all historical data changes. I would be happy to set up a time with you to review your concerns. Please feel free to contact me directly anytime.

Sincerely,
Mike Driscoll
mike.driscoll@dtn.com
402-255-8018

Thanks for the reply. Don't mean to sound harsh, but this really does nothing to address the fact that your EOD data is inconsistently adjusted and therefor unusable/dangerous. Even if that weren't the case, posting changes going forward serves no purpose; in order to keep an accurate database one would need continually check a message board and propagate any new postings through a time series...completely impractical. Candidly, I wouldn't waste Tim's time with it. All that said, I do appreciate your attention to the matter and I will give you a call this afternoon or Monday. Have a good weekend if I don't speak to you.

hi johnnyb,

I agree we are looking at a couple of different options for the record keeping that makes more sense. Thanks for your comments and look forward to discussing it further.
have a great day
Mike

Hi Mike-

It's been a few months since my initial post, so I wanted to check in and see how you've decided to address the inconsistent EOD data issue?

Thanks

Hi jonnyb

We are still providing a posting of when we adjust the data going forward. At this time there is not going to be a change to that.

Thanks
Mike

Not sure what to say other than I'm perplexed and thoroughly disappointed by this decision. DTN acknowledges a glaring flaw in their data and decides to shrug their shoulders at it. I won't say anything else on the matter other than this is irresponsible and inconsistent with the diligence I've come to expect from the IQFeed staff.

jonnyb,

I have been dealing with this issue for several years. What I did to clean everything up is use Yahoo to download all daily data up to a certain point. Then on each day afterwards I download the close from IQFeed and maintain a db of dividends, splits and reorgs. Its a pain but the only way to be 100% accurate I think. Otherwise IQFeed intraday data is all non-adjusted so you could pick an intraday point for your historical analysis. You should still be able to download data from Yahoo. If you have trouble I have some C# code that does it.

---

Brook
Edited by shatinstrats on May 20, 2014 at 02:39 PM

Brook-

Thanks. That's essentially what I'm doing myself with Yahoo + Java. The problem is that Yahoo is usually late updating divs that newly went ex and there's always a handful of times per year when their price data fails to update at all for a day (sometimes two). I can't complain though because it's free. What I can complain about is that my IQFeed bill is $221/month and they've made a deliberate decision to look the other way in the presence of blatantly erroneous data. I really don't get it...if you and I are able to pull together a properly adjusted data set from the internet for free, how hard could it be for them to put one together? Perhaps they don't realize the perils that exist with unadjusted EOD equities data? At best it's useless, at worst dangerous...but now I'm repeating myself. Hopefully someone with a trading background at DTN will step in and convince the decision makers to go in a more sensible direction on this.

Please excuse my ignorance when posting on this topic, since I'm new to IQFeed and market data feeds in general. While it is easy to see why and how splits need to be considered when showing or processing continuous historical data, it is not as easy to see how exactly this should be done for dividends. However I expect this may become a future concern of mine as well.

In any case, there seem to be reasons for DTN to keep the history unadjusted for dividends (I don't know yet what is done for splits). And there seem to be ways to adjust the data in client software with additional information. Perhaps it will be a valid compromise to ask that IQFeed supplies this information in a separate API request, such that it supplies, for example, a list of timestamps and amounts of any dividends that were paid? Perhaps it will also include splits and similar events, even in case splits are already adjusted for.

Such a list would surely be very useful with stocks like AAPL, for example. (Upcoming 7-1 split, and dividends in a popularly traded stock.)

I wouldn't expect such a list to have lots of data, so it shouldn't be a problem for the servers other than the initial programming effort.

Sorry for my ignorance again, in case this wouldn't really solve the problem.

Thanks for the post quickTick. Your point is well taken (by me at least) that raw unadjusted EOD price data coupled with separate API calls for historical splits and divs would be ideal and give the user the option to adjust/not adjust however he/she sees fit. In the meantime (forever?), I would be very cautious using the current EOD data set as it's not consistently adjusted under any interpretation. From what I gather from old message boards posts here, prices were adjusted for divs "sometimes"...like when a chart looked funny or someone on the staff thought the div was big enough to merit an adjustment....good luck to anyone trying to work with that. For anyone putting their capital to risk based on IQ data, this is far more than just an academic point. Clearly, by my whining in this thread, I find this decision/policy/apathy maddening.

Any updates in the works for EOD data, splits, divs?

Also, did the API fee go up this year? (didn't think this deserved its own thread)

Good afternoon,

I believe there was an increase this year as of January first, and there are plans for where we want to see the EOD data go, but there is no detail or timeline that I could give out yet on this.

Tim
Edited by DTN_Tim Walter on Nov 12, 2015 at 05:16 PM

Checking in on the EOD data, divs, and splits improvements...anything in the works?

We do have this on our product roadmap. However, I don't have a lot to share regarding timing or the exact implementation at this time. We are in the research stages to make sure we can do this effectively and accurately. The challenge we run into is not only providing accurate data going forward, but being able to adjust the existing historical data we provide to ensure consistency as well as backward compatibility.

---

Jay Froscheiser
Vice President, Active Trader Products

Ok, fair enough. In my mind, this is simply three new api calls (raw prices, divs, split ratios) and it's done. The current EOD data/calls can remain as is.

At I high level it seems easy. The requests/API calls aren't the issue. The main issue is that we have deep historical data stored one way currently. To provide it 3 different ways means we need to find ways to adjust the back data to accommodate the new styles. If we didn't have some much history to deal with, we could easily add 3 new methods to request data and it would work on data from today forward (not something you or our customers would want nor expect). We want to do it right, so it will take a bit more time.

---

Jay Froscheiser
Vice President, Active Trader Products

If I'm understanding you correctly, you're trying to go back and undo the manual one-off adjustments that were made in this past? I can see why that would be tricky and I always assumed you were going to start fresh with a new dataset of raw EOD prices.

Perhaps as an interim step you would consider adding calls for historical divs and splits only? That would allow a user to recreate raw EOD prices from intraday bars and then adjust as they see fit. I get that the EOD history is likely deeper, which is why you're not taking this approach yourself.

Apologies if I've made some bad assumptions about the issue.

Any update on this?

Good morning,

There are tasks in front of development to improve this, some of which are already completed, but I do not know the schedule that will be used to release these changes. We should have historical split and dividend information available via the API soon though, so that will be a good start.

Tim

Is there a timeline for split and dividend information being available? I drop my stock data subscription previously because I didn't trust the daily data it produced. However, once IQFeed offers daily data with corresponding split and dividend information, that would allow me to calculate both unadjusted and adjusted (split+div) daily prices, I would sign up again.

Thanks for keeping this topic going!

James

Hello,

It is a project we continue to work on and have completed several pieces of, but there are challenges, as you know, to unraveling our data to an un-adjusted state and I don't have an ETA on when those steps will be completed.

Tim

Having a routine to get ex-dates and dividends would be a great start. Is it still being worked on?

This project continues to be worked on as part of a larger overall historical data initiative. We don't have a release date/timeline to share, but it is in development by our engineering teams.

---

Jay Froscheiser
Vice President, Active Trader Products

Are you able to post a list of equity symbols which have had dividends applied so at least users know which EOD prices to avoid?

Did IQ Feed not save the original unmodified prices or something?

If you change your OHLC bars to Open, High Low, Close, Adjusted Close that would be great.

I agree that it is dangerous and irresponsible to publish historical daily OHLC data without mentioning that some arbitrarily set of undisclosed symbols have been adjusted to reflect some, but not all, historical price adjustments.