Historical tick data seems to be missing for certain symbols on certain days.

Here are some example symbols and days where they are known to have trading volume based on other charting websites, but IQFeed is returning historical ticks as if they didn't:

RIDE:
Available as expected:
Oct 23 2020 and earlier
Nov 2 2020 and later
Missing:
Oct 26 2020 to Oct 30 2020

TSNPD:
Available as expected:
Sept 28 2020 to Oct 22 2020 (any missing days in this range are because there was actually 0 volume)

Missing:
Oct 23 2020 to Mar 26 2021

Hello Napoleon

We can have our data team look into this. What is the description and the exact symbol that you are using for TSNPD?

Thanks
Todd DTN Support

TSNPD is the exact ticker that I successfully used to download tick data in the last few weeks.
History continuing through friday is shown here: https://sg.finance.yahoo.com/quote/TSNPD/history?p=TSNPD
But the symbol has completely ceased to exist as far as IQFeed is concerned.

After I wrote my original post, I found out that there have been ticker changes with TSNP->TSNPD->HMBL
"HUMBL’s stock symbol will change to “TSNPD” on February 25, 2021 and then to “HMBL” on March 26, 2021."
https://www.globenewswire.com/news-release/2021/02/25/2182875/0/en/HUMBL-Inc-Completes-Corporate-Actions.html

From the user's perspective using IQFeed, the symbol just ceased to exist, giving the impression that the data was missing after the last day that I had downloaded. Is there any automated way for IQFeed users to be informed of such ticker changes (preferably in advance)? When downloading historical data for many symbols this can be a tedious manual job to research and keep track of delisted / changed / merged symbols. Rather than each user manually doing that through their own research, it would seem much preferred for it to be part of IQFeed's data. Or if there is a reliable web site with that info that would also help, but all that I have found are very hit and miss.

As for the question about RIDE, that seems to be a different case without any symbol changes, just missing data.

Also the symbol SCR has missing tick / minute data on 2021-02-24. See the attached time and sales screenshot that shows daily data but no minute data

At this time there is not an automated process to inform customers of delisted symbols.

That being said we do update our Symbol Guide before the market opens each morning.

http://iqfeed.net/symbolguide/display.cfm?eight=08011D35.cfm&six=0600036C&title=DTN%20%2D%20U%2ES%2E%20STOCKS%20%2D%20CHANGES

If you would like to contact us at support@iqfeed.net with your IQ Login/User and reference this forum we can enter a Customer Request for an Enhancement.

---

Stephen Shockey
Senior Customer Support Representative and Product Support Specialist

DTN
800-779-7299
support@iqfeed.net

Thank you. That is helpful to know about the symbol guide. Do you have a list of the possible types of changes that I can expect to be in the symbol guide? I.e. does it include delisted, mergers, acquisitions, etc?

Are historical symbol guide files available, or are they lost once overwritten?

Also, I would still appreciate it if your data team could also respond regarding the missing data for RIDE and SCR that are mentioned in the posts above.

Hello Napoleon

We should have a determination regarding the data for the symbols RIDE and SCR tomorrow. The symbol guide does not include specific details on delists etc. After those guide files are overwritten there is not a record of them available. For a more direct interaction customer service is available for chat at www.iqfeed.net from 5pm Sunday to 6pm Friday Central time

Thanks
Todd DTN

Missing data on the symbol RIDE and SCR has been corrected.

---

Stephen Shockey
Senior Customer Support Representative and Product Support Specialist

DTN
800-779-7299
support@iqfeed.net

I see that the RIDE and SCR data is fixed. Thank you!

I'm not sure If I'm understanding the format of the symbol changes file. I attached a screenshot of the one from today to this post for reference so that it will still make sense after the file is overwritten.

The new symbol changes file from today seems to have yesterdays changes in it as well with some modification to the NEW ISSUER fields, which were blank yesterday for BTX. The entries that are leftover from yesterday don't have the *.
Will changes always be included for multiple days and appear without the * after the first day, or is this a special case? How many days will the older entries remain in the file? Can you elaborate on what the * means and the new items database? The reason I ask is because knowing this would help assign the appropriate date to changes if the file wasn't downloaded on the previous day, but one wanted to know what date the older changes were made. If they always stay in the file only one day after the first day, that would mean any entries without the * are from the first day in the date range (03/29/2021) and items with the * are from the end of the date range (03/30/2021). That is true for this particular file, but is that a valid assumption in general?

Thanks for your help.

Actually, on closer inspection, there are still problems with the RIDE and SCR data that was just made available.

For RIDE tick data, the total volume field resets itself counting from 0 + the last incremental volume at 16:30 and again at 17:20 on all 5 days that were previously missing (Oct 26 -Oct 30 2020). On days before and after that week, total volume is never decreasing throughout each day. It should be impossible for total volume to ever decrease unless there is a data error, right?

Also, for the same 5 days of RIDE tick data, queries for ticks on those days return empty over half the time. When querying data for other days or other symbols, I am getting 100% reliable behavior (i.e. never a NO DATA response unless there really is no data). On just these days for this symbol, over 50% of the time I get a NO DATA response. I wonder if the changes haven't propagated to all of your servers possibly? I do not see this reliability issue with the new SCR data.

For SCR on 2021-02-24, there is an inconsistent number of characters after the decimal for Last, Bid, and Ask price (other days before and after this has consistently 4 decimal places). This data sometimes has no decimal at all when the price is an integer number of dollars, and doesn't have the same number of digits after the decimal between Last, Bid, and Ask, even on the same tick. While not technically wrong, it does not match the format of your other data and makes my parsing code fail, even though it has not failed before on many thousands of days of other tick data. I'll fix my parsing to handle missing decimal points for this special case, but it is still very strange that this is the only day I am aware of that has any price data without decimal points and a fixed number of trailing digits.

What time zone are you looking at the data in? What software are you using to look at the data?

---

Stephen Shockey
Senior Customer Support Representative and Product Support Specialist

DTN
800-779-7299
support@iqfeed.net

We have reviewed the symbol RIDE with our developers add Market Data team and at this time we see no issues.

We do not see any issue with the SCR Decimals when looking at tick data on 2/24.

At this time if you feel there are still unanswered or unaddressed questions please contact us at support@iqfeed.net via email or contact us by phone 800-779-7299 or lastly by chat from www.iqfeed.net and the technical support chat will be on the left hand side.

We would like to get more specifics to help answer your question(s) fully.

Thank you

---

Stephen Shockey
Senior Customer Support Representative and Product Support Specialist

DTN
800-779-7299
support@iqfeed.net

I am referring to times in Eastern Time Zone. The RIDE Volume errors were visible using your own Time and Sales app. You have to have extended hours turned on in the settings since they are after 16:00 EST.
I rechecked today and they are fixed. Maybe someone from your data team read my post before you did and snuck in the correction ;)

In case you want to know for preventing it in the future, here is a sample of what was downloaded yesterday (raw text data downloaded using IQML with all parsing turned off): (The last field is volume - note the reset on the first tick after 16:30 EST:
Timestamp,Last,Bid,Ask,LastSize,BasisForLast,TradeConditions,TradeMarketCenter,TickID,TotalVolume
2020-10-26 16:28:49.933475,18.8100,18.8100,18.9000,1,O,874717,26,6139,4959655
2020-10-26 16:28:49.933484,18.8100,18.8100,18.9000,25,O,874717,26,6140,4959680
2020-10-26 16:28:49.933828,18.8100,18.8100,18.9000,69,O,8717,11,1987,4959749
2020-10-26 16:30:21.641880,18.8200,0.0000,0.0000,25,O,874717,26,6141,25
2020-10-26 16:30:21.642753,18.8100,0.0000,0.0000,31,O,8717,11,1988,56
2020-10-26 16:30:21.643587,18.8000,0.0000,0.0000,444,E,1747,26,6142,500

Here are the same ticks downloaded today (identical except for the volume field does not reset):
Timestamp,Last,Bid,Ask,LastSize,BasisForLast,TradeConditions,TradeMarketCenter,TickID,TotalVolume
2020-10-26 16:28:49.933475,18.8100,18.8100,18.9000,1,O,874717,26,6139,4959655
2020-10-26 16:28:49.933484,18.8100,18.8100,18.9000,25,O,874717,26,6140,4959680
2020-10-26 16:28:49.933828,18.8100,18.8100,18.9000,69,O,8717,11,1987,4959749
2020-10-26 16:30:21.641880,18.8200,18.8000,18.8900,25,O,874717,26,6141,4959774
2020-10-26 16:30:21.642753,18.8100,18.8000,18.8900,31,O,8717,11,1988,4959805
2020-10-26 16:30:21.643587,18.8000,18.8000,18.8900,444,E,1747,26,6142,4960249

This reset was there yesterday on all 5 days that you just restored that were missing (Oct 26-30 2020) at the 16:30 EST transition and 17:20 EST transition. They are all fixed as of today.

Also, I retested the NO DATA reliability issue I mentioned yesterday and it is also fixed. I now have perfect reliability when querying these days, whereas yesterday it was failing more than half the time (only on those 5 days for RIDE - not other days or symbols). Maybe it finished propagating to your other servers.

Anyway, it is correct now so no need for further fixes for RIDE. Thank you to whoever fixed it :)

The SCR formatting issues don't appear in time and sales because it appears to apply a fixed number of digits as part of it's display formatting, even if the raw underlying data does not have them. I've already fixed my parsing for SCR so it's not important to me anymore, I just mentioned it in case you want to maintain formatting consistency to prevent parsing errors for your other users. The formatting I am talking about are the raw unparsed characters as sent over the API from IQConnect and logged when queried using IQML. SCR on 2021_02_24 is the only day out of hundreds of thousands I've downloaded that is formatted this way. It is still showing up like this today after redownloading. Here is a sample (Note the inconsistent number of digits on last price ranging between 0 and 4):

Timestamp,Last,Bid,Ask,LastSize,BasisForLast,TradeConditions,TradeMarketCenter,TickID,TotalVolume
2021-02-24 09:30:17.815000,31.6375,0.00,0.00,50,O,87,3,37292,2656
2021-02-24 09:30:20.038000,32,0.00,0.00,200,C,01,3,38668,2856
2021-02-24 09:30:20.099000,31.7974,0.00,0.00,50,O,87,3,38699,2906
2021-02-24 09:30:28.077000,31.5,0.00,0.00,50,O,87,3,41996,2956
2021-02-24 09:30:28.484000,31.64,0.00,0.00,50,O,87,3,42046,3006

Here is a sample from the previous day (and is consistent with all other days). Notice there that even if the price is a whole integer dollar amount, there are always exactly 4 digits after the decimal point for Last Price, and for Bid and Ask:
Timestamp,Last,Bid,Ask,LastSize,BasisForLast,TradeConditions,TradeMarketCenter,TickID,TotalVolume
021-02-23 09:32:34.045000,29.3731,0.0000,0.0000,100,C,01,3,98963,46312
2021-02-23 09:32:34.046000,29.3730,0.0000,0.0000,950,C,01,3,98967,47262
2021-02-23 09:32:36.059000,29.0000,0.0000,0.0000,100,C,01,3,99346,47362
2021-02-23 09:32:36.060000,29.0000,0.0000,0.0000,75,O,87,3,99349,47437
2021-02-23 09:32:44.049000,28.8000,0.0000,0.0000,1,O,87,3,101063,47438

The number of digits in a price should be consistent for a symbol. The items Price Format and Precision in the Fundamental Message will tell you what it should be. According to this, SCR should always have 4 digits in the price, so I don't know why the 9:30:20.038 tick would say "32" instead of "32.0000". or "31.5" instead of "31.5000". I retrieved all available ticks for SCR just now, and all have four digits after the decimal as they should:

2021-03-24 13:42:41.072849,26.3650,50,228222,26.2400,26.4900,730,O,19,87,0,24,
2021-03-24 13:39:34.805258,26.5400,20,226580,26.5100,26.5400,18,O,25,87,0,24,
2021-03-24 13:41:03.333657,26.4000,1,227592,26.2900,26.5100,728,O,19,87,0,24,

I can't check the 23rd right now, because trading markets are open and that is more than 8 days ago. But I will look for any examples of unexpected decimal precision when possible. If you know of any other examples, please let me know.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Yes, I agree with everything you said, except it is February 24th that has the weird formatting, not the 23rd.

I agree that all other days including the last 8 that you can see during market hours are all formatted as expected.

You're right, I misread your message. I'll look on the correct date when I can.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Additional information on the "Symbol Change File"

3) When it says " * DENOTES ITEMS TO BE SELECTED FROM THE NEW ITEMS
DATABASE ", are there multiple databases, one with old items and one with
new? I don't really know what that means unless maybe it is just saying
"this change is new as of today"? is it talking about the * at the
beginning of the OLD ISSUER lines like *PIVOTAL INVESTMENT, or the * after
some symbols like PICC*?

Answer: It is talking about the * at the beginning of the line.


4) in the deletions file, sometimes the comments say ISSUE DELETIONS and
sometimes they say REMOVED FROM LISTING & REGISTRATION. Can you explain
what is the difference between these? I think I know what removed from
listing means, but not ISSUE DELETIONS.

Answer: Nasdaq is using the term Issue Deletions while others are using Removed from listing & Registration.

---

Stephen Shockey
Senior Customer Support Representative and Product Support Specialist

DTN
800-779-7299
support@iqfeed.net