||Jan 4, 2019 08:08 PM
||Dec 27, 2021 04:27 PM
||Dec 28, 2021 05:14 PM
ChainsawDR has contributed to 17 posts out of 20453 total posts
(0.08%) in 1,119 days (0.02 posts per day).
20 Most recent posts:
My objective is to understand for the last 90-180+ days, which stocks had the highest pre-market trading volume. I'm currently using TradingView's screener of 4000+ nasdaq stocks and downloading it each day, building it up over time - which is obviously taking a lot of time...
I was hoping to ask please if there is an efficient way of obtaining this via iqfeed? I thought about using the API to pull the data for all NASDAQ stocks over the last 90-180 days, but wasn't sure if there was a better approach (and if I'd get flagged for pinging so many stock tickers).
Many thanks in advance for any guidance you can provide
In case anyone else has this issue in future, there's more info on workaround steps on the NT8 forum thread here: https://ninjatrader.com/support/forum/forum/ninjatrader-8/platform-technical-support-aa/1178301-corrupted-tick-data-iqfeed-downloads
Thanks for looking into this Todd
In case it helps, these are the stock symbols (all nasdaq or nyse) that were corrupted during the overnight download:
I then deleted all historical data and redownloaded for these symbols, which fixed the issue for all except the following 5:
I've just noticed that for GPS, the corrupted data on the original download was for different days (not 9/13) so I do think it'll be a NT8 issue (but if you could please confirm that'd be great). Many thanks
Re-Posting on this topic as its causing some frustration. When downloading historical data from iqfeed using NinjaTrader 8, I'm frequently seeing issues with the tick data (unsure if its a problem with iqfeed, NT, or user error). Yesterday I set off a download of ~6 months of tick data for ~500 stock symbols, and today when testing using that data there were tick data issues with ~50 of them (obvious issues such as having 1MM trade volume for the same microsecond time stamp). What I have been doing as a workaround is to delete the data for the affected symbols and redownloading them. However when I just did this a moment ago, I'm still seeing the same issue for some of the symbols. A good example is the stock GPS...
I setoff a download for GPS Tick, Minute & Day data for Bid/Ask/Last from 4/1/21 to 9/30/21 (in NinjaTrader>Import Historical Data>Download.
The tick data for 9/13/2021 looks corrupted (minute data looks fine).
The screenshot attached shows tick data and 1min data side by side for GPS. In the tick data there are ~2MM+ trades in GPS with the timestamp 3:57:00:438, which 1min data for 3:57 has ~88k volume.
As I mentioned earlier, I'm not sure if the issue is at iqfeed's end or NT8's end, but hoping you can please look into the data from iqfeed's side to rule out a data issue please.
Many thanks in advance
Edited by ChainsawDR on Nov 11, 2021 at 09:23 PM
Thank you both. I was able to recreate the issue for another stock at a later time, and clicking 'reload historical data' updated the tick data. Stephen your comment on timezone is likely an apt one. I'm on pacific time and at one point I changed it to Eastern time. I suspect the data I'm looking at may have an issue because of this. I will purge and reload. Thanks for your help with this.
adding tick screenshot
I use Iqfeed with NinjaTrader 8. I'm seeing material differences between tick and 1 min data, and unsure if I should contact Iqfeed or NinjaTrader (presume Iqfeed as its a data issue).
Please see attached screenshots for VUZI on 1/20/21. On a 1min bar the High reaches $9.69 at 11:47, On tick data the high reaches $10.13. If you look at the 1min chart for 15mins either side, it gets nowhere close to $10. It's clear that for the same time and the same stock, pulling tick and 1min data is telling different stories. Please could you take a look at the data for VUZI to see if there is an issue, and which source of truth is correct?
I am building an algorithm that auto-trades with tick and 1min data, so obviously very nervous about this issue and guarding against future instances. Do you have any advice please on how to guard against repeat issues?
I think I know the answer to this but wish to avoid assuming so hoping its a QQ: Within the Time & Sales report, When Trade Conditions contains 'Oddlots' I understand that these are individual shares (e.g. Inc Vol = 10 & Trade Conditions = ODDLOT/FORMT means that there was 10 shares). When 'oddlot' is not a trade condition, does this mean the 'Inc Vol' is showing lots of 100 shares, or still showing individual shares please? E.g. Inc Vol = 600 & Trade Condition = FORMT, doe this mean 600 shares or 60,000 shares?
Attaching an image with the same question, restated in case it helps clarify.
Many thanks in advance
Thanks for looking into it Todd. I'll look into our systems to see if we're doing any unexpected transformations or gaps fills that might be causing this. Thank you
Adding a clearer version. Attached is the 1 sec OHLCV data for BBY on 02/03/2020. Higlighted in yellow is a drop from $86 close price to $85.65, then back to $86. Neither the tick or 1min data from 'Time & Sales' shows any drop down to $85.65. My system triggered a signal on this 1 sec OHLCV data that wouldn't be triggered with the other iqfeed dataset. Unsure of the reason for the discrepancy, or wondering if I have a bug in my data retrieval of 1 sec OHLCV data.
I'm retrieving the historical 1 second OHLCV data for backtesting, however I'm seeing some odd spikes in the data that I don't see when I use the 'IQFeed Time & Sales' application to view tick and/or 1min data. It's causing me to doubt whether I should pull/use the historical 1 sec OHLCV or whether I need to start pulling tick data and compiling this myself. I've attached an image that puts the 1sec vs tick vs 1min side by side and have highlighted the 85.39 1sec OHLCV that I can't reconcile with the tick or 1min data. Please could anyone let me know if I am misinterpreting this?
Thanks in advance for any help
Edited by ChainsawDR on Apr 12, 2020 at 06:22 PM
Thank you Stephen
Hi Stephen, please could you kindly confirm if the output of the Common Shares Outstanding is in 000’s? The documentation doesn’t mention it but looking at the output for companies like AAPL & AMZN I think its missing three zeros in the output but would like to confirm please.
Hoping to ask for some advice on an automated trading system setup please. I’m using iqfeed already and now trying to decide whether to use an off the shelf backtesting and live trading service (such as Ninja Trader) vs a mix of open source systems (such as PyAlgoTrade and IBridgePy) and was hoping for some advice from more practiced hands.
Short version of the question: I’m a little confused as to how flexible some of the off the shelf systems are at using external/custom metrics and trading (interval) frequency. After having a less than amazing experience with a developer trying to build a custom solution I’m looking at off the shelf software. Is anyone aware of a backtesting and live trading software system that is: standalone software that can be hosted on AWS (or a VPS), can live trade with interactive brokers, make live trading decisions based on DTN Iqfeed data (rather than IB feed data), backtest rules (event) based on DTN Iqfeed data, incorporate custom metrics within backtest & live rules, and operate on a 1 sec ‘heartbeat’?
Longer version of the question: Some considerations and requirements for the setup I’m looking for:
1. I’m not a developer by trade. I’m learning Python but I don’t want to wait 6-18months to become adept enough to put live a trading system that I can trust enough with real money. From a systems perspective, whether it’s an off the shelf or open source system, I’m going to need to hire a developer to help configure a system that I can live trade with for the first time (and then evolve the rules myself). From this perspective I’m leaning towards an off the shelf system as I won’t need to worry about bugs as much, and upgrades are presumably easier to manage than iterating open source versions.
2. From a rules perspective, I’m confident in being able to write my own trading logic that runs in a system (when to buy, when to sell etc).
3. I’m a Product Manager & Business Analyst by trade. I can write system design documents and requirements, and have hired developers via upwork in the past (so open source powered systems are feasible).
4. I want to (must Requirement) keep my rules private and only use standalone software that I can host on AWS (which I’m very familiar with). This rules out cloud based systems like quantconnect
5. I’ve already hired a developer who has helped automate the retrieval of 1 sec & 1 min OHLCV data for 500 US Equity Stock symbols from DTN Iqfeed, and created some custom metrics that I want to be able to use within a trading rule. An example metric would be ‘count of stocks that closed the interval higher than it opened’ (which would be used in conjunction with other rules such as moving average crossover). I’ve been using jupyter notebooks, pandas and matplotlib to run analysis and hypothesize some rules that I now want to backtest and hopefully put live.
6. I’d prefer to (should Requirement) continue using DTN Iqfeed for making live trading decisions as the data quality is good and I’ve read it’s better quality than most broker feeds like interactive brokers.
7. I want to (Must Requirement) use interactive brokers as my style of trading needs low transaction fees and I eventually want to expand to more markets.
8. Back testing must be event based so I can easily flip from backtesting to live trading (or ideally paper trading before live trading but not essential)
9. I’d prefer (should Requirement) any software to use Python as the rule coding logic as it’s more transferable (rather than being locked in to a language that only works on one system).
As a next step (if no joy from this forum) I was intending on emailing various trading software providers to ask if they can satisfy the above requirements, but as I’m sure many will say they are the best, I was hoping to either learn from this trading community or be educated on my requirements if they are too restrictive.
Thanks in advance
Edited by ChainsawDR on Aug 18, 2019 at 03:57 AM
Thank you altmany, that’s very helpful! I’ll post a separate question for #3.
I'm hoping to please ask for some advice from the forum members who are more experienced than me regarding some novice/basic system design questions (as I'm new to building an algorithmic trading system from scratch). I’ve 3 questions (#1-#2 are relevant together, #3 I could post as its own thread if preferred but didn’t want to clog up the forum with lots of novice questions so grouping in one thread).
Context: The end result I'm working towards is a live trading system that uses minute bars to make trading decisions (in future I may go down to N second bars). I've signed up for DTN IQFeed and have consumed historical minute data for my tickers (using request: "HIT,%s,60,20180601 075000,,,093000,160000,1\n" % sym) so I think I’m now in good shape with the 'backfill' piece of work and currently building a backtester.
Q1: I'm now struggling with the design aspect of keeping the minute data updated (every minute) and being able to use it for future paper & live trading. As I understand it, the most common option is to consume live tick data and compile minute bars from this, every minute. Are there any other options people recommend? For example, I wasn't sure if it'd be possible to consume 'historical' minute data, every minute (using the same HIT request format). If so, do you know how frequently the historical data is updated? For example, if I make a request for 11.03am-11.04am data at 11.04am+1sec, would the data be available then? I’ll never be running a HFT system where millisecond data is needed for decision making.
Q2: I can’t discern from the documentation if it’s possible to request and retrieve historical level 2 data, does anyone know if this is possible and what command to use? If it’s not available, then I suppose that dictates needing to consume and compile tick level data? (signed up for Nasdaq level 2 data)
Q3: I’ve read a lot of forum threads regarding how Linux isn’t properly supported and requires wine to launch. I’ve got it working in quite a manual way currently so hoping if I outline my approach, others might be able to point me in the right direction for an automated alternative so that I can cron job turning iqfeed on and off... I’ve setup an AWS EC2 instance with Ubuntu 18.04 LTS, installed vncserver and an Ubuntu desktop GUI that I access via tightvnc, installed wine and iqfeed via the GUI. To launch my iqfeed python file I start vncserver via an ssh terminal, then connect to the GUI via tightvnc, navigate to the wine DTN folder and manually launch iqconnect and iqlinklauncher, clicking ‘startiqlink’ and ‘connect’ on each pop up. Finally, from the ssh terminal I launch the python file. I’ve tried a whole bunch of different methods for launching iqconnect and iqlinklauncher from the terminal in headless mode but each has their own errors - I can state each option attempted and errors faced, but if anyone is able to share the terminal commands they use to launch without a GUI then I can try to recreate and debug via the error prompts (knowing that it’s a solvable problem).
Thanks in advance for any help you can provide
Edited by ChainsawDR on Jan 4, 2019 at 08:57 PM