Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I had always used ******* but for the past 2 weeks have been trying DTN IQFeed. Customer support has been extraordinary. They call just to make sure your problem hasn't recurred." - Comment from Public Forum
"Boy, probably spent a thousand hours trying to get ******* API to work right. And now two hours to have something running with IQFeed. Hmmm, guess I was pretty stupid to fight rather than switch all this time. And have gotten more customer service from you guys already than total from them… in five years." - Comment from Jim
"I just wanted to let you know how fast and easy I found it to integrate IQFeed into our existing Java code using your JNI client. In my experience, such things almost never go so smoothly - great job!" - Comment from Nate
"I just wanted to tell you what a fine job you have been doing. While *******, from what I hear, has been down and out, off and on, IQ feed has held like a champ this week." - Comment from Shirin
"Awesome response, as usual. It is a sincere and refreshing pleasure to do business with DTN, compared to your competition." - Comment from Ryan
"DTN feed was the only feed that consistently matched Bloomberg feed for BID/ASK data verification work these past years......DTN feed is a must for my supply & demand based trading using Cumulative Delta" - Comment from Public Forum Post
"I am a hedge fund manager here. It’s funny, I have a Bloomberg terminal and a Bridge feed, but I still like having my DTN feed!" - Comment from Feras
"Interactive Brokers tick data was inconsistent, so I have switched to using DTN exclusively. It is great to no longer have to worry about my datafeed all day long." - Comment from Philippe
"Thank God for your Data Feed as the only Zippers I see are on my pants (LOL), and no more 200 pip spikes to mess up charts." - Comment from Spiro via Email
"For anyone considering using DTN.IQ for a data feed, my experience with the quality of data and the tech support has been very positive." - Comment from Public Forum
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »Product Support »DTN.IQ Client Software Support »Historical Pre-Market Volume
Author Topic: Historical Pre-Market Volume (15 messages, Page 1 of 1)

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Dec 27, 2021 04:27 PM          Msg. 1 of 15
Hi,

My objective is to understand for the last 90-180+ days, which stocks had the highest pre-market trading volume. I'm currently using TradingView's screener of 4000+ nasdaq stocks and downloading it each day, building it up over time - which is obviously taking a lot of time...

I was hoping to ask please if there is an efficient way of obtaining this via iqfeed? I thought about using the API to pull the data for all NASDAQ stocks over the last 90-180 days, but wasn't sure if there was a better approach (and if I'd get flagged for pinging so many stock tickers).

Many thanks in advance for any guidance you can provide

ChainsawDR

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Jun 8, 2022 09:16 AM          Msg. 2 of 15
I still have this use case that I'm trying to address - it looks like I will need to pull the data from the API. Does anyone have any advise on how to structure the iqfeed request? for example, if wishing to pull the MSFT pre-market volume on 6/7/22, is there a way of requesting the volume for 9.29am & 59seconds?

Thanks in advance for any help (wishing to avoid making too many requests to the API)

DTN_Gary_Stephen
-DTN Guru-
Posts: 267
Joined: Jul 3, 2019


Posted: Jun 8, 2022 12:09 PM          Msg. 3 of 15
Depends on what you mean by "avoid making requests." There are two approaches I can suggest:

1. The EDS market summary report. It won't be timed at 9:29:59 exactly, but depending on what data you want to collect, it might meet your needs. It is available on the History/Lookup port, and the syntax is:

Report,Security Type,Group ID,Date

Security Type is the number representing the type of securities you want, such as 1 for Equities and 8 for Futures. The list of codes can be retrieved from the SST command.

Group ID is the group of exchanges you want, like 5 for NASDAQ or 34 for CME. The list of Group IDs can be retrieved from the SLM command. Each exchange contains the group it belongs to: “1,NGM,Nasdaq Global Market,5,NASDAQ” would be part of Group 5.

Date is the date in YYYYMMDD format. Date is omitted for the 5MS command.

This returns a report containing the following fields:

Symbol,Exchange,Type,Last,TradeSize,TradedMarket,TradeDate,TradeTime,Open,High,Low,Close,Bid,BidMarket,BidSize,Ask,AskMarket,AskSize,Volume,PDayVolume,UpVolume,DownVolume,NeutralVolume,TradeCount,UpTrades,DownTrades,NeutralTrades,VWAP,MutualDiv,SevenDayYield,OpenInterest,Settlement,SettlementDate,ExpirationDate,Strike

2. It is possible to perform a tick history request which includes "filter time", meaning it will only return data from a certain time window each day. This is usually done to retrieve ticks data only from 9:30 to 4:00. But you could make a request like this:

HTT,AAPL,20220601 000000,20220609 130000,,092900,093000

This will request all ticks from June 1, midnight through June 9, 1 PM Eastern time. But there is a BeginFilterTime and EndFilterTime, 9:29 and 9:30 respectively. So the response will only include ticks during that time:

2022-06-02 09:30:00.000812,147.8000,26,1515398,147.7900,147.9300,910,O,18,87,0,2,
2022-06-02 09:30:00.000804,147.8400,2,1515372,147.7900,147.9300,909,O,18,87,0,2,
2022-06-02 09:29:59.825505,147.8100,100,1515370,147.7500,147.9300,4682,E,5,17,0,2,
2022-06-02 09:29:59.825502,147.8100,100,1515270,147.7500,147.9300,4681,E,5,17,0,2,

And so on. The highlighted number is the Total Volume. This approach may not be optimal for what you want to collect, but it does work.

By the way: the only hard limits on API requests is 50 individual historical requests per second, and your symbol limit for Level 1 watches (typically 1400). Beyond that it's a matter of how much data your code can process.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Jun 15, 2022 09:40 AM          Msg. 4 of 15
Thank you very much Gary. Yes option 2 looks like it will solve the problem. Thanks again for your help!

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Jun 15, 2022 09:08 PM          Msg. 5 of 15
Hi Gary,

Hoping this may be a quick one... I'm trying to execute in Python the command you shared. During testing I'm trying to download for 4 stock symbols, however requests for GOOG are not returning. Any ideas if I'm doing something wrong here, such as missing specifying the exchange or similar when making the HTT request?

# iqfeed.py

import sys
import socket


def read_historical_data_socket(sock, recv_buffer=4096):
"""
Read the information from the socket, in a buffered
fashion, receiving only 4096 bytes at a time.

Parameters:
sock - The socket object
recv_buffer - Amount in bytes to receive per read
"""
buffer = ""
data = ""
while True:
data = sock.recv(recv_buffer).decode('utf-8')
buffer += data

# Check if the end message string arrives
if "!ENDMSG!" in buffer:
break

# Remove the end message string
buffer = buffer[:-12]
return buffer

# iqfeed.py

# iqfeed.py

if __name__ == "__main__":
# Define server host, port and symbols to download
host = "127.0.0.1" # Localhost
port = 9100 # Historical data socket port
#syms = ["SPY", "AAPL", "GOOG", "AMZN"]
#syms = ["SPY", "AAPL", "GOOG", "AMZN"]
syms = ["AMZN"]

# Download each symbol to disk
with open("all.csv", mode='a+', encoding="utf8", newline='') as f:
for sym in syms:
print("Downloading symbol: %s..." % sym)

# Construct the message needed by IQFeed to retrieve data
message = "HTT,%s,20220609 000000,20220610 130000,,092900,092959\n" % sym

# Open a streaming socket to the IQFeed server locally
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))

# Send the historical data request
# message and buffer the data
sock.sendall(message.encode())
data = read_historical_data_socket(sock)
sock.close

# Remove all the endlines and line-ending
# comma delimiter from each record
#data = "".join(data.split("\r")) note: this is the original that doesn't include the symbol per line.
#data = data + ","
data = ("%s" % sym).join(data.split("\r"))
data = data.replace(",\n","%s \n" % sym)[:-1]

# Write the data stream to disk
# f = open("%s.csv" % sym, "w") note: this is the original that saves each in an individual file

# Move read cursor to the start of file.
f.seek(0)
# If file is not empty then append '\n'
countdata = f.read(100)
if len(countdata) > 0:
f.write("\n")
# Append text at the end of file
f.write(data)

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Jun 16, 2022 03:52 PM          Msg. 6 of 15
or to simplify the question, any ideas why requests for "HTT,GOOG,20220609 000000,20220610 130000,,092900,092959\n" do not work, but requests for others like "HTT,AMZN,20220609 000000,20220610 130000,,092900,092959\n" do work?
Edited by ChainsawDR on Jun 16, 2022 at 03:53 PM

DTN_Gary_Stephen
-DTN Guru-
Posts: 267
Joined: Jul 3, 2019


Posted: Jun 16, 2022 04:15 PM          Msg. 7 of 15
I'm guessing you didn't set the protocol. The S,SET PROTOCOL command should always be set when you first connect. (If you don't do this, IQFeed API assumes you want to use the much older protocol 4.9, which does not always accept the same commands as the current protocol 6.2 does. Your GOOG command fails if I don't SET PROTOCOL 6.2 first. AMZN succeeds. To be honest I'm not 100% sure why they don't both fail, but setting protocol first should render the problem moot.

There are other possible causes I can think, but none would apply to the commands as you list them here:

- Tick data older than 8 days cannot be retrieved during trading hours. (Your GOOG command requests data no older than 7 days)
- You lack the necessary exchange subscriptions. (GOOG is a NASDAQ symbol which is included in the IQFeed core package, so everyone has it)
- If you have delayed data, you can't make history requests that would return that day. In other words, if your NASDAQ data is the standard 10-minute delay as opposed to realtime, you can't make historical requests that would circumvent that. If you try to request ticks from 9:20 to 9:29 at 9:30, you'll get no data back. The same request would work at 9:40, because the 10-minute delay would have passed by then. (You're requesting a day's worth of data so it can't all be unavailable due to delay)
- Make sure the command is being assembled correctly. (The code suggests that it is. And the same command works fine for AMZN)

Send S,SET PROTOCOL 6.2 to port 9100 after you connect and try your code again. If that doesn't sort it, I may want to review your IQConnect.txt log file.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Jun 16, 2022 06:01 PM          Msg. 8 of 15
Thanks Gary. So the steps I'm currently performing are:
1) In Windows, open "IQFeed Applications" > Charts (which opens the connection to iqfeed from my computer)
2) In my Python IDE, I run the script shown above.

I have downloaded and installed the latest 6.2 version of the client and tried my existing code again, but no joy for GOOG again. TBH I'm not an experienced developer and I adapted the above script from a tutorial I found online. Do you know please if:
A) there is a way to set the protocol to 6.2 in step #1 above? Or
B) there's a method for adapting the above Python script to achieve this?

I've tried a few methods opening a socket to send that message, but the script stalls. I understand if not able to provide coding support (tried looking for sample files but unfortunately python isn't one of the languages listed).

Many thanks again

ChainsawDR

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Jun 17, 2022 11:00 AM          Msg. 9 of 15
In case it helps. I tried setting the protocol as shown below, which generated the error further below:

Code:

# Construct the message needed by IQFeed to retrieve data
message = "S,SET PROTOCOL,6.2\nHTT,%s,20220616 000000,20220616 130000,,092900,092959\n" % sym
#also tried multiple variations such as message = "S,SET PROTOCOL,6.2<CR><LF>\nHTT,%s,20220616 000000,20220616 130000,,092900,092959\n" % sym

# Open a streaming socket to the IQFeed server locally
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))

# Send the historical data request
# message and buffer the data
sock.sendall(message.encode())
data = read_historical_data_socket(sock)
sock.close
print("First part of data:" + data[0:100])


Error:

Downloading symbol: SPY...
First part of data:E,6.2 is not a valid protocol.,
2022-06-16 09:29:59,370.4600,100,3609350,370.4300,370.4600,80847523

DTN_Gary_Stephen
-DTN Guru-
Posts: 267
Joined: Jul 3, 2019


Posted: Jun 17, 2022 12:20 PM          Msg. 10 of 15
To answer question A: No. You must explicitly tell IQFeed API which protocol you want to use each time you connect, or it will assume the oldest one, 4.9. All you should need to do is send it a command, like your code does:

message = "S,SET PROTOCOL 6.2"
sock.sendall(message.encode())

or something like:

self._send_cmd("S,SET PROTOCOL,6.2\r\n" % protocol)

You will get a response S,CURRENT PROTOCOL 6.1 confirming this has been successful. Then your HTT command for GOOG should work correctly. I'm pretty sure not doing this is your problem, since i can reproduce the failure of the GOOG command when I don't set it.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

DTN_Gary_Stephen
-DTN Guru-
Posts: 267
Joined: Jul 3, 2019


Posted: Jun 17, 2022 12:45 PM          Msg. 11 of 15
Also, I thought of a way to improve the request I suggested. If you're defining "pre market volume" as "the total volume in the last tick before 9:30 am", you can actually do this more efficiently. Make this addition:

HTT,AAPL,20220601 000000,20220609 130000,1,092900,092959

The 1 is MaxDatapoints. This will give you only one datapoint that fits the description and filter. The default order is newest to oldest, so this will give you the very last tick before 9:30. That way you don't have to retrieve all the ticks in that minute.

Note also that the EndFilterTime is 092959. This will include that entire second, up to 9:29:59.999999, which I suspect is what you want. If you make the end time 093000, ticks as late as 9:30:00.999999 will be included in the results.

I should also point out that this approach requires that some trading have occurred between 9:29 and 9:30. For heavily-traded symbols like the ones in your examples, this is a safe assumption. But for lesser-traded symbols, this approach may not always work.


Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Edited by DTN_Gary_Stephen on Jun 17, 2022 at 03:24 PM

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Jun 17, 2022 05:22 PM          Msg. 12 of 15
Firstly, thanks for all of your help with this Gary. Secondly, love your last update - great idea! To confirm, will it always choose the latest tick in the range? Also, am I correct in thinking that for a multi-day request like 06/15 to 06/16, it will only pick the latest across all days, or would it return the latest per day? (if the former I can just use python to cycle through each day).

Coming back to the main problem, here's the summary of what I'm seeing...
If I update my message to send an OLD protocol like 5.1...

message = "S,SET PROTOCOL,5.1\r\nHTT,%s,20220615 000000,20220616 130000,,092900,092959\n" % sym 


... then I get a good response back: S,CURRENT PROTOCOL,5.1 - plus I get all of the info for GOOG (still testing, but specifying an old protocol seems to be working). However if I take this exact same code and the only change I make is to update the protocol from 5.1 to 6.2, then it gets weird. A) I get this error response: E,6.2 is not a valid protocol. B) I now get 1 row of data for GOOG back (before specifying protocol, I used to get 0 rows back). This feels super weird - sending 6.2 is getting an error, but at the same time it is meaning I get 1 row of data for GOOG vs 0 rows of data.

All in all, one request I'd love to make if possible would be to add a basic sample code for Python to the IQ Developer Support section. I know a lot of experienced devs use C++ & Java, but Python is quite a common entry level language so hoping it could be supported to a MVP level like opening a socket, setting protocol, submitting a request (to take off the table situations like this when I'm wondering if its my coding or the system - for this it looks like a problem with the system, both with GOOG data responding differently and with accepting the latest protocol incorrectly).

Will crack on with trying to get it working using the old 5.1 approach, and loop back if no joy (but please shout if you already know there'll be an issue with using the old protocol).

Thanks again for your help Gary, really appreciate it!

ChainsawDR
Edited by ChainsawDR on Jun 17, 2022 at 05:24 PM

altmany
-Interested User-
Posts: 69
Joined: Jul 30, 2018

IQML - IQFeed-MATLAB connector


Posted: Jun 18, 2022 12:51 PM          Msg. 13 of 15
Quote: Firstly, thanks for all of your help with this Gary. Secondly, love your last update - great idea! To confirm, will it always choose the latest tick in the range? Also, am I correct in thinking that for a multi-day request like 06/15 to 06/16, it will only pick the latest across all days, or would it return the latest per day? (if the former I can just use python to cycle through each day).

Coming back to the main problem, here's the summary of what I'm seeing...
If I update my message to send an OLD protocol like 5.1...

message = "S,SET PROTOCOL,5.1\r\nHTT,%s,20220615 000000,20220616 130000,,092900,092959\n" % sym 


... then I get a good response back: S,CURRENT PROTOCOL,5.1 - plus I get all of the info for GOOG (still testing, but specifying an old protocol seems to be working). However if I take this exact same code and the only change I make is to update the protocol from 5.1 to 6.2, then it gets weird. A) I get this error response: E,6.2 is not a valid protocol. B) I now get 1 row of data for GOOG back (before specifying protocol, I used to get 0 rows back). This feels super weird - sending 6.2 is getting an error, but at the same time it is meaning I get 1 row of data for GOOG vs 0 rows of data.

All in all, one request I'd love to make if possible would be to add a basic sample code for Python to the IQ Developer Support section. I know a lot of experienced devs use C++ & Java, but Python is quite a common entry level language so hoping it could be supported to a MVP level like opening a socket, setting protocol, submitting a request (to take off the table situations like this when I'm wondering if its my coding or the system - for this it looks like a problem with the system, both with GOOG data responding differently and with accepting the latest protocol incorrectly).

Will crack on with trying to get it working using the old 5.1 approach, and loop back if no joy (but please shout if you already know there'll be an issue with using the old protocol).

Thanks again for your help Gary, really appreciate it!

ChainsawDR
Edited by ChainsawDR on Jun 17, 2022 at 05:24 PM
--- Original message by ChainsawDR on Jun 17, 2022 05:22 PM
To use protocol 6.2, you must of course install the IQFeed 6.2 client. If for example you only have a 6.0 client, then you will not be able to use the 6.1 or 6.2 protocols, only 6.0 or older.
You can download the latest IQFeed client from http://www.iqfeed.net/index.cfm?displayaction=support§ion=download

Yair Altman
IQML - IQFeed-MATLAB connector
https://UndocumentedMatlab.com/IQML

I am not a DTN employee; my post reflects my personal opinion

ChainsawDR
-Interested User-
Posts: 26
Joined: Jan 4, 2019


Posted: Jun 18, 2022 01:36 PM          Msg. 14 of 15
Thanks altmany. Yes understood. I originally had an older client, however updated to 6.2 prior to my recent posts. Unfortunately I'm still getting the issue when setting 6.2 or not setting a protocol at all, however specifying 6.1 seems to be working absolutely fine.



File Attached: 62client.png (downloaded 57 times)

DTN_Gary_Stephen
-DTN Guru-
Posts: 267
Joined: Jul 3, 2019


Posted: Jun 20, 2022 10:23 AM          Msg. 15 of 15
Quote: Firstly, thanks for all of your help with this Gary. Secondly, love your last update - great idea! To confirm, will it always choose the latest tick in the range? Also, am I correct in thinking that for a multi-day request like 06/15 to 06/16, it will only pick the latest across all days, or would it return the latest per day? (if the former I can just use python to cycle through each day).

Coming back to the main problem, here's the summary of what I'm seeing...
If I update my message to send an OLD protocol like 5.1...

message = "S,SET PROTOCOL,5.1\r\nHTT,%s,20220615 000000,20220616 130000,,092900,092959\n" % sym 


... then I get a good response back: S,CURRENT PROTOCOL,5.1 - plus I get all of the info for GOOG (still testing, but specifying an old protocol seems to be working). However if I take this exact same code and the only change I make is to update the protocol from 5.1 to 6.2, then it gets weird. A) I get this error response: E,6.2 is not a valid protocol. B) I now get 1 row of data for GOOG back (before specifying protocol, I used to get 0 rows back). This feels super weird - sending 6.2 is getting an error, but at the same time it is meaning I get 1 row of data for GOOG vs 0 rows of data.

All in all, one request I'd love to make if possible would be to add a basic sample code for Python to the IQ Developer Support section. I know a lot of experienced devs use C++ & Java, but Python is quite a common entry level language so hoping it could be supported to a MVP level like opening a socket, setting protocol, submitting a request (to take off the table situations like this when I'm wondering if its my coding or the system - for this it looks like a problem with the system, both with GOOG data responding differently and with accepting the latest protocol incorrectly).

Will crack on with trying to get it working using the old 5.1 approach, and loop back if no joy (but please shout if you already know there'll be an issue with using the old protocol).

Thanks again for your help Gary, really appreciate it!

ChainsawDR
Edited by ChainsawDR on Jun 17, 2022 at 05:24 PM
--- Original message by ChainsawDR on Jun 17, 2022 05:22 PM
Quote: I correct in thinking that for a multi-day request like 06/15 to 06/16, it will only pick the latest across all days


Yes. The MaxDatapoints parameter applies to the entire command. If I ask for 1 datapoint of a two-week period, I get one datapoint total, not one datapoint per day:

HTT,AAPL,20220612 000000,20220620 130000,1,092900,092959
LH,2022-06-17 09:29:59.992136,130.0600,511,1955010,130.0600,130.0700,5783,E,5,17,0,17,
!ENDMSG!,

Omitting the MaxDatapoints feature will return the 9:29-9:30 time period for all days in range. Which may have other benefits: if you want to confirm the volume of the very last tick before trading begins, this may not be the very last tick before 9:30:00.000000. You can collect all ticks in a minute (or any number of seconds) and look to the Basis For Last datapoint (the E in the above example) to identify the final trade that was Extended as opposed to C (qualified) or O (other). It depends on how precisely you're defining "last."

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist
 

 

Time: Thu August 18, 2022 12:30 AM CFBB v1.2.0 31 ms.
© AderSoftware 2002-2003