Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I just wanted to say how happy I am with your service. I was able to download the API docs last week and I was able to replicate Interactive Brokers historical bar queries and realtime bar queries over the weekend. That was about one of the fastest integrations that I've ever done and it works perfectly!!!!" - Comment from Jason via Email
"IQFeed version 4 is a real screamer compared to anything else I have seen." - Comment from Tom
"I ran your IQFeed DDE vs. my broker vs. a level II window for some slow-moving options. I would see the level II quote change, then your feed update instantaneously. My broker's DDE, however, would take as much as 30 seconds to update. I am not chasing milliseconds, but half a minute is unacceptable." - Comment from Rob
"IQ feed works very well, does not have all of the normal interruptions I have grown used to on *******" - Comment from Mark
"After all the anxiety I had with my previous data provider it is a relief not to have to worry about data speed and integrity." - Comment from Eamonn
"If you want customer service that answers the phone, your best bet is IQFeed. I cannot stop praising them or their technical support. They are always there for you, and they are quick. I have used ****** too but the best value is IQFeed." - Comment from Public Forum
"Thanks for following up with me. You guys do a great job in tech support." - Comment from Phelps
"For anyone considering using DTN.IQ for a data feed, my experience with the quality of data and the tech support has been very positive." - Comment from Public Forum
"I would just like to say that IQFeed version 4 is running very well and I am very happy with its performance. I would also like to extend a big thanks for the fast and efficient help that I always receive. My questions and concerns are always addressed promptly. Way to go!" - Comment from Josh in CO.
"Thank God for your Data Feed as the only Zippers I see are on my pants (LOL), and no more 200 pip spikes to mess up charts." - Comment from Spiro via Email
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »How to create bar from tick data
Author Topic: How to create bar from tick data (20 messages, Page 1 of 1)

xvz
-Interested User-
Posts: 13
Joined: Mar 28, 2022


Posted: Jul 25, 2022 02:53 AM          Msg. 1 of 20
Hi,

I'm trying to create 1 minute bar from tick data.
I take tick data with "wAAPL" request.

My logic:
1min bar OPEN - I'll take from field Open, from the first tick at current minute
1min bar HIGH - I'll take from field High , from the tick with Max value at current minute
1min bar LOW - I'll take from field Low, from the tick with Min value at current minute
1min bar CLOSE - I'll take from field Close, from the last tick at current minute
1min bar VOLUME - I'll take from field Last Size, I just sum values from all ticks at current minute

Am I doing write? Maybe you can give me some advices? Or there any thing to optimize in my logic ?

And another question:
Why I can't get tick data on premarket session?
Edited by xvz on Jul 25, 2022 at 03:06 AM

DTN_Gary_Stephen
-DTN Guru-
Posts: 295
Joined: Jul 3, 2019


Posted: Jul 25, 2022 08:13 AM          Msg. 2 of 20
There's already a command that will build live bars for you: Streaming Intervals Bars (BW). Connect to port 9400 and send BW,Symbol,Interval in Seconds,Begin Date/Begin Time. This will give you history up to the current moment, then new incoming ticks as they happen. http://www.iqfeed.net/dev/api/docs//Derivatives_StreamingIntervalBars_TCPIP.cfm goes into more detail.

If you want to build them yourself, there are a few considerations:

1. IQFeed only uses Last-Eligible and Extended ticks to build bars, ignoring Other. The Message Contents field in each ticks will tell you which it is.
2. Volume willl not consistently match the sum of all ticks. This is because Volume can include transactions that aren't ticks, like stock options and implied trades.
3. Make sure you know the difference between "Last" "Most Recent" and "Extended":
- “Last” includes only last -eligible trades.
- “Most Recent” includes all trades.
- “Extended” includes on extended trades (also call Form T or after hours trades).

You would have to consider both "Last" and "Extended" data to re-create bars as IQFeed builds them.

4. You can get tick data on premarket session, but you must be watching the "Extended" fields. "Last" includes only last-eligible trades, which do not occur outside of regular trading hours. These distinctions apply to the fields listed in http://www.iqfeed.net/dev/api/docs/Level1UpdateSummaryMessage.cfm. You can view the one(s) you need.

I hope this is all helpful!

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

xvz
-Interested User-
Posts: 13
Joined: Mar 28, 2022


Posted: Jul 25, 2022 09:23 AM          Msg. 3 of 20
Quote: There's already a command that will build live bars for you: Streaming Intervals Bars (BW). Connect to port 9400 and send BW,Symbol,Interval in Seconds,Begin Date/Begin Time. This will give you history up to the current moment, then new incoming ticks as they happen. http://www.iqfeed.net/dev/api/docs//Derivatives_StreamingIntervalBars_TCPIP.cfm goes into more detail.


Thank a lot!!
This is like bomb for me!
I read all documentation , and from the naming I thought that is only for futures and others.
But now I khow, that I can use it for stocks , and it's really good new!

N4Apound
-Interested User-
Posts: 4
Joined: Jun 18, 2012


Posted: Dec 4, 2022 10:02 AM          Msg. 4 of 20
I am also trying to build interval bars from tick data.
My builds NEARLY match iqfeed interval bars, however there is a mismatch when ticks occur whose time label is on the exact interval boundary end time.

I observe (and expect) that iqfeed intervals are INCLUSIVE of the exact interval start time and I expect NON-inclusive of the interval end time. Both cannot be inclusive because resulting interval bars would overlap.

So an iqfeed 1minute interval bar labeled "09:31:00" should include all ticks from 09:30:00.000000 up through 09:30:59.999999.

But there seems to be a problem that if a tick happens to occur at 09:31:00.000000, it is included also in the iqfeed interval bar labeled "09:31:00". But that tick should instead belong only to the "09:32:00" interval, right?

DTN_Gary_Stephen
-DTN Guru-
Posts: 295
Joined: Jul 3, 2019


Posted: Dec 5, 2022 07:13 AM          Msg. 5 of 20
No, that is correct. Any tick with a timestamp between 9:30:00.000000 and 9:30:00.999999 will be part of the "9:30:00" or "9:30:01" bar.

One other thing I should have mentioned in my prior post: that you can specify whether the label represents the beginning or the end of the time period. It defaults to the end, so "10:40:00" represents 10:39:59 to 10:40:00. If you set LabelAtBeginning to 1 in your HIT/HID/HIX historical request (see http://www.iqfeed.net/dev/api/docs/HistoricalviaTCPIP.cfm) then "10:40:00" will represent 10:40:00 to 10:40:01. This choice does not affect the underlying data in any way, just how it is labeled. A second will consist of ticks with timestamps between .000000 and .999999.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

N4Apound
-Interested User-
Posts: 4
Joined: Jun 18, 2012


Posted: Dec 5, 2022 08:08 AM          Msg. 6 of 20
Thanks Gary.

quickTick
-Interested User-
Posts: 53
Joined: Nov 17, 2013


Posted: Feb 12, 2023 12:37 AM          Msg. 7 of 20
I'm not completely clear on this comment:
"It defaults to the end, so "10:40:00" represents 10:39:59 to 10:40:00."

The meaning of the text seems to imply that more precisely, it includes only 10:39:59.000000 to 10:39:59.999999. So that at the upper end, "10:40:00" is just a shorthand for saying "10:39:59.999999", regardless of the labelling direction?

And 10:40:00.000000 is in the interval labeled "10:40:01", if the labelling is at the end ?

The point of my question is to make sure that with labelling at the end, I'd have to ask for
"00:00:01" to "24:00:00" to get the whole day. If 24 is an allowed value, otherwise I'd have to take the next days date and "00:00:00".

I'm not quite sure I'm asking this question in the best possible way. I wonder if it's more complicated. Or more simple.
Edited by quickTick on Feb 12, 2023 at 12:37 AM
Edited by quickTick on Feb 12, 2023 at 12:39 AM

quickTick
-Interested User-
Posts: 53
Joined: Nov 17, 2013


Posted: Feb 12, 2023 02:21 AM          Msg. 8 of 20
I think my previous message needs at least one correction:

I asked: "So that at the upper end, 10:40:00 is just a shorthand for saying 10:39:59.999999, regardless of the labelling direction?"

However I now think that at the upper end, 10:40:00 is a shorthand for 10:39:59.999999 only if the labelling is at the end.

If labelling is at the beginning, it is a shorthand for 10:40:00.999999 .

Strangely that seems to mean that at the lower end, with labelling at the end, 10:40:00 stands for 10:39:59.000000 .

So labelling at the end seems very confusing if one is interested in that detail at all.

The default now seems to be "1" for labelAtBeginning.
For that, the range for the whole day appears to be 00:00:00 to 23:59:59 .
Which will include 23:59:59.999999 .

The specification 00:00:00 to 00:00:00 (next day) would be wrong in either case.

stargrazer
-DTN Guru-
Posts: 301
Joined: Jun 13, 2005

Right Here & Now


Posted: Feb 12, 2023 09:00 AM          Msg. 9 of 20
Maybe think of it differently.

For the range [10:40:00 - 10:41:00) - note the difference in brackets. A square bracket means 'includes', a round bracket means 'exclude'.

The best implementation is 'less than'. Anything 'less than' 10:41:00 goes into the 10:40:00 bucket. You then do not need to worry about the irrationality of 10:40:59.999999 as an endpoint.

A 24 hour period would be: [00:00:00 - 00:00:00), that is it starts on midnight, and includes everything up to but not including the next midnight (less than)
Edited by stargrazer on Feb 12, 2023 at 09:02 AM

DTN_Gary_Stephen
-DTN Guru-
Posts: 295
Joined: Jul 3, 2019


Posted: Feb 13, 2023 11:35 AM          Msg. 10 of 20
Quicktick,

You're conflating two things that aren't really related.

"It defaults to the end" has to do with the label a bar has. A bar from 10:40 to 10:45 can be called either 10:40 or 10:45, depending on what LabelAtBeginning is set to. It just affects what time frame is represented.

No matter what label you use, all bars include everything up to .999999 of the specified ending minute and second. A bar from (start time) to (end time) will include all ticks from (start time).000000 to (end time).999999. So 10:40 to 10:45 would include 10:45:00.999999. This is a standard behavior of bars, and how it handles fractions of a second.

I hope this helps.


Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Edited by DTN_Gary_Stephen on Feb 13, 2023 at 11:35 AM

altmany
-Interested User-
Posts: 71
Joined: Jul 30, 2018

IQML - IQFeed-MATLAB connector


Posted: Feb 13, 2023 11:57 AM          Msg. 11 of 20
Quote: So 10:40 to 10:45 would include 10:45:00.999999

I think you meant that 10:40 to 10:45 would include 10:44:59.999999, not 10:45:00.999999, otherwise the following bar will include overlapped data with this bar (trades that occurred in the first second of 10:45:00) and this will cause data inconsistencies.



Yair Altman
IQML - IQFeed-MATLAB connector
https://UndocumentedMatlab.com/IQML

I am not a DTN employee; my post reflects my personal opinion

Edited by altmany on Feb 13, 2023 at 11:57 AM
Edited by altmany on Feb 13, 2023 at 12:00 PM

DTN_Gary_Stephen
-DTN Guru-
Posts: 295
Joined: Jul 3, 2019


Posted: Feb 13, 2023 01:00 PM          Msg. 12 of 20
altmany, you're right - I was conflating two things that really weren't related.

I was thinking of historical tick requests, which can include a tick in multiple time ranges. For example:

HTT,EQS,20230210 093035,20230210 093036
LH,2023-02-10 09:30:36.446,1.6800,100,200,1.5900,1.7200,808473395,C,7,3D,0,10,
!ENDMSG!,
HTT,EQS,20230210 093036,20230210 093037
LH,2023-02-10 09:30:36.446,1.6800,100,200,1.5900,1.7200,808473395,C,7,3D,0,10,
!ENDMSG!,

The historical tick request from 9:30:35 to 9:30:36 technically includes everything from 9:30:35 to 9:30:36.999. So A tick at 9:30:36.446 is included in both time ranges.

A HIT request doesn't include the milliseconds after the stop point, because that would put the same data in multiple bars, as you point out. You can see that only one bar contains the 1.68 tick (I picked a low-volume stock for this example):

HIT,EQS,1,20230210 000000,20230210 180000
LH,2023-02-10 09:30:38,1.7200,1.7200,1.7200,1.7200,300,100,0,
LH,2023-02-10 09:30:36,1.6800,1.6800,1.6800,1.6800,200,100,0,
LH,2023-02-10 09:30:00,1.6000,1.6000,1.6000,1.6000,100,100,0,
!ENDMSG!,

I hope I didn't make this even more confusing.


Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Edited by DTN_Gary_Stephen on Feb 13, 2023 at 01:01 PM

quickTick
-Interested User-
Posts: 53
Joined: Nov 17, 2013


Posted: Feb 14, 2023 02:00 PM          Msg. 13 of 20
> A 24 hour period would be: [00:00:00 - 00:00:00), that is it starts on midnight,
> and includes everything up to but not including the next midnight (less than)

That actual implementation appears to be 'less than or equal' for the interval specification.

The following example is from the documentation, asking for the days between the 19th and the 30th. It includes both the 19th and the 30th:

Request: HDT,GOOG,20080919,20080930,,,TESTREQUEST,2500,0<CR><LF>
TESTREQUEST,2008-09-30,425.10,392.32,395.98,400.52,8824587,0,<CR><LF>
TESTREQUEST,2008-09-29,423.51,380.71,419.51,381.00,10764969,0,<CR><LF>
TESTREQUEST,2008-09-26,437.16,421.03,428.00,431.04,5292751,0,<CR><LF>
TESTREQUEST,2008-09-25,450.00,435.98,438.84,439.60,5026917,0,<CR><LF>
TESTREQUEST,2008-09-24,445.00,430.11,430.34,435.11,4242795,0,<CR><LF>
TESTREQUEST,2008-09-23,440.79,425.72,433.25,429.27,5206644,0,<CR><LF>
TESTREQUEST,2008-09-22,454.13,429.00,454.13,430.14,4409119,0,<CR><LF>
TESTREQUEST,2008-09-19,462.07,443.28,461.00,449.15,10008745,0,<CR><LF>
TESTREQUEST,!ENDMSG!,<CR><LF>

So, my current understanding is this:

If you intend to ask for a 24 hour period by using 00:00:00 and 00:00:00 as BeginTime and EndTime, and you do it for two consecutive days, you actually get the 00:00:00 interval twice.

Depending on LabelAtBeginning, you get either the interval before midnight twice, or the intervall after midnight. Although each intervall begins with .000000 and ends with .999999 microsconds. Only the unspecified values are 'less than', whereas the specified interval is 'less than or equal'.

So if you are querying for minute intervals in a 24 hour period, you need to query either
00:00 to 23:59 or
00:01 to 00:00 next day, depending on LabelAtBeginning.

Although in each case, the bar before midnight ends with 23:59:59.999999 , and the bar after midnight ends with 00:00:59.999999.

Please correct if incorrect in any detail.
Edited by quickTick on Feb 14, 2023 at 02:01 PM
Edited by quickTick on Feb 14, 2023 at 02:02 PM
Edited by quickTick on Feb 14, 2023 at 02:04 PM
Edited by quickTick on Feb 14, 2023 at 02:05 PM

DTN_Gary_Stephen
-DTN Guru-
Posts: 295
Joined: Jul 3, 2019


Posted: Feb 15, 2023 08:36 AM          Msg. 14 of 20
If you're requesting tick data, you should request from 000000 to 235959 (or 000001 to 000000) to get an entire 24-hour day today with no overlap. This can be demonstrated with an example:

HTT,QCL#,20230214 000000,20230215 000000
LH,2023-02-15 00:00:00.788827,78.37,1,11813,78.37,78.38,11887050,C,112,01,2,15,
LH,2023-02-15 00:00:00.788827,78.37,1,11812,78.37,78.38,11887050,C,112,01,2,15,p

LH,2023-02-14 23:59:53.336951,78.38,1,11809,78.38,78.39,11887046,C,112,01,2,15,
LH,2023-02-14 23:59:40.433622,78.38,1,11808,78.38,78.39,11887042,C,112,01,2,15,
(lots of ticks)
LH,2023-02-14 00:00:25.045924,79.26,1,11300,79.26,79.27,11415859,C,112,01,2,14,
LH,2023-02-14 00:00:02.727061,79.26,2,11299,79.25,79.26,11415857,C,112,01,1,14,
LH,2023-02-14 00:00:00.784311,79.25,1,11297,79.25,79.26,11415852,C,112,01,2,14,
LH,2023-02-14 00:00:00.784311,79.25,1,11296,79.25,79.26,11415852,C,112,01,2,14,
LH,2023-02-14 00:00:00.755201,79.26,1,11294,79.26,79.27,11415849,C,112,01,2,14,
LH,2023-02-14 00:00:00.755201,79.26,1,11293,79.26,79.27,11415849,C,112,01,2,14,

!ENDMSG!,

As you can see, a request from 000000 to 000000 includes the first second after midnight for both days. A request from 000000 to 235959 includes up to 235959.999999, but nothing after 000000.0000000.

HTT,QCL#,20230214 000000,20230215 235959
LH,2023-02-14 23:59:53.336951,78.38,1,11809,78.38,78.39,11887046,C,112,01,2,15,
LH,2023-02-14 23:59:40.433622,78.38,1,11808,78.38,78.39,11887042,C,112,01,2,15,
(etc)
!ENDMSG!,

Again, this only applies to tick-level requests. Bars will not overlap in this way:

HIT,QCL#,60,20230214 235700,20230215 000300
LH,2023-02-15 00:03:00,78.37,78.35,78.36,78.36,11878,30,0,
LH,2023-02-15 00:02:00,78.37,78.35,78.36,78.36,11842,7,0,
LH,2023-02-15 00:01:00,78.37,78.37,78.37,78.37,11833,6,0,
LH,2023-02-15 00:00:00,78.37,78.35,78.37,78.36,11822,9,0,
LH,2023-02-14 23:59:00,78.38,78.36,78.36,78.38,11809,19,0,
LH,2023-02-14 23:58:00,78.36,78.34,78.35,78.34,11785,19,0,
LH,2023-02-14 23:57:00,78.35,78.35,78.35,78.35,11761,6,0,
!ENDMSG!,

The 00:00:00 bar includes only the ticks from 00:00 to 00:00.999999.


Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Edited by DTN_Gary_Stephen on Feb 15, 2023 at 08:43 AM

quickTick
-Interested User-
Posts: 53
Joined: Nov 17, 2013


Posted: Feb 16, 2023 04:22 AM          Msg. 15 of 20
Within a single request, there is no overlap.

DTN_Gary_Stephen
-DTN Guru-
Posts: 295
Joined: Jul 3, 2019


Posted: Feb 16, 2023 06:22 AM          Msg. 16 of 20
I was going by your statement "If you intend to ask for a 24 hour period by using 00:00:00 and 00:00:00 as BeginTime and EndTime, and you do it for two consecutive days." In this scenario, if it is an HTT request, you would get the first second after midnight as part of both requests. HIT requests will not have this problem.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

quickTick
-Interested User-
Posts: 53
Joined: Nov 17, 2013


Posted: Feb 16, 2023 12:44 PM          Msg. 17 of 20
Quote: I was going by your statement "If you intend to ask for a 24 hour period by using 00:00:00 and 00:00:00 as BeginTime and EndTime, and you do it for two consecutive days." In this scenario, if it is an HTT request, you would get the first second after midnight as part of both requests. HIT requests will not have this problem.

--- Original message by DTN_Gary_Stephen on Feb 16, 2023 06:22 AM
Your example doesn't show an HIT request using 00:00:00 and 00:00:00 as BeginTime and EndTime. Only the example HTT request does so.

So far, I would expect two HIT requests for two consecutive days, using 00:00:00 and 00:00:00 as BeginTime and EndTime, to both include the 00:00:00 bar that is between these two days. I believe this according to the example in the documentation.

However within a single request that crosses midnight, I would not expect any problem within the middle of the request, as your example shows. I would not expect such a problem for HTT either. The bars within a single request never overlap. Or if they did, that would a bug and completely different. I don't think we have a bug here, just that the documented behavior is easy to misunderstand when deciding which labels to use for BeginTime and EndTime, if you actually want the first bar to be *after* the BeginTime label, and the last bar to be *before* the EndTime label, which is spresumably the case if you use 00:00:00 to 00:00:00 intending to get 24 hour period. Since then, depending on the value of LabelAtBeginning, both the first and the last bar will be either before, or after each midnight.

quickTick
-Interested User-
Posts: 53
Joined: Nov 17, 2013


Posted: Feb 16, 2023 01:13 PM          Msg. 18 of 20
If so, there might be a value in having a third option for LabelAtBeginning, where the label for BeginTime would be at beginning, and the label for EndTime would be at end.

drjubbs
-Interested User-
Posts: 1
Joined: Mar 24, 2023


Posted: Today @ 10:55 AM          Msg. 19 of 20
I'm having some trouble which I think is related to this topic. I'm making a HIT request for @ES#, 60 second interval.

I notice that 0:00:00 is missing for the day and after-hours trading starts at 18:01:00. This would be consistent with LabelAtBeginning being set to 0, but I tried using the default (which should be 1) and explicitly setting this argument to 1, and neither changed the result.

Am I doing something wrong? What am I missing?

DTN_Gary_Stephen
-DTN Guru-
Posts: 295
Joined: Jul 3, 2019


Posted: Today @ 04:43 PM          Msg. 20 of 20
The only possibility is normal for intervals to be missing if no trading occurred during that time period. You don't mention what day you inquired about, but it's possible there were no trades between 00:00-00:01 and 18:00-18:01 on that day. If you post the request here, I can dig deeper.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist
 

 

Time: Fri March 24, 2023 8:53 PM CFBB v1.2.0 31 ms.
© AderSoftware 2002-2003