I have encountered major history errors in my program when retrieving certain files that do not have particular time periods filled in. I verified this behavior using the history.exe and HistoryCOM.exe programs supplied in the SDK

This is a serious error and I suspect a remnant of the past history hang-ups not being fully fixed. This is with the latest client 2.3.0.6.

TICK.Z does not have weekly or monthly data available - daily and interday are there. When one tries to get weekly of monthly data on this symbol the following occurs:

1.0 History.exe No response to weekly/monthly, however you can still get daily or interday data. When you exit the program the IQConnection Manager hangs - I have to use the Windows Task Manager to shut it down, even then it takes 10 to 15 seconds.

2.0 HistoryCOM.exe No response on weekly or monthly in fact the program itself becomes unresponsive and you cannot get daily data afterwards. Closing it, the IQConnection Manager hangs like 1.0 above.

For both programs, if you only retrieve daily or interday data then all is ok, and exit is normal.

FMAGX, a mutual fund - slightly different behavior when you try to get interday or monthly data:
1.0 History.exe Gets daily/weekly ok - no response on monthly. Interday gives an "!ERROR! Unknown field specifier" Exiting, the IQConnection Manager is NOT hung.

2.0 HistoryCOM.exe Gets daily/weekly ok - no response on monthly and the program is inoperative like in TICK.Z - Interday request gets the !ERROR! Unknown field specifer return. On exit, regardless of request types the IQConnection Mgr does not hang!

I offer this test so you can reproduce it there. There appears to be a serious communication issue in the client or server. All requests should have a response. None should hang the system. Also, if new messages like the !ERROR! is added to the history return it should be documented. In the past '..Error' and '-Err' were embedded in history error messages.

Please document ALL error messages that can come back in a request, especially if these are new! The !ERROR! format is great if it were used everywhere to make it consistent.

I have no idea as to how History.exe and HistoryCOM.exe connects to the client (TCP or COM) however you can see the difference. I use TCP and this condition renders my application dead in terms of getting further history once this error has been encountered like in the TICK.Z example. Also, the IQManager does not go away at exit and most users do not see that and when my program restarts strange results occur as the IQManager is mostly inoperative.

Thanks,

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

David,

Thanks for posting such detailed information! I am able to recreate this, and will work on a fix. Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

New history error on new listings

On a new listing on the day it first appears there is no data on the server. Requesting history data on it does not return any data or any error messages. Symbols that are invalid or truely not on the server, like XYZ return " !ERROR! Symbol not found" and all works OK after that.

The symbol that caught this error is GAN - yesterday it had no data and no error message, and after trying to get it my history connection would not get any other history. Today it is fine as it has one day of history.

I hope this helps you catch one more subtle error in the history retrieval system.

GL!

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

Natalie

I have been testing 2.3.0.7 and encounter a similar problem although not symbol related.

If you request historic data from multiple threads (2) via com, for 100 symbols (most times the hang will occur before 50 are processed), IQConnect will hang at some point, never at the same place or symbol. It appears to be a timing issue, IQConnect becomes completely unresponsive with no cpu usage as if it is in a deadlock. In every case both threads continue to work normally after having requested data and waiting for OnDayCompleted.

Reverting back to 2.3.0.1 the same routine runs perfectly. I have not tried any of the versions in between.

Also, when 2.3.0.7 does return historic data, the time between the request and the callback to OnDayCompleted is considerably longer, probably, if the routine did complete, would take at least 4 times longer to process the same series of symbols.

Regards

Leo

Leo,

Thanks for the post. It looks like you are requesting daily historical data. How many day, or does it matter? Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

I thought this was just me, or a bad connection, but perhaps it is something within iqfeed. I run about 60 simultaneous socket connections to obtain historical day data. I ran first with 2 days, and had it stop in the S's somewhere. I then tried with 1 day for optionable equities, about 2500 symbols all told, it, too, stopped in the S's somewhere. I had to do a followup run with symbols starting where the previous run left off to finish the collection.

In followup to my previous post, I'd like to mention that I do handle !ERROR! and !ENDMSG! responses.

For requesting historical data, I initially created 60 threads to handle 60 requests simultaneously. Once one thread request completes, I reuse it for the next request.

I then changed this to 200 threads.

The response back to this was that I processed 2554 symbols. Out of 200 thread requests, 58 threads did not complete, even after waiting another 5 minutes to see if anything would trickle in.

In versions prior this this .7 release, I could do this type of processing, and all threads would complete.

Is there a new message type I need to handle, or are my threads simply not getting the final !ENDMGS! indicator?

FYI: It has been my experience that if there is some error between the DTN client and the server in this version (as discussed in the OP) there is no completion of a history request - no error message. I monitor all of my sockets to check for delayed responses. I suspect some sort of hung condition - the connection is unuseable from there on. A serious error.

My hope is that we will be seeing a solution in a matter of days.

GL!

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

Thanks for your patience, folks. Right now, I'm working on a fix for David's issue. It may be causing the other issues, but I'm not sure yet. I'll let you know what I find out. Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

Natalie

Yes the requests are for day data. I don’t think the request details make a difference, but fyi for all the requests the no. of days was less than 30.

Regards

Leo

Thanks, Leo. This issue is on my list!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

Hi all,

IQFeed v2.3.0.8 was just put on the website, and it includes a fix for David's issue. Could you test it for me to confirm that it works ok? Leo, I didn't specificially address your issue (yet), but can you test to see if it is still an issue? Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

I've tried v2.3.0.8. It does not appear to fix the issue of incomplete socket sessions.

Ok, thanks for checking. You issue is still on my list of items to look at. Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

Hi Natalie

The problem is still there. Out of 10 attemps to process the data, during 6 attempts, 1 thread remained hung due to createdispatch (iq_api) never returning, in which case, the remaining thread will succeed in processing the symbol list without iqconnect hanging.

During the 4 times both thread successfully established a connection, iqconnect hung after processing 16, 44, 27 and 47 symbols. All requesting 22 days of data.

The problem with createdispatch not returning was also present in 2.3.0.7 although happened a lot less often.

If you want, contact me directly and I will provide you with a program and instructions allowing you to set up a test environment which should enable you to debug the problem.

Regards

Leo

I'm wondering if there might be something on the server side that was changed. I rolled back to iqfeed .3 and am still getting the same problem of hung threads. I'm pretty sure I had good success with multiple threads with that version. Anyone experiencing anything similar?

In addition, with the .8 version, I believe I was getting some corrupted lines back with extra spaces or characters or termination characters. I havn't been able to dig into it, but my software provides funny error lines. I'm pretty sure this is all meaningless to everyone, including me, but I thought I'd throw it out there just in case someone can fathom some hidden meaning. My software resyncs properly on errors --!ERROR!-- in v.3, but has problems with v.8.

stargrazer,

As far as the corrupted lines in 2.3.0.8, that is my fault. I forgot to include those changes in the release notes. Here is the info you need:

Changed termination characters for error messages:

Old formats:
NONE_MESSAGE = "\r\n!NONE!\r\n";
SYNTAX_ERROR_MESSAGE = "\r\n!SYNTAX_ERROR!\r\n";
ERROR_MESSAGE = "\r\n!ERROR! ";

New formats:
NONE_MESSAGE = "!NONE!";
SYNTAX_ERROR_MESSAGE = "!SYNTAX_ERROR!\r\n";
ERROR_MESSAGE = "!ERROR! ";

Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

Hello,

At present I'm only a DTNIQ/IQFeed user, although I'm very interested in the developer API when IQFeed finally runs stable.

With my following post I just want to confirm that backfill requests often hang endlessly. This is a SERIOUS problem and should be put on top of the priority list. It often makes IQFeed almost useless.

I'm using Amibroker to access the DTNIQ feed, so I do not know the API details yet. Because the developer of Amibroker is a very good programmer, however, I think his comment is credible that the deadlock problem (when trying to backfill) must be within the IQ connection manager.


My own question is why the IQ connection manager does not trigger a timeout when a backfill request hangs. It appears that the IQ connection manager simply waits forever without attempting a retry.

Following below are some excerpts from the Amibroker mailing list:



Date: Wed Jul 27, 2005 2:43 pm
Subject: IQFeed backfilling

I have 2 main problems using IQFeed with AmiBroker. (...)

procedure of scanning/backfilling is stopping several times. It
means it backfill several symbols and then stops. No timeout nothing,
just stops. Very often on index symbols (like DJUSAS.X DJUSRA.X and so
on), but when running on stocks only it stops several times too.



Date: Wed Jul 27, 2005 3:32 pm
Subject: Re: [amibroker] IQFeed backfilling

> Does anybody else having problems with backfilling/collecting intraday
> data from IQFeed ? Or similar issues with IQfeed ? What to do ?

Yes I do, unfortunately the only way I have found to deal with this problem
is to deactivate the plugin, then reactivate the same and restart my
exploration.

I have noticed that this happens more often when I am using other
applications, so i suggest not to browse the net while
downloading/backfilling form IQFeed.



Date: Fri Jul 29, 2005 3:27 pm
Subject: Re: IQFeed backfilling tech917
Offline Offline

I use IQFeed too and can confirm this problem. When trying to
backfill data, Amibroker's IQFeed plugin often waits endlessly.
Because no timeout is triggered, the backfill process gets
effectively stopped. It would be much better to attempt a retry
if the plugin does get not a response from IQFeed within a certain
time period, but I do not know whether this can be technically
implemented with the IQFeed API. Maybe the problem arises from
limitations of the IQFeed API. By comparison, however, Quotetracker
appears to run much more smoothly with IQFeed!

Maybe Thomasz can explain why his IQFeed plugin triggers no timeout
and doesn't attempt a retry.

The problem is SERIOUS and makes Amibroker often almost useless in
connection with IQFeed. Nevertheless, the reason I keep an
DTN/IQFeed subscription is that IQFeed has a much higher symbol limit
than eSignal.


From: "Tomasz Janeczko" <amibroker@...>
Date: Fri Jul 29, 2005 3:50 pm
Subject: Re: [amibroker] Re: IQFeed backfilling amibroker

Hello,

IQFeed recently released a new version of API (beta) that
is supposed to fix many issues present in previous versions.

We will see when I build a new version of the plugin using new API.

As for
> Maybe Thomasz can explain why his IQFeed plugin triggers no timeout
> and doesn't attempt a retry.

well, you probably don't know it, but IQFeed's own Connection manager
tends to lockup inside so, once one request blocks the other one made
after timeout often hangs as well and retries do not help. The only
thing that help in such case is restarting Windows. And I have seen
exactly the same problem when running IQFeed with QuoteTracker. It may
occur less often because QuoteTracker backfill length is very limited -
maximum 10 days (less data is requested and request is less likely to
fail) compared to 120 days in AmiBroker.

As I wrote above IQFeed confirmed many issues and worked on fixing them.
New IQFeed API promises some improvements - we will see.

Best regards,
Tomasz Janeczko
amibroker.com

Edited by iqfeeduser on Aug 17, 2005 at 10:37 AM

Hi all,

I understand that the errors on this thread are serious, which is why I have been working with the the people who have posted these issues. The 2 outstanding history issues mentioned here were only reported week or too ago. So, please be patient while I work on them.

As for the timeout question, that appears to be the problem. Thanks!

---

Natalie Hannan DTN Market Access, LLC.

Natalie:

Today using version 2.3.0.8 I ran into the 'hung' state again and maybe this will help you get at it as it is repeatable. Symbol PDGT is a thinly traded stock and I suspect does not have any history on it. It is a legitimate symbol. A request for history results in a hung history client and I can no longer get any history without re-start.

Feedback on ".8": It appears that the other issues that I reported in my first posts have been addressed with the use of return error code for the condition. I have not been able to hang the client with invalid symbols or for history requests that have no data for a given period (like weekly for tick.z data) Thanks for fixing this.

GL on the hung client! (no pun intended )

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

David,

I just tried to recreate the "hung state" issue with symbol PDGT. I used the VC++ HistoryCOM and HistorySocket examples and did not receive the error. Can you tell me what app and request you receive the hung state with? Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

The hung state also occurs when only history requests are sent for symbols (stocks) which definitely have a history. It appears that the problem is not dependent on specific symbols, otherwise it would easily be reproducible. Backfill requests can work successfully for a specific symbol, but sometimes they can produce the "hung state".

PS: Nathalie, I didn't want to put pressure on you with my previous post. I understand that bugs can be hard to find. It's amazing however that the problem has not been reported much earlier. Already with the predecessor of IQFeed v2.3.0.1, the problem occurred.

Edited by iqfeeduser on Aug 17, 2005 at 02:10 PM

more notes:

It appears that the problem occurs with higher probability when there is load on the IQFeed servers, ie. during market hours or shortly before the US stock market opens. During other times the hung state happens less often, although even then it can occur.

Further, from the user perspective it seems that the problem happens more often when there are other applications which use bandwidth. But even when IQFeed is the only application using bandwith, the deadlock can happen on a DSL connection.

Nevertheless, I already wondered whether the communication between the IQFeed server and connection manager is really reliable, but it appears that TCP is used as transport protocol and not UDP (if I'm not mistaken).
Edited by iqfeeduser on Aug 17, 2005 at 02:33 PM

Natalie:

I just tried it again and now I get an "!ERROR! Invalid Symbol" - although incorrect in itself (it is a valid symbol) the system does not hang as it has closed the loop and returned an error message. I have no idea why there was no error result earlier - however this points to the issue at hand and that is server response.

I have not looked at the incoming message stream (to the client, I have a sniffer that can show all communications) for some time to see what comes back on a request but I suspect nothing if everything is not totally proper server side. The server needs to give something back. Maybe it should also have an immediate "ACK" to the client so it 'knows' it is not timing out and then have some sort of "!ERROR! No response" message back to the 3rd party software after some reasonable time and also server side have a clear function that aborts a response after a given period and then resets itself so it can function again. Maybe a cancel message back to the client. This requires an ID for requests / responses - a very good idea for reliable communications. Time outs can be a bear to establish however if too quick it only forces a re-request or go on to next. At least there is a closed loop.

I hope my prattle is of some help.

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

David:

Glad to hear that you got an error message, and the client did not hang.

The reason why you got an "invalid symbol" message is because PDGT is a Pink Sheet Stock, and is not automatically added to our servers for daily, weekly, and monthly history data...it must be requested. We did add it for you, and you should now be receiving data on it. So, the server saw it as an invalid symbol, which is why you received the error that you did. Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

I wanted to let everyone know that I have been working with leo, and have fixed the history issues he was having. I need to do some more work and testing, but we should have a new build soon. Thanks for your patience!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

No history error returned - timed out - v4.0.0.0

Today (10 Sep 05, at 16:00 or so CST) after getting some history using the new client all appeared well (albeit a little slow on the retrieval - 2 to 4 seconds) I did an update on symbol ACO (nothing special abut it) and nothing came back - just my own time out message, no report of an error from the client. I tried other symbols and same result, no error reports, no data, just a time-out raised by my application. (20 second time-out). I exited and rebooted, and could not get history on anything. Exited again, restart my program and now I am getting history. Seems faster too (about 1.5 sec).

I suspect that there must be some server maintenance going on or an enormous load that is causing delays or no response.

If so, there should be some mechanism between the client and server (communication) of status so that appropriate error/response messages are issued from the client to the 3rd party software clients so they do not sit in limbo and have to evoke their own time-out messages.

There should always be a response to request if that request is supposed to return data or a message.

In the past when ever I got a time out evoked by lack of response in a history request it never got cleared up until I excited and restarted. I hope this has been cleared up but I am a little worried as there is still not (as far as I have tested it today) a closed loop in the history retrieval process for all cases. (I use tcp/ip).

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/
Edited by David on Sep 10, 2005 at 04:35 PM