Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I've never had DTN go out on me since switching. ******* would go down a couple times every month when I was using them." - Comment from Bryce in AL.
"I am enjoying the feed very much - so superior to the broker provided feed I was previously using." - Comment from George
"If someone needs the best quality data and backfill beyond what their broker provides at a rate that is the best in the industry, I highly recommend IQFeed." - Comment from Josh via Public Forum
"Very impressed with the quality of your feed - ******* is a real donkey in comparison." - Comment from A.C. via Email
"I just wanted to tell you what a fine job you have been doing. While *******, from what I hear, has been down and out, off and on, IQ feed has held like a champ this week." - Comment from Shirin
"Its working FABULOUSLY for me!! Holy cow...there has been so much I've been missing lately, and with this feed and Linnsoft software...I'm in the game now." - Comment from Chris R.
"I use IQ Feed, Great stuff as far as data analysis information, storage and retrieval is concerned." - Comment from Public Forum
"Just a quick one to say I'm very impressed so far :) The documentation for developers is excellent and I've quickly managed to get an app written to do historical downloads. The system is very robust and pretty quick considering the extent of data that's available. The support guys have been very helpful too, in combination with the forums it's been plain sailing so far!" - Comment from Adam
"Awesome response, as usual. It is a sincere and refreshing pleasure to do business with DTN, compared to your competition." - Comment from Ryan
"I have been using IQFeed now for a few years in MultiCharts and I have zero complaints. Very, very rare to have any data hiccups or anything at all go wrong." - Comment from Public Forum
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Closing Historical Port (9100) using TCP/IP
Author Topic: Closing Historical Port (9100) using TCP/IP (15 messages, Page 1 of 1)

EspressoE
-Interested User-
Posts: 30
Joined: Mar 8, 2006


Posted: Mar 14, 2006 10:39 PM          Msg. 1 of 15
While testing the Historical connection via TCP/IP, I ran into an interesting issue that I wanted to figure out if it's my fault, or a race condition in iqconnect.exe.

My initial test program would open a socket for each request to the historical data and then close the socket (dumb, I know, but it was proof-of-concept code). Over time, this would create a lot of extra threads (over 600 at the point I rewrote and restarted the program). The threads did not get leaked every time, rather they would occur after several connect/disconnect cycles (say 20 or so), so it looks like some sort of race condition on the shut-down of the sockets.

So, this brings up the question: Am I suppose to issue some sort of message to signal that I'm done with the port, or is closing the socket suppose to handle the disconnect?

If it is a bug, it's not really an important issue, since I've modified my program to not open and close the historical port all the time, but thought I should double check.

-Eric

DTN_Steve_S
-DTN Guru-
Posts: 2086
Joined: Nov 21, 2005


Posted: Mar 15, 2006 08:24 AM          Msg. 2 of 15
There is not a command to signal disconnecting from the lookup port. Closing the socket is really all you need to do.

After some brief testing this morning I was unable to duplicate it.

In order to track this down I will need some more information.

When you say "over time", how much time are we talking about?
How many requests did you make to the server over that time?
Were they all the same type of request? (minute, tick, end of day, etc)
Were you making the requests simultaneously on the different sockets or did you wait for one to complete before opening a new socket?

EspressoE
-Interested User-
Posts: 30
Joined: Mar 8, 2006


Posted: Mar 15, 2006 11:25 AM          Msg. 3 of 15
Hi Steve,

Thanks for trying to reproduce the problem. I'll see if I can come up with a simple program that reproduces it for you, since it's still entirely possible that the problem was/is on my end.

To answer your questions:

  • Over time is normally enough time for about 100 connections, which was normally about 5 minutes.

  • All of the requests were for daily data (HD).

  • All of the requests were in a serial fashion where I would open a socket, make a request, wait for !.*!, or timeout, and then close the socket.



After rewriting the program to keep a socket open across requests, I have been able to run the program for over 10 hours without a single glitch.

I'll post the results of my testing later and provide a snippet of code.

Thanks again.

EspressoE
-Interested User-
Posts: 30
Joined: Mar 8, 2006


Posted: Mar 16, 2006 09:23 AM          Msg. 4 of 15
Hi Steve,

I've figured out what is happening and why it seemed to be random on my end. It turns out that if I close a socket that has unread data (disorderly shutdown of socket), then iqconnect.exe seems to be keeping the thread that's servicing that connection running.

Normally I would make a request, read data until the !ENDMSG!, and then close the socket. This works fine except for the case where I make a request and there isn't any data for the symbol and I get the following response:


!ERROR! !NONE!

!ENDMSG!
!ERROR! Invalid symbol.

!ENDMSG!


My parser would pick up the first !ENDMSG! and quit reading socket data and then close the socket with the !ERROR! Invalid symbol.* data potentially still in transit. At this point, the iqconnect.exe application doesn't close the thread servicing that socket, even though I've closed the socket on my end.

Question:
I tried using shutdown(...), reading all data until there was none left, and then calling closesocket(...), but it didn't seem to help. Does iqconnect.exe properly handle aborted socket connections?

DTN_Steve_S
-DTN Guru-
Posts: 2086
Joined: Nov 21, 2005


Posted: Mar 16, 2006 10:35 AM          Msg. 5 of 15
The multiple error message bug strikes again.

The multiple errors will no longer be an issue as of our next release which will bypass this issue in the form you were able to duplicate it.

I will look into how IQConnect is handling aborted sockets as well to check on a permanent fix for this issue as well.

Thanks for looking into further.

andrzejs
-Interested User-
Posts: 23
Joined: May 27, 2004


Posted: Apr 10, 2006 05:55 AM          Msg. 6 of 15
I encounter the same problem when attempting to read historical data for a symbol obtained using the SES command. Specifically:

SES,^A;

returns symbols including 'AAAN'. Now, 'AAAN' is an existing symbol, however there's no price data for it. Consequently when I do:

HD,AAAN,10;

I get:

!ERROR! !NONE!

!ENDMSG!
!ERROR! Invalid Symbol.

!ENDMSG!

EspressoE
-Interested User-
Posts: 30
Joined: Mar 8, 2006


Posted: Apr 10, 2006 07:09 AM          Msg. 7 of 15
You have to separate the option root from the expiration and price code.

Try HD,AA_AN,10;

-Eric

EspressoE
-Interested User-
Posts: 30
Joined: Mar 8, 2006


Posted: Apr 10, 2006 08:12 AM          Msg. 8 of 15
Hi Steve,

In looking over this old post, I was wondering if you were ever able to reproduce this potential bug. I looked at it this morning and am able to reproduce the problem very easily, now. Here's the process to reproduce it:


  • Open a socket connection to 9100

  • Send a request (I used 'HD,MSFT,10;' for this case)

  • Read 0 to N-1 bytes where N is the number of bytes in the response message

  • Close the socket (with data still available to be read)



I also tried doing an orderly shutdown using shutdown(sHistoricalSocket,SD_BOTH) before closing the socket without any luck.


Here's some quick-and-dirty code that will reproduce the potential bug:


// Execute this code after connecting to IQConnect.exe
for(int nIndex=0; nIndex < 10; ++nIndex)
{
ThreadLeakTest();
}



/**
* Connects to the historical quote feed at port 9100.
*
* @param s Reference to a SOCKET instance used to
* maintain the socket connection.
*
* @returns True if socket connected, False otherwise
*/
bool ConnectToHistorical(SOCKET& s)
{
sockaddr_in addr;
bool bConnected = false;

// set the address to localhost:9100
addr.sin_family = AF_INET;
addr.sin_port = htons(9100);
addr.sin_addr.S_un.S_addr = inet_addr("127.0.0.1");
memset(&addr.sin_zero, 0, sizeof(addr.sin_zero));

// connect to the server
if (connect(s,(struct sockaddr*)&addr,sizeof(addr))==0)
{
bConnected = true;
}

return bConnected;
}



/**
* Opens a new socket for each history request and doesn't read all of the data
* before closing the socket. This tends to leave sockets open in the
* iqconnect.exe application which shows up as an excessive number of threads
* in the Windows task manager.
*/
void ThreadLeakTest()
{
const size_t uTEMP_SIZE = 1024*16;
SOCKET sHistoricalSocket;
char achTemp[uTEMP_SIZE];

// Create a socket and attach to the historical data feed
sHistoricalSocket = socket(AF_INET,SOCK_STREAM,0);

if (sHistoricalSocket != INVALID_SOCKET)
{
if ( !ConnectToHistorical(sHistoricalSocket) )
{
cerr << "Unable to connect to historical socket" << endl;
return;
}


// Request data over the historical port
strncpy(achTemp,"HD,MSFT,10;",uTEMP_SIZE);
send(sHistoricalSocket, achTemp, strlen(achTemp), 0);

// As of this morning at 2006-04-10 08:28, there will be 652 bytes
// in the response message.

// Wait 1 second for data to come in
Sleep(1000);

// Retrieve up to 100 bytes of the 652-byte response
(void)recv(sHistoricalSocket,achTemp,100,0);


// Close the history socket
closesocket(sHistoricalSocket);
}
}

Edited by EspressoE on Apr 10, 2006 at 08:13 AM
Edited by EspressoE on May 3, 2006 at 10:00 AM

andrzejs
-Interested User-
Posts: 23
Joined: May 27, 2004


Posted: Apr 10, 2006 02:09 PM          Msg. 9 of 15
How do you know it's an option and not a stock symbol?
Thanks

andrzejs
-Interested User-
Posts: 23
Joined: May 27, 2004


Posted: Apr 11, 2006 01:41 AM          Msg. 10 of 15
Since symbol 'AAAN' exists, but there's no historical data for it, IQFEED should return simply a blank line followed by !ENDMSG!, rather than some type of error message. Generallly, IQFEED should not return an error message for any symbols retrieved using its symbol lookup protocol, rather an empty set if applicable. Could DTN comment on this issue? I think this issue has similar results to the broken socket issue but is triggered in a different manner.

EspressoE
-Interested User-
Posts: 30
Joined: Mar 8, 2006


Posted: Apr 11, 2006 08:53 AM          Msg. 11 of 15
Sorry, since I'm working with options, I automatically assumed that AAAN was an option symbol

Anyway, as Steve pointed out previously in this post and in several other threads that may be more appropriate for the issue that you are talking about, the double-error message that you are currently seeing will be resolved in the next release. So, when you do a query and there isn't any historical data available, you will currently get:

ERROR! !NONE!

!ENDMSG!
!ERROR! Invalid symbol.

!ENDMSG!


I believe in the next version, this will become just:

ERROR! !NONE!

!ENDMSG!


This is nearly the same response that you would like, so it looks like DTN may have already resolved this -- you will just have to wait for the next release of the software.

DTN_Steve_S
-DTN Guru-
Posts: 2086
Joined: Nov 21, 2005


Posted: Apr 11, 2006 10:19 AM          Msg. 12 of 15
Eric,

I have found the cause of this bug (the one from your original post). It will be fixed in the next release.


andrzejs,

Eric was correct that the duplicate error messages will be fixed in the next release. He is also correct that for most cases, You will recieve an

!ERROR! !NONE!
!ENDMSG!

when an empty set is returned.



However, AAAN has never traded. As such, there is no history and our History servers will return

!ERROR! Invalid Symbol
!ENDMSG!
Edited by DTN_Steve_S on Apr 11, 2006 at 01:16 PM

mwerk
-Interested User-
Posts: 2
Joined: May 3, 2006


Posted: May 3, 2006 09:55 AM          Msg. 13 of 15
Steve S:

I have been dealing with the problem of multiple !ERROR! / !ENDMSG! for a while. The first mention of this goes back to June 2005. When can we expect that this will be resolved? It's been a long time.

This is particularly important for AmiBroker users that use EOD to do market explorations. If their symbol lists include invalid symbols, the extraneous error messages cause subsequent EOD data to get shifted around between symbols. This is due to AmiBroker expecting a certain behavior from IQFeed.

I have had extensive discussions with Thomasz Janeczko on this subject matter. His plugin is written according the the IQ Feed API; however, IQ Feed is not following it.

Obviously, a simple solution is to not have invalid symbols. Then again, one would also expect that a service with a published API would adhere to it. ;) :)

For all of our sanities' sakes, when can we finally expect this to be resolved.

Best regards,

Michael Wil. Swiercz
MWerk Inc.

DTN_Steve_S
-DTN Guru-
Posts: 2086
Joined: Nov 21, 2005


Posted: May 3, 2006 10:20 AM          Msg. 14 of 15
Michael, I am a bit confused as to what you are asking here.

are you asking when the release is going to be?

The release with these fixes is currently in the testing (QA) and review process.

mwerk
-Interested User-
Posts: 2
Joined: May 3, 2006


Posted: May 5, 2006 01:42 PM          Msg. 15 of 15
Steve:

My apologies for the TMI and confusion. But yes, I was asking whether this double !ERROR! issues will be fixed in the next release and when we can expect it. Any idea when the next release will be made public?

Best,

Mike :)
 

 

Time: Mon October 26, 2020 8:27 AM CFBB v1.2.0 32 ms.
© AderSoftware 2002-2003