While testing the Historical connection via TCP/IP, I ran into an interesting issue that I wanted to figure out if it's my fault, or a race condition in iqconnect.exe.

My initial test program would open a socket for each request to the historical data and then close the socket (dumb, I know, but it was proof-of-concept code). Over time, this would create a lot of extra threads (over 600 at the point I rewrote and restarted the program). The threads did not get leaked every time, rather they would occur after several connect/disconnect cycles (say 20 or so), so it looks like some sort of race condition on the shut-down of the sockets.

So, this brings up the question: Am I suppose to issue some sort of message to signal that I'm done with the port, or is closing the socket suppose to handle the disconnect?

If it is a bug, it's not really an important issue, since I've modified my program to not open and close the historical port all the time, but thought I should double check.

-Eric

There is not a command to signal disconnecting from the lookup port. Closing the socket is really all you need to do.

After some brief testing this morning I was unable to duplicate it.

In order to track this down I will need some more information.

When you say "over time", how much time are we talking about?
How many requests did you make to the server over that time?
Were they all the same type of request? (minute, tick, end of day, etc)
Were you making the requests simultaneously on the different sockets or did you wait for one to complete before opening a new socket?

Hi Steve,

Thanks for trying to reproduce the problem. I'll see if I can come up with a simple program that reproduces it for you, since it's still entirely possible that the problem was/is on my end.

To answer your questions:

• Over time is normally enough time for about 100 connections, which was normally about 5 minutes.


• All of the requests were for daily data (HD).


• All of the requests were in a serial fashion where I would open a socket, make a request, wait for !.*!, or timeout, and then close the socket.

After rewriting the program to keep a socket open across requests, I have been able to run the program for over 10 hours without a single glitch.

I'll post the results of my testing later and provide a snippet of code.

Thanks again.

Hi Steve,

I've figured out what is happening and why it seemed to be random on my end. It turns out that if I close a socket that has unread data (disorderly shutdown of socket), then iqconnect.exe seems to be keeping the thread that's servicing that connection running.

Normally I would make a request, read data until the !ENDMSG!, and then close the socket. This works fine except for the case where I make a request and there isn't any data for the symbol and I get the following response:

!ERROR! !NONE!

!ENDMSG!
!ERROR! Invalid symbol.

!ENDMSG!

My parser would pick up the first !ENDMSG! and quit reading socket data and then close the socket with the !ERROR! Invalid symbol.* data potentially still in transit. At this point, the iqconnect.exe application doesn't close the thread servicing that socket, even though I've closed the socket on my end.

Question:
I tried using shutdown(...), reading all data until there was none left, and then calling closesocket(...), but it didn't seem to help. Does iqconnect.exe properly handle aborted socket connections?

The multiple error message bug strikes again.

The multiple errors will no longer be an issue as of our next release which will bypass this issue in the form you were able to duplicate it.

I will look into how IQConnect is handling aborted sockets as well to check on a permanent fix for this issue as well.

Thanks for looking into further.

I encounter the same problem when attempting to read historical data for a symbol obtained using the SES command. Specifically:

SES,^A;

returns symbols including 'AAAN'. Now, 'AAAN' is an existing symbol, however there's no price data for it. Consequently when I do:

HD,AAAN,10;

I get:

!ERROR! !NONE!

!ENDMSG!
!ERROR! Invalid Symbol.

!ENDMSG!

You have to separate the option root from the expiration and price code.

Try HD,AA_AN,10;

-Eric

Hi Steve,

In looking over this old post, I was wondering if you were ever able to reproduce this potential bug. I looked at it this morning and am able to reproduce the problem very easily, now. Here's the process to reproduce it:

• Open a socket connection to 9100


• Send a request (I used 'HD,MSFT,10;' for this case)


• Read 0 to N-1 bytes where N is the number of bytes in the response message


• Close the socket (with data still available to be read)

I also tried doing an orderly shutdown using shutdown(sHistoricalSocket,SD_BOTH) before closing the socket without any luck.


Here's some quick-and-dirty code that will reproduce the potential bug:

// Execute this code after connecting to IQConnect.exe
for(int nIndex=0; nIndex < 10; ++nIndex)
{
    ThreadLeakTest();
}

/**
 * Connects to the historical quote feed at port 9100.
 *
 * @param s     Reference to a SOCKET instance used to
 *              maintain the socket connection.
 *
 * @returns True if socket connected, False otherwise
 */
bool ConnectToHistorical(SOCKET& s)
{
    sockaddr_in addr;
    bool        bConnected = false;

    // set the address to localhost:9100
    addr.sin_family = AF_INET;                                  
    addr.sin_port = htons(9100);
    addr.sin_addr.S_un.S_addr = inet_addr("127.0.0.1");
    memset(&addr.sin_zero, 0, sizeof(addr.sin_zero));

    // connect to the server
    if (connect(s,(struct sockaddr*)&addr,sizeof(addr))==0)
    {   
        bConnected = true;
    }

    return bConnected;
}



/**
 * Opens a new socket for each history request and doesn't read all of the data
 * before closing the socket.  This tends to leave sockets open in the
 * iqconnect.exe application which shows up as an excessive number of threads
 * in the Windows task manager.
 */
void ThreadLeakTest()
{
    const size_t uTEMP_SIZE         = 1024*16;
    SOCKET       sHistoricalSocket;
    char         achTemp[uTEMP_SIZE];

    // Create a socket and attach to the historical data feed
    sHistoricalSocket = socket(AF_INET,SOCK_STREAM,0);

    if (sHistoricalSocket != INVALID_SOCKET) 
    {
        if ( !ConnectToHistorical(sHistoricalSocket) )
        {
            cerr << "Unable to connect to historical socket" << endl;
            return;
        }


        // Request data over the historical port
        strncpy(achTemp,"HD,MSFT,10;",uTEMP_SIZE);
        send(sHistoricalSocket, achTemp, strlen(achTemp), 0);

        // As of this morning at 2006-04-10 08:28, there will be 652 bytes 
        // in the response message.

        // Wait 1 second for data to come in
        Sleep(1000);

        // Retrieve up to 100 bytes of the 652-byte response
        (void)recv(sHistoricalSocket,achTemp,100,0);


        // Close the history socket
        closesocket(sHistoricalSocket);
    }
}

Edited by EspressoE on Apr 10, 2006 at 08:13 AM
Edited by EspressoE on May 3, 2006 at 10:00 AM

How do you know it's an option and not a stock symbol?
Thanks

Since symbol 'AAAN' exists, but there's no historical data for it, IQFEED should return simply a blank line followed by !ENDMSG!, rather than some type of error message. Generallly, IQFEED should not return an error message for any symbols retrieved using its symbol lookup protocol, rather an empty set if applicable. Could DTN comment on this issue? I think this issue has similar results to the broken socket issue but is triggered in a different manner.

Sorry, since I'm working with options, I automatically assumed that AAAN was an option symbol

Anyway, as Steve pointed out previously in this post and in several other threads that may be more appropriate for the issue that you are talking about, the double-error message that you are currently seeing will be resolved in the next release. So, when you do a query and there isn't any historical data available, you will currently get:

ERROR! !NONE!

!ENDMSG!
!ERROR! Invalid symbol.

!ENDMSG!

I believe in the next version, this will become just:

ERROR! !NONE!

!ENDMSG!

This is nearly the same response that you would like, so it looks like DTN may have already resolved this -- you will just have to wait for the next release of the software.

Eric,

I have found the cause of this bug (the one from your original post). It will be fixed in the next release.


andrzejs,

Eric was correct that the duplicate error messages will be fixed in the next release. He is also correct that for most cases, You will recieve an

!ERROR! !NONE!
!ENDMSG!

when an empty set is returned.



However, AAAN has never traded. As such, there is no history and our History servers will return

!ERROR! Invalid Symbol
!ENDMSG!
Edited by DTN_Steve_S on Apr 11, 2006 at 01:16 PM

Steve S:

I have been dealing with the problem of multiple !ERROR! / !ENDMSG! for a while. The first mention of this goes back to June 2005. When can we expect that this will be resolved? It's been a long time.

This is particularly important for AmiBroker users that use EOD to do market explorations. If their symbol lists include invalid symbols, the extraneous error messages cause subsequent EOD data to get shifted around between symbols. This is due to AmiBroker expecting a certain behavior from IQFeed.

I have had extensive discussions with Thomasz Janeczko on this subject matter. His plugin is written according the the IQ Feed API; however, IQ Feed is not following it.

Obviously, a simple solution is to not have invalid symbols. Then again, one would also expect that a service with a published API would adhere to it. ;) :)

For all of our sanities' sakes, when can we finally expect this to be resolved.

Best regards,

Michael Wil. Swiercz
MWerk Inc.

Michael, I am a bit confused as to what you are asking here.

are you asking when the release is going to be?

The release with these fixes is currently in the testing (QA) and review process.

Steve:

My apologies for the TMI and confusion. But yes, I was asking whether this double !ERROR! issues will be fixed in the next release and when we can expect it. Any idea when the next release will be made public?

Best,

Mike :)