Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"You are much better than lawyers or the phone company because you answer the phone when I call! I just love your customer service." - Comment from Isreal
"I'm very glad I switched to IQFeed. It's working perfectly with no lag, even during fast market conditions." - Comment from Andy via Email
"I am enjoying the feed very much - so superior to the broker provided feed I was previously using." - Comment from George
"You are either overstaffed or people just don't have problems with your feed because customer support always answers the phone quickly." - Comment from Jay via Email
"You have an excellent feed. Very few spikes for Spot Forex." - Comment from Public Forum Post
"I "bracket trade" all major news releases and I have not found one lag or glitch with DTN.IQ feed. I am very comfortable with their feed under all typical news conditions (Fed releases, employment numbers, etc)." - Comment from Public Forum
"Everything is working great with the API. I love it." - Comment from Calvin
"Just a quick one to say I'm very impressed so far :) The documentation for developers is excellent and I've quickly managed to get an app written to do historical downloads. The system is very robust and pretty quick considering the extent of data that's available. The support guys have been very helpful too, in combination with the forums it's been plain sailing so far!" - Comment from Adam
"The service is great, I see a noticeable improvement in my volume profiles over [broker]'s data feed" - Comment from Larry
"IQ feed is brilliant. The support is mind-bending. What service!" - Comment from Public Forum Post
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Invalid Update Messages using IQFeed socket API
Author Topic: Invalid Update Messages using IQFeed socket API (4 messages, Page 1 of 1)

lvarga
-Interested User-
Posts: 12
Joined: Oct 7, 2004


Posted: Oct 7, 2004 10:40 AM          Msg. 1 of 4
I've been receiving invalid update messages through the socket API;
first I thought it was a bug in my update-parsing process so I modified the console_stream example to duplicate the problem -- sent the program

- it subscribes to the Nasdaq100 stocks,
- reads 1000 updates,
- unsubscribes for the symbols,
- after suspending for a minute starts the iteration again.
/This modified console_stream uses CString so use MFC static/shared library + replace #include <windows.h> to #include <afs.h> in stdafx.h/

After some iterations I receive updates like:

Q,FLEX,F,13.6701,-0.1299,-0.00941304,1766768,181,13.8800,.PSFT,13.6700,13.6700,1100,800,,173,13.88,11:51b,,13.8100,13.8000,0.,,,,c,N,,,,10/06/2004,,13.6701,,,,-0.1299,0,1380.,-0.741625037,0.,0.,-0.1399,1,1.015354679,7602092.9811,14,4,,6610306,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,4607,,,,
Note the 9th column: ".PSFT". Similar happens in other price fields as well (bid, ask, last, etc).

Could somebody also compile/run the program and see what happens (pm me , I can send the code)? Ususally the longer it runs, the more invalid updates it receives.

I appreciate any ideas, comments!!!

Lajos
Edited by DTN_Steve_S on Sep 19, 2011 at 09:22 AM

DTN_Jay_Froscheiser
-VP, Product Operations-
Posts: 1746
Joined: May 3, 2004

DTN IQFeed/DTN.IQ/DTN NxCore


Posted: Oct 7, 2004 10:20 PM          Msg. 2 of 4
I think you received a response via email, but for the benefit of others (since we have seen this before), this problem of corrupt data is nearly always a result of having to small a TCP buffer. Our feed is one of the fastest and most comprehensive, so while you may have been able to get away with a small TCP buffer on feeds that consolidate or filter their feed, with ours you need to open it up. I think we recommend a 64k buffer. If I am wrong, I am sure someone will correct me with the proper size they use to keep from corrupting messages.

Jay Froscheiser
DTN Market Access, LLC.

lvarga
-Interested User-
Posts: 12
Joined: Oct 7, 2004


Posted: Oct 8, 2004 10:30 AM          Msg. 3 of 4
Thank you for the fast reply, I really appreciate it!

I've tried the 64k TCP buffer modification and double-checked the partial
message handling -- I could not find any bugs in it, so I would assume that
the invalid updates were not due to partial messages as they are handled by
the test application. But just in case I've started the 64k TCP buffered
application today shortly after market open and got invalid updates again
(lines from the test application's output):

...
Starting iteration #3
Parse error in column 3 in line:
P,SPLS,F,.SNPS,,,3185,100,0.0000,0.0000,16.7100,16.8000,200,300,,173,0.,09:29t,,0.0000,29.8700,0.09,,,,t,N,,,,10/07/2004,,.SNPS,,,,,0,,,,,,1,,,14,4,,1513610,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,23,,,,
Parse error in column 3 in line:
P,WFMI,F,.ADBE,,,1500,100,0.0000,0.0000,50.8600,50.9500,100,100,,175,0.,09:29t,,0.0000,84.8400,0.09,,,,t,N,,,,10/07/2004,,.ADBE,,,,,0,,,,,,1,,,14,4,,591682,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,12,,,,
Parse error in column 3 in line:
Q,SPLS,F,.SNPS,,,3185,100,0.0000,0.0000,29.8000,29.8000,500,300,,175,0.,09:29b,,0.0000,29.8700,0.,,,,t,N,,,,10/07/2004,,.SNPS,,,,,0,,,13.09,13.,,1,,,14,4,,1513610,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,23,,,,

...

Parse error in column 9 in line:
Q,AMGN,F,56.6700,0.11,0.001944837,126434,100,56.6700,.PAYX,56.6500,56.6800,100,200,173,173,56.67,09:31t,,56.2900,56.5600,0.03,,,,p,N,,,,10/08/2004,,56.6700,,,,0.11,0,28.9,-0.981450411,0.,0.,0.38,1,1.,71745183.390000001,14,4,,6750460,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,453,,,,
Parse error in column 9 in line:
Q,AMGN,F,56.6700,0.11,0.001944837,126434,100,56.6700,.PAYX,56.6500,56.6800,300,200,,173,56.67,09:31b,,56.2900,56.5600,0.03,,,,c,N,,,,10/08/2004,,56.6700,,,,0.11,0,28.9,-0.981450411,0.,0.,0.38,1,1.,71745183.390000001,14,4,,6750460,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,453,,,,

...

I would be very happy if you could compile and run this test application (sent in email) on
some of your development machines, to see whether this error can be
duplicated in your environment.

Best Regards,
Lajos

sasha
-Interested User-
Posts: 54
Joined: Jul 21, 2004


Posted: Oct 8, 2004 02:15 PM          Msg. 4 of 4
I too have just noticed this problem and can replicate it using small buffers and large buffers.

I don't think it has anything to do with the buffer size as long as there are no bugs in your buffer handling and you are processing new data quickly enough.

It seems to be a problem of sending too many Watch request messages too quickly. If I send max 50 Watch symbol messages at a time then it doesn't happen. I haven't bothered to fine tune it and find the magic number because it will vary on different machines. I know 100 doesn't work.

Like Lajos I have received similar message corruption but with all message types - P, F and Q (Summary, Fundamental and Update). Most of the fields are returned but a few of the fields are rubbish, e.g. Last price = ".AMAT", etc...

I have used between 256 byte and 4096 byte buffers for each read and then multiple re-reads if more data is present. Whether you handle it quickly, very quickly or extremely quickly it doesn't matter. I have only been able to test reading up to 300KB per second from the socket in one hit if data is available, and don't seem to have any problems with my buffer handling.

Corruption only happens when:
1. Sending multiple Watch requests of 100 symbols, and only
2. If IQ Connection Manager is NOT already subscribed to them

I am dispatching a total of 95 Watch request messages one at a time, consecutively and asynchronously.

I don't think this is symbol related but I'm sure if you pick the busiest Nasdaq 100 symbols you have a greater chance of seeing corruption. I had 90 Nasdaq 100 and 5 Dow symbols.

I am using DTN.IQ and when it starts I can open up the IQ Connection Manager and check that there are no symbols being watched.

So the process is:
1. Close IQConnect process/es
2. Connect to TCPIP Level 1 stream
3. Send 100 Watch request messages
4. One of the first response messages (P, F or Q) of the 80-90th symbol will have bad fields
5. Maybe a few more corrupt messages come back
6. The rest of the messages from then on are OK
7. Un-Watch all these symbols and repeat from Step 2 and you will have the same problem

However, if you just kill the app without unwatching, IQConnection Manager shows there are still 100 symbols being watched. If you repeat from Step 2 you won't have a problem - at least I haven't seen it.

Alternatively you can Send only 50 Watch request messages at a time and not have this problem.

I'm using my own C# code.

A sample bad message I detected from error logs converting string fields to numeric types - I didn't check if the fields make sense at this point - just an attempt to convert to types.

=> Disney has a Last price of ".2485", High price of "U", Low price of "F|" and Bid Size of "55.95", Market Open of "0.001"
P,DIS,D,.2485,0.0010,0.004040404,,,U,F|,,55.9100,55.9500,100,700,,,,,,.2475,.2475,0.04,,,,n,,,,,,,.2485,,,,0.0010,0,,,,,0.001,1,,,,4,,,CSE-NYSE-NMS,,,,,1,,,,

The next message for Disney is OK.

Repeat and I get the problem with the first message for the same symbol Disney, similar and/or different fields. I could also get one more Update message for Disney that is also corrupt.

Regards,
Sasha
Edited by sasha on Oct 8, 2004 at 02:17 PM
 

 

Time: Tue November 29, 2022 5:59 AM CFBB v1.2.0 15 ms.
© AderSoftware 2002-2003