Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I'm satisfied with IQFeed. It's the most reliable and fastest quote feed I have ever used. Although I'm a resident in China, it's still very fast!" - Comment from Xiaofei
"I noticed that ******* quotes locked up shortly after the interest rate announcement yesterday while yours stayed stable." - Comment from Ron in Utah
"Boy, probably spent a thousand hours trying to get ******* API to work right. And now two hours to have something running with IQFeed. Hmmm, guess I was pretty stupid to fight rather than switch all this time. And have gotten more customer service from you guys already than total from them… in five years." - Comment from Jim
"Version 4.0.0.2 has been working well for me and I appreciate that it is now a much tighter client to work with. I feel I can go to press with my own application and rely on a stable platform" - Comment from David in IA.
"After all the anxiety I had with my previous data provider it is a relief not to have to worry about data speed and integrity." - Comment from Eamonn
"I've been using IQFeed 4 in a multi-threaded situation for the last week or two on 2600 symbols or so with 100 simultaneous daily charts, and I have had 100% responsiveness." - Comment from Scott
"IQFeed version 4 is a real screamer compared to anything else I have seen." - Comment from Tom
"DTN feed was the only feed that consistently matched Bloomberg feed for BID/ASK data verification work these past years......DTN feed is a must for my supply & demand based trading using Cumulative Delta" - Comment from Public Forum Post
"Everything is working great ! Very impressive client. The news refreshes better and is more pertinent than the ******* feed I paid $ 100/month for. I Also like the charts a lot." - Comment from Leon
"Just a quick one to say I'm very impressed so far :) The documentation for developers is excellent and I've quickly managed to get an app written to do historical downloads. The system is very robust and pretty quick considering the extent of data that's available. The support guys have been very helpful too, in combination with the forums it's been plain sailing so far!" - Comment from Adam
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »out of memory in sockets in v 5.0.0.11
Author Topic: out of memory in sockets in v 5.0.0.11 (21 messages, Page 1 of 1)

otabek
-Interested User-
Posts: 1
Joined: May 11, 2013


Posted: May 11, 2013 06:28 PM          Msg. 1 of 21
Hello!We have application, which create multiple asyncronius sockets to level 1 port , one socket for one ticker, about 200.. In iqfeed version 4.9.0.3 we had no problem...Everything worked very well... After updating to last version 5.0.0.11, when application trys to connect to level 1 port via async sockets, iqconnect send such error:

Error: Out of memory.
Box: small information box.
Title in window of box: iqconnect.
Icon: exclamation mark in a yellow triangle (means warning).

After deleting newest version and install older version 4.9.0.3 - everything work without any problems...

After some observe, we found, in older version one socket occupies about 2MB of memory, in the newest version occupies more 8Mb for one sockets....

Please, explain, what can help us?

DTN_Steve_S
-DTN Guru-
Posts: 2084
Joined: Nov 21, 2005


Posted: May 14, 2013 08:29 AM          Msg. 2 of 21
This was handled via chat but I wanted to post here in case anyone else has noticed the same thing.

We increased the size of the local socket buffers in IQFeed 5.0 accounting for this difference. Most users will not notice a huge difference but since the above poster is using ~200 socket connections to IQFeed it became noticeable very quickly in his case.

In this case the solution is to reduce the number of socket connections to IQFeed.

programmer
-Interested User-
Posts: 35
Joined: Feb 3, 2010


Posted: May 25, 2013 01:33 PM          Msg. 3 of 21
I'm getting this error too and I'm not making lots of simultaneous socket connections. It seems that over time iqconnect.exe uses more and more memory until it consumes all memory on the machine. If you make a lot of connections for example to pull historical data (NOT simultaneous connections; just sequential) it will use up memory faster.

I believe there is some sort of memory leak in the 5.0.0.11 version of iqconnect.exe.

Craig
-DTN Guru-
Posts: 258
Joined: Apr 16, 2010


Posted: May 28, 2013 12:16 AM          Msg. 4 of 21
To add to these observations, I have noticed that if I leave version 5 IQFeed running for longer than a session, it starts to become very slow for history requests. If I start it again, everything returns to normal.

DTN_Steve_S
-DTN Guru-
Posts: 2084
Joined: Nov 21, 2005


Posted: May 28, 2013 09:28 AM          Msg. 5 of 21
Thanks for the feedback everyone. I'll look into this. The one theme I've seen between all of your scenarios is that you are making multiple connections to the feed (not simply maintaining a single connection and making multiple requests). Is this accurate?

Craig, can you give me a bit more detail about the "very slow" history requests? Do you have any numbers that you can give us to define how much slower the feed becomes?

programmer
-Interested User-
Posts: 35
Joined: Feb 3, 2010


Posted: May 28, 2013 01:51 PM          Msg. 6 of 21
Quote: The one theme I've seen between all of your scenarios is that you are making multiple connections to the feed (not simply maintaining a single connection and making multiple requests). Is this accurate?


That is true for me. It seems to leak around 200 bytes per connection. Not so much by itself but over a long enough time the iqconnect.exe process causes the machine to run out of RAM and I get the out of memory error.
Edited by programmer on May 28, 2013 at 01:59 PM

Craig
-DTN Guru-
Posts: 258
Joined: Apr 16, 2010


Posted: May 28, 2013 03:05 PM          Msg. 7 of 21
I don't have any hard numbers, to give more detail, I usually download the last days daily bar for ~450 contracts at about 5 est. Usually each request is sub-second. For a couple of days at the end of last week I was waiting another maybe 6 hours before I did the history requests, leaving iqfeed running. When I did so each request was taking in the order of maybe 1-1.5 seconds (remember, this is for one bar). At first I thought that I had a database problem, but no, restarting iqfeed restored the speed. I can take exact numbers if that helps.

programmer
-Interested User-
Posts: 35
Joined: Feb 3, 2010


Posted: May 29, 2013 08:06 AM          Msg. 8 of 21
Just thought I would note that it's easy to recreate the memory leak simply by connecting to any iqfeed socket then disconnect (no command need to be executed). Do it several thousand times and watch the memory usage grow.
Edited by programmer on May 29, 2013 at 08:07 AM

DTN_Nathan_Bartsch
-IQ Developer-
Posts: 33
Joined: May 3, 2004

Telvent DTN


Posted: May 29, 2013 09:14 AM          Msg. 9 of 21
We have already replicated the problem and have a fix ready for the next version of IQ Feed.

Nathan W. Bartsch - Senior Software Engineer - Telvent DTN

DTN_Steve_S
-DTN Guru-
Posts: 2084
Joined: Nov 21, 2005


Posted: Jun 4, 2013 09:51 AM          Msg. 10 of 21
Expanding on Nathan's earlier response.

The fix we are releasing in the next build of IQFeed 5.0 will minimize this behavior but we are unable to resolve it completely at this time.

There are 2 sources of the increasing memory footprint that we have been able to identify. The first one (and the most significant) we have direct control over and have limited it to a maximum size in the next build (it was previously unlimited). We will modify this behavior further in a future release of IQFeed. The second source is buried within the socket library we are using and consumes ~60bytes per connection. Unfortunately, due to the extensiveness of the development involved, we are not able to resolve this in a timely manner so it has been slated for being fixed in a future release of IQFeed.

At the current time, the workaround for this would be to either restart the feed periodically or to maintain a persistent connection to the feed for your requests instead of continually creating new connections.

davidt07
-Interested User-
Posts: 8
Joined: Jul 31, 2011


Posted: Jun 13, 2013 03:47 PM          Msg. 11 of 21
Hi,

I just upgraded to 5.0.0.14 after receiving this same error message from 5.0.0.11 and I am still suffering the same issue as described in this post.

I am only watching 4 symbols and I am not pulling any historical data.

I am watching, YMM13, YMU13, ESM13 and ESU13

I am running on Windows 2008 R2 64bit on a Quadcore XEON CPU with 12GB of RAM running MSSQL 2008 R2

It seems like there is still a memory leak with IQConnect?

Thanks

--
David Tran
MCP, MCSA Windows Server, MCTS Vista, MCITP MS SQL 2008 R2 Database Administrator Pro

DTN_Steve_S
-DTN Guru-
Posts: 2084
Joined: Nov 21, 2005


Posted: Jun 13, 2013 04:17 PM          Msg. 12 of 21
David, the fix mentioned in this post was specific to Lookup Port connections so your issue would be unrelated in terms of what is causing it and it wouldn't surprise me that upgrading from 5.0.0.11 to 5.0.0.14 wouldn't have an effect.

With that said, the way you are using the feed (from what you've described) is pretty common amongst thousands of our users who aren't experiencing the same problem (or at least not contacting support if they are) which makes me believe there is something in your setup somewhere that is exposing the problem.

Can you to provide us with a copy of your app that we can run locally to try and duplicate the problem? Alternatively, can you duplicate the issue you duplicate the problem with an apps we distribute with IQFeed? For example, you should be able to open up 4 copies of the IQWatchQuote app simultaneously and watch the 4 symbols or you can use the ConsoleBenchmarkTool example app that is included with the dev version of IQFeed and configure it to watch the exact same fieldset that you are using and watch the symbols using a single connection to IQFeed (if that's what your app is doing).

Being able to duplicate it will make it easiest for us to track down exactly what is going on. Or even knowing that you can't duplicate it with the end user apps we provide would isolate the issue to something with your machine/network setup or something within your app.

davidt07
-Interested User-
Posts: 8
Joined: Jul 31, 2011


Posted: Jun 13, 2013 04:21 PM          Msg. 13 of 21
Absolutely, please let me know how I should send you my app? It's basically Level1SocketsForm tied to a MSSQL DB using ADO.NET. I've used a try catch for error exception and my app is not throwing any exceptions, just that error popup from IQConnect.

I will wrap up my whole app for you guys so that you can give it a run including my db schema definition.

Kindly

David

DTN_Steve_S
-DTN Guru-
Posts: 2084
Joined: Nov 21, 2005


Posted: Jun 13, 2013 04:31 PM          Msg. 14 of 21
If it is less than about 15MB compressed, you can email your app in a zip file that is password protected (to bypass mail filters) to developer support. I will send you a PM with the dev support email address. If it is larger we will have to get another option for transferring it if you don't have a way to host it for us to download.

davidt07
-Interested User-
Posts: 8
Joined: Jul 31, 2011


Posted: Jun 13, 2013 04:50 PM          Msg. 15 of 21
Email will be fine, thanks

David

mdu
-Interested User-
Posts: 2
Joined: Jul 15, 2013

I see more than you see, I C++.


Posted: Jul 15, 2013 03:35 AM          Msg. 16 of 21
Hello,

I have a similar problem like that previously described above. I have an application which connects to IQConnect. I use 3 sockets, one for admin status connection, one for Level1 and one for Level2.
I connect all of them once. I made subscriptions for 500 symbols for Level1 and a single subscription for both levels. Since last week I started to receive a message box with "Out of memory" and I have looked in Task Manager and I noticed that virtual memory is counting up.
At first I thought that there is a bug in my application, slow processing of incoming data but is not true I erased that part of processing, I kept it simple just a counter with number of messages received and the problem persist.

I use 5.0.0.11. Can you provide me a solution for this problem, a workaround for this before a new update ?

Thank you..

DTN_Steve_S
-DTN Guru-
Posts: 2084
Joined: Nov 21, 2005


Posted: Jul 15, 2013 01:42 PM          Msg. 17 of 21
mdu, I would recommend updating to IQFeed 5.0.0.14 (the current release version). The 5.0.0.11 release had a bug that caused corrupted data to be received from the level 2 port of IQFeed on occasion (and you mentioned using level 2).

If after updating you still see the same issues, my guess would be that there is something unique in your setup that is exposing the issue in IQConnect and at that point I will need to be able to duplicate the issue locally in order to track this down. Is it possible for you to email developer support with a copy of your app for testing?

mdu
-Interested User-
Posts: 2
Joined: Jul 15, 2013

I see more than you see, I C++.


Posted: Jul 30, 2013 03:42 AM          Msg. 18 of 21
Hello Steve, I updated to 5.0.0.14 version and still I have the problem. Also I noticed that I've started to receive more data on Level1 Socket for the same symbol which I did subscription on Level2. Can you provide me an email address from developer support ? What do you mean by sending a copy of my app for testing ? You want the source code or only the application ?

DTN_Steve_S
-DTN Guru-
Posts: 2084
Joined: Nov 21, 2005


Posted: Jul 30, 2013 08:33 AM          Msg. 19 of 21
I'm not finding an account based on the information in your forum account. Please send me a PM with your loginID and/or the email address your account is registered under.

drc34
-Interested User-
Posts: 2
Joined: Mar 8, 2013


Posted: Aug 13, 2013 01:50 PM          Msg. 20 of 21
Hi,

I'm also experiencing the same "out of memory" error. I have the most updated release version of IQConnect 5.0.0.14 (downloaded it sunday and installed it on a completely fresh server instance).

After the IQConnect error pops up, I lose my data connection. Then when its restarted everything works as expected.

I'm listening to between 350-400 symbols.

Dylan

jonnyb
-DTN Evangelist-
Posts: 116
Joined: Aug 15, 2012


Posted: Aug 26, 2013 09:28 PM          Msg. 21 of 21
I have also received the out of memory error after upgrading to .14. However, thus far it has only happened once.
 

 

Time: Sat August 8, 2020 1:04 PM CFBB v1.2.0 31 ms.
© AderSoftware 2002-2003