SimpleDB connections (keepalive / pooling [sort of])

Dec 9, 2011 at 9:13 PM

Hi,

First, I love Simol!  Great work on a refreshingly intuitive library.

One of the very few issues I have ran into with SimpleDB / Simol in general is that the HTTP / HTTPs sessions are kind of expensive to create and I've ran into trying to balance out having to wait and deal with the timeouts that might occur if a new HTTP / HTTPs session happens to be needed.  So, my thoughts here immediately go to keeping a SimpleDB connections open via HTTP keepalives.  This seems like sort of a natural function that a library like Simol might provide; however, I don't see indication in the code or documentation that points towards Simol providing this sorts of connection specific profiling.  I also unfortunately don't see anything in the SimpleDB apis that allows managing connections at this level.

So, my reaction is that I probably am missing something.  Does anyone know if I am perhaps looking in the wrong place.  Is this possibly just a setting in the .Net ServicePointManager?  I'm now investigating stuff like this.  If it is most easily managed at the runtime level then I probably will stumble on a quick fix and it's just a matter of time before I find the magic global .Net setting.  But, just wanted to put this out there in case anyone else has any insight on this and can tell me if I'm looking in the wrong places or the right places or whatever.  Also, looking for advice if it is not manageable at the runtime level then any suggestions on how to manage this?

Thanks!

Mat

Dec 10, 2011 at 8:24 PM

This may be related: I run into the error "System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server" on occasion when I've left the application idle for a while.  I wonder if a keepalive might apply to my situation as well.

Coordinator
Dec 10, 2011 at 8:38 PM

Hi Matt,

Thank you very much for the kind feedback!

I haven't run into any issues with that myself, but I can tell you that Simol is coded to minimize the chattiness with SimpleDB (caching and batch operations whenever possible) and that all communication with SimpleDB goes through the AWS SDK for SimpleDB. You might want to spend some time looking at the configuration options in the AWS SDK and pass a custom configured AmazonSimpleDBClient instance to Simol. I don't think any http connection configuration options are exposed through the SDK so you are probably on the right track looking for .NET framework global configuration hooks to improve performance.

Also, are you creating a new Simol client instance for each call to SimpleDB? You will see much better performance in a number of areas if you keep and reuse the same client instance (it's thread-safe) and that might help you http session issues as well.

Regards,

Ashley

Dec 11, 2011 at 11:14 PM
Edited Dec 11, 2011 at 11:15 PM

Thanks for the quick response, @Ashley!  I really appreciate your help.

I think you've made me aware of our main issue... At some point, we did indeed start creating new Simol client instances for each call to SimpleDB in our code.  I think we started off with only one Simol client instance but then we changed it at some point :)... apparently, we shouldn't have changed it.  So, yeah, it shouldn't be hard at all for us to put it back so that it just reuses the same Simol client instance.

The only thing that I'm a bit confused about now that I'm aware of this... is my own code :)... because It's been long enough that I can't actually recall why I started creating new client instances.  So, there's that.  It's possible that I  started creating new client instances because of something similar / along the same lines as @bdm mentioned?  Truth is though, the reasoning is totally erased from my memory [lots of code written since then :) ].  @bdm, I know our issues may or may not be related and this is just speculation... but I wouldn't mind knowing if you have a single Simol client instance that is reused?... or do you always create new Simol client instances?

At one point, I do remember that I was very seriously considering creating a pool of like 10 Simol clients.  Again, can't remember why :).  Lol, apparently I have a bad memory.  So, I may end up doing that just to be on the safe side / just in case I was having difficulties with a single reusable Simol instance.

At any rate, I think you've pointed me in the right direction and I would say that looking into the number of Simol client instances that I am creating will help my situation.  I wasn't thinking about that aspect for some reason.

Simol is awesome!  Thanks for your help!

_

Mat

Dec 11, 2011 at 11:32 PM
Edited Dec 11, 2011 at 11:32 PM

This does naturally lead into he topic that @bdm has mentioned...

If I'm reusing the same Simol client then... now at this point I'm curious and wondering what is the best way to check for a state such as "closed connections".  @bdm, it seems to me that to deal with the scenario that you have mentioned you would just want to do a check to make sure the connection is open before fetching any data via the Simol client.  Only thing is, I'm not aware of the "best" ways to check for an open connection and reopen it if it's closed.  I'll research this a bit and post what I find.  I'll check the Simol and AWS Sdk documentation of course :).

_

Mat

Dec 12, 2011 at 12:25 AM

I use a single client instance.  As for checking before every CRUD, there has to be a better way :)

Dec 12, 2011 at 1:07 AM

Cool, thanks for the input @bdm!  Yes, you are correct checking before every CRUD is sort of tedious; however, it wouldn't be hard to make your own object that just wraps the Simol client and exposes a simple public property where the getter is defined to do any necessary checks before it returns the Simol client object.

Coordinator
Dec 12, 2011 at 12:43 PM

Keep in mind that the AWS library automatically retries failed requests (3 times by default) before throwing an exception back to the caller. So there could be a bug in the AWS retry logic or you may want to bump up the number of retries (it's a configurable property).

Coordinator
Dec 15, 2011 at 8:20 PM
mtipton wrote:
At one point, I do remember that I was very seriously considering creating a pool of like 10 Simol clients.  Again, can't remember why :).  Lol, apparently I have a bad memory.  So, I may end up doing that just to be on the safe side / just in case I was having difficulties with a single reusable Simol instance.

Keep in mind that the Simol cache is tied to your client instance. So 10 or 100 or 1000 client instances means that many copies of your cached data if caching is enabled (it is by default).

There really is no need to create multiple client instances unless you require different configurations for different types operations (consistent vs eventually consistent reads, for example).

Dec 15, 2011 at 8:48 PM
Edited Dec 15, 2011 at 10:52 PM

Hi again, and thanks for all of the responses, Ashley!

After looking into this more the issues that I'm seeing are actually being caused by the AWS SDK library.  I hadn't looked at the source for the AWS SDK library much but after looking at it... the main problem for me is that it's retry conventions are very odd.  There are just some all around aspects that could be implemented much better in the AWS SDK library.

I will switch over to a single instance and see if that helps.  Even when I do change to a single instance I think these underlying issue in the AWS SDK will still be relevant.  Maybe a little less relevant but it's still a concern... so for now I've made some customizations to the AWS SDK.  For instance one change that we've made is to tweak the backoffs for the retries... currently if there is a timeout in the AWS SDK and it has to reconnect then it can take quite a bit of time because it does some exponential backoffs with the retries when connecting.  While this type of backoff is technically good practice in general their rate of back off seems a bit ridiculous.  I can't remember the exact numbers but 3 retries amount to like a 2 minute wait... which is very bad.  So, I've customized the backoff to be much less drastic and it has actually helped quite a bit.

Anyway, I thought I'd give a quick update on the customizations that I've made in the AWS SDK.  I will also get it switched over to just use a single client instance and can give an update on that as well.  It might be a few days before I can do that because I'm in a release crunch currently.

Thanks again for all the advice!