Skip to main content

javax.servlet.AsyncContext.start() limited usefulness

Some time ago I came across What's the purpose of AsyncContext.start(...) in Servlet 3.0? question. Quoting the Javadoc of aforementioned method:

Causes the container to dispatch a thread, possibly from a managed thread pool, to run the specified Runnable.


To remind all of you, AsyncContext is a standard way defined in Servlet 3.0 specification to handle HTTP requests asynchronously. Basically HTTP request is no longer tied to an HTTP thread, allowing us to handle it later, possibly using fewer threads. It turned out that the specification provides an API to handle asynchronous threads in a different thread pool out of the box. First we will see how this feature is completely broken and useless in Tomcat and Jetty - and then we will discuss why the usefulness of it is questionable in general.

Our test servlet will simply sleep for given amount of time. This is a scalability killer in normal circumstances because even though sleeping servlet is not consuming CPU, but sleeping HTTP thread tied to that particular request consumes memory - and no other incoming request can use that thread. In our test setup I limited the number of HTTP worker threads to 10 which means only 10 concurrent requests are completely blocking the application (it is unresponsive from the outside) even though the application itself is almost completely idle. So clearly sleeping is an enemy of scalability.
@WebServlet(urlPatterns = Array("/*"))
class SlowServlet extends HttpServlet with Logging {

  protected override def doGet(req: HttpServletRequest, resp: HttpServletResponse) {
    logger.info("Request received")
    val sleepParam = Option(req.getParameter("sleep")) map {_.toLong}
    TimeUnit.MILLISECONDS.sleep(sleepParam getOrElse 10)
    logger.info("Request done")
  }
}

Benchmarking this code reveals that the average response times are close to sleep parameter as long as the number of concurrent connections is below the number of HTTP threads. Unsurprisingly the response times begin to grow the moment we exceed the HTTP threads count. Eleventh connection has to wait for any other request to finish and release worker thread. When the concurrency level exceeds 100, Tomcat begins to drop connections - too many clients are already queued.

So what about the the fancy AsyncContext.start() method (do not confuse with ServletRequest.startAsync())? According to the JavaDoc I can submit any Runnable and the container will use some managed thread pool to handle it. This will help partially as I no longer block HTTP worker threads (but still another thread somewhere in the servlet container is used). Quickly switching to asynchronous servlet:

@WebServlet(urlPatterns = Array("/*"), asyncSupported = true)
class SlowServlet extends HttpServlet with Logging {

  protected override def doGet(req: HttpServletRequest, resp: HttpServletResponse) {
    logger.info("Request received")
    val asyncContext = req.startAsync()
    asyncContext.setTimeout(TimeUnit.MINUTES.toMillis(10))
    asyncContext.start(new Runnable() {
      def run() {
        logger.info("Handling request")
        val sleepParam = Option(req.getParameter("sleep")) map {_.toLong}
        TimeUnit.MILLISECONDS.sleep(sleepParam getOrElse 10)
        logger.info("Request done")
        asyncContext.complete()
      }
    })
  }
}
We are first enabling the asynchronous processing and then simply moving sleep() into a Runnable and hopefully a different thread pool, releasing the HTTP thread pool. Quick stress test reveals slightly unexpected results (here: response times vs. number of concurrent connections):
Guess what, the response times are exactly the same as with no asynchronous support at all (!) After closer examination I discovered that when AsyncContext.start() is called Tomcat submits given task back to... HTTP worker thread pool, the same one that is used for all HTTP requests! This basically means that we have released one HTTP thread just to utilize another one milliseconds later (maybe even the same one). There is absolutely no benefit of calling AsyncContext.start() in Tomcat. I have no idea whether this is a bug or a feature. On one hand this is clearly not what the API designers intended. The servlet container was suppose to manage separate, independent thread pool so that HTTP worker thread pool is still usable. I mean, the whole point of asynchronous processing is to escape the HTTP pool. Tomcat pretends to delegate our work to another thread, while it still uses the original worker thread pool.

So why I consider this to be a feature? Because Jetty is "broken" in exactly same way... No matter whether this works as designed or is only a poor API implementation, using AsyncContext.start() in Tomcat and Jetty is pointless and only unnecessarily complicates the code. It won't give you anything, the application works exactly the same under high load as if there was no asynchronous logic at all.

But what about using this API feature on correct implementations like IBM WAS? It is better, but still the API as is doesn't give us much in terms of scalability. To explain again: the whole point of asynchronous processing is the ability to decouple HTTP request from an underlying thread, preferably by handling several connections using the same thread.

AsyncContext.start() will run the provided Runnable in a separate thread pool. Your application is still responsive and can handle ordinary requests while long-running request that you decided to handle asynchronously are processed in a separate thread pool. It is better, unfortunately the thread pool and thread per connection idiom is still a bottle-neck. For the JVM it doesn't matter what type of threads are started - they still occupy memory. So we are no longer blocking HTTP worker threads, but our application is not more scalable in terms of concurrent long-running tasks we can support.

In this simple and unrealistic example with sleeping servlet we can actually support thousand of concurrent (waiting) connections using Servlet 3.0 asynchronous support with only one extra thread - and without AsyncContext.start(). Do you know how? Hint: ScheduledExecutorService.

Postscriptum: Scala goodness

I almost forgot. Even though examples were written in Scala, I haven't used any cool language features yet. Here is one: implicit conversions. Make this available in your scope:
implicit def blockToRunnable[T](block: => T) = new Runnable {
 def run() {
  block
 }
}
And suddenly you can use code block instead of instantiating Runnable manually and explicitly:
asyncContext start {
 logger.info("Handling request")
 val sleepParam = Option(req.getParameter("sleep")) map { _.toLong}
 TimeUnit.MILLISECONDS.sleep(sleepParam getOrElse 10)
 logger.info("Request done")
 asyncContext.complete()
}
Sweet!

Comments

  1. Why not use Netty? What are the advantages of using the asynchrous servlet API over Netty?

    ReplyDelete
    Replies
    1. I have never used Netty, certainly I should give it a try. But the main advantage is: Servlet 3.0 is a standard - and not a bad one when implemented and used correctly.

      Delete
  2. While I can understand your points, I would like to add a different perspective (I am a Jetty core developer).

    I think the assumption that Runnables passed to AsyncContext.start() must run in a pool different from the "HTTP pool" is a bit stretched, and I think the spec is vague on the matter on purpose. I think that only applications know exactly what kind of tasks they run, and what kind of pool fits them better, and it's perhaps asking a bit too much to the servlet container to fit every use case. A typical example is when applications need to run FJ tasks: the best pool for that is ForkJoinPool and not other implementations. Applications needs perhaps to perform different kind of tasks that suit different type of pools, and the servlet container cannot guess when a certain type of pool is needed over another type.

    Hence, Tomcat and Jetty default on running those unknown tasks on their "HTTP pool", so that they can offer a single point of resource control (and that is a feature): a single thread pool to configure.

    I am not sure I understand your "unrealistic" example with ScheduledExecutorService: if a scheduled task runs, and never returns because it sleeps for minutes, no other scheduled task can run, if the ScheduledExecutorService only has one thread.

    Which brings me to your example, which does synchronous sleeping rather than asynchronous processing.
    If you replace your sleeps with real asynchronous processing like asynchronous I/O, for example by performing an upstream HTTP request with Jetty's asynchronous HTTP client (http://wiki.eclipse.org/Jetty/Tutorial/HttpClient), you'll see that you won't be blocked in the Runnable, so you can send the upstream request asynchronously, return the thread to the pool to run other requests, and process the upstream response asynchronously too.
    There are the first signs of asynchronous RDBMS drivers, that will leverage this style of asynchronous programming - synchronous sleeping is just not the right usecase. WebSocket and JDK 7's AsynchronousChannels will leverage this style too.

    Thread pool tuning is very much tied to what the application needs: you can starve a servlet container, and make it unresponsive, if you start N threads on a N-core system, and those threads burn each 100% of the core cycles (from spin-looping to math calculations, etc.).
    It does not matter much if you started the N threads from another pool, from calling new Thread().start() or from the "HTTP pool".
    The point is that only the application knows better.

    AsyncContext.start(Runnable) certainly has a short and vague Javadoc, though :)

    ReplyDelete
    Replies
    1. First of all it is great to hear someone from the Jetty team. Seems like the poorly defined semantics and lack of implementation guidelines is a major issue with that method. However as an API consumer I would expect that AsyncContext.start() uses a different pool. Period. After all the sole purpose of starting an asynchronous processing is to release an HTTP thread. Not to release the current one and pick different one from the pool. This doesn't buy as much.

      I don't agree that having a single thread pool is an advantage. Of course it is easier to maintain a system with less configuration options, but once I am into fine-tuning, having a single pool serving completely different purposes is a pain. The same way you could say that application servers should have a single pool for handling HTTP connections and incoming JMS messages - because there is only one pool to configure. Of course I am exaggerating a bit and I see your point. But I would at least want to have an option between reusing HTTP pool and an external one (for power users).

      I agree that only the developer knows what kind of pool she needs. Fork-Join vs. standard pool, full control over queueing and rejecting policies, etc. But in this case why are we giving a tool (AsyncContext.start()) that works as a black box and the developer never knows what is the underlying mechanism? This only proves my points that this method is very limited and not really portable.

      Back to my example with sleeping. My point was that sleeping isn't really common when serving HTTP requests. And if we really need to wait a second or two, we should utilize ScheduledExecutorService to run some code after the delay. This way we are not holding the thread without purpose. In other words: instead of sleeping for 10 seconds and running some code after that - schedule that code to run after 10 seconds. Much more scalable. The same goes for synchronous HTTP client calls. Instead of using synchronous HttpClient one should consider asynchronous clients like the one you mentioned (thanks). I actually have an article about that, just need to translate it from my native language.

      Nevertheless great thanks for your thorough and in-depth response.

      Delete
    2. Tomasz, I think it would help if you could come up with a better example than "sleep". Who the fuck sleeps in a thread? It's almost like reading a joke when you write: "schedule that code to run after 10 seconds. Much more scalable". More scalable sleep code?

      Find a real example and go from there.

      Delete
    3. @Anonymous: indeed, more real life example would be useful. Probably some non-blocking IO in a servlet compared to blocking approach. I didn't want to make the example too complicated, but you are right. On the other hand, I can imagine sleeping in a servlet in rare cases, e.g. you read some device status, wait 1 second, read it again and return the difference. Of course one can come up with a better design, just an example.

      Thank you for your comment, I'll try to write a follow-up article with more real-life example.

      Delete
  3. There are many scenarios for async processing and it's hard to generalize. Using a ScheduledExecutorService, a single thread can process many suspended requests, passing new information as it becomes available from some external source (e.g. stock prices). Or to run a long database query, it might be still necessary to dedicate a thread.

    Clearly, the first scenario has very low overhead with only one thread. In the second scenario a servlet container thread is released only to create another thread to run the database query. It's true that a thread is a thread to the JVM but the HTTP servlet pool is optimized for processing servlet requests, so the value is in selectively spawning separate threads where async processing can alleviate the burden on the Servlet container thread pool in exchange for a thread pool tuned to application-specific needs.

    I am a Spring MVC developer and I've blogged about Spring MVC 3.2, which brings the Servlet 3 async feature to the application level allowing choices that depend on the specific scenario. It would be great to get your thoughts if you have specific scenarios in mind.

    ReplyDelete
    Replies
    1. I will have a closer look to Spring MVC asynchronous support (waiting so long for that!) soon. I know you are a Spring MVC developer, you helped me a lot on SO.

      Yes, I believe that scenarios with one-to-many relationships between thread and request are the most valuable. I wish there was a mature JDBC-like library performing asynchronous database calls (after all the database connection is just an open socket - we don't need to wait for the response synchronously!) I am currently experimenting a lot with actor-based systems (namely: Akka) so I am looking for asynchronous libraries where possible. Be sure I will post some articles about Spring MVC and Akka integration exploiting asynchronous processing of web requests.

      Delete
    2. Hey Sp-MVC and jetty guys...

      We are trying out some async support using Jetty and SMVC, and we are trying to also use the deflation filter that jetty has.

      Our "implementation" of the service only returns a string, so it is very fast, but we are seeing strange behavior: sometimes the deflation occurs, sometimes it doesn't, in a random fashion. Almost like the response thread may not know what filters to run or not...

      Apparently this occurs only when we use SMVC 3.2 async support. Any ideas?

      Delete
  4. Hi

    The title "javax.servlet.ServletRequest.startAsync() limited usefulness" is bit of miss leading, I think that what you mean is

    "javax.servlet.AsyncContext.start() limited usefullness" , as we always need to create a AsyncContext to be able to return the result to client, however we can pass the AsyncContext to a Thread of ThreadPool that is different from the default one for further processing.

    BTW your example of using SCALA is sweet, do you need to do anything special to make use of SCALA in servlet container, or just include scala lib is enough ?

    ReplyDelete
    Replies
    1. Ouch! You are absolutely right! I changed the title, thanks. Wrt to Scala, yes, just add scala.jar (quite big, unfortunately). Scala compiles to very pure JVM bytecode, however all Scala objects inherit from scala.ScalaObject, so extra library is required.

      Delete
  5. Hi.

    Reading the comments I saw some people claiming about a more realistic use case. But you had already blogged [1] about a very useful use case here in your blog. Isn't this a real use case to use Async Context support in Sevlet 3.x? Or am I confusing the concepts...

    [1] http://nurkiewicz.blogspot.com.br/2011/03/tenfold-increase-in-server-throughput.html

    ReplyDelete
    Replies
    1. You are right. I just believe that asynchronous processing has so many different use-cases that it deserves proper examples. That's why I want to write a little bit more about it. Also, since it's a standard, we should look for opportunities to use it more often.

      Delete
  6. This comment has been removed by a blog administrator.

    ReplyDelete

Post a Comment