actor or standard class? - part 2

In previous post an actor was benchmarked against a standard class, and for a single user there was almost no diffrence in performance, but with multiple issues a class run in to a concurrent access problems, so the performance dropped more than 10prc and we got some not processed requests ended with 500. 
Now we will replace storage with a MariaDB database (why Maria? - my friend asked for help and  I just used it here as well) and run those  parallelly to exhaust  DB capabilities.

As we can see, JMeter was spreading the workload, and we hit the database limit which is around 1500 requests/sec.

Now our business guys come to us and ask: we need it faster, don't send us Id of the created record - as we don't needed, we need to save at least 5x more records.

So how we can do it?
Hmm. The best way will be to bulk insert, like 250 records at once, or every 1 second depending on the traffic.

In first case let's focus on enabling a bulk save, which will be a counter and an if statement

An actor can be utilised as FSM (Finite State Machine) and process incoming requests in a way that a current state is demanding.

Below on listing1 incorporated changes to change our storage provider.

listig1

Now let's modify our actor to work with it's internal state.

After modification a quick test to see if we gained anything (WARNIG: I DISABLED LOGGING TO CONSOLE AS THAT WAS SLOWING THIS THING AS HELL)

I think with two test run in the same time, we are losing some capabilities. Let's run those guys in separate tasks and JMeter will produce same results table for us.

So let's check how we are going , when we were testing our solution in separation.

Now - we need to be honest - the class method does not have any queueing mechanism, so it is writing record by record - and a quick change in the actor behavior allowed us to see the change.

But  - in fact all the tests shall be re-run as console logger was disabled and we drastically changed testing environment. 

To make it comparable a few lines were added to our controller to handle bulk writing as well - pls see listing2.

First run in sync (after my PC was restarted) and I was amazed by the total speed of the solution.

8k requests per second handled on a single machine. Just amazing.

So now we could go crazy with a premature optimalisation, but's let keep things simple. In fact, I just realised that switching behavior for this case is just an overkill, we really don't need to do it.

now on listing2 all changes so far

listing2 

Now - let's try to solve another problem. Our storage engine needs 5 seconds for a save operation, we are under constant load of around 10 - 15 users, we need to have average response time less than 4 second and provide the new element ID.

How that could be done?

So for a class solution it is a kind of easy peasy stuff. Our storage engine got a Thread.Sleep(5500) and now let's see what is going on:

As the actor processes only 1 message at a time, the whole system was just stopped, and the response time went up to the sky (as you can see the test was cancelled after a few probes).

So what we can do? How we can scale this? How we can create more actor to deal with that situation?

The great thing is that we don't need to change our actor at all, but we need to produce more actors and distribute the work to them by for example a round robin way. When creating an actor we need to add a routing strategy and we are good to go.

So, let's check te results:

As expected, with 15 actors we are on same level as our classic way and here is the question: why we need to keep all 15 workers in the memory for all the time? This is inefficient, so we can add dynamic scaling based on number of messages in an actor inbox, but that will slowdown our processing, as some of the workers will receive 2 or 3 messages before the other will be ready to use. See the docs here

var bookActor = system.ActorOf(Props.Create(() => new BookActor()).

     WithRouter(new RoundRobinPool(15, 

      new DefaultResizer(

     lower: 1, upper: 15, pressureThreshold: 1, rampupRate: 5, messagesPerResize: 1))), "book");

As expected our dynamic approach is slower than a classic one. But what we can do to be faster? Can we? In a previous case - there was no need to return id of the item, so bulk write was possible. Can we manage this with an actor easily?

Let's try and see.

As an actor has an address, we can intercept a sender address and store it with the document data, so when it is saved - we can forward the id back. For that we will use tuple to store sender actor reference. Moreover we will use a trick. As an actor by design can process only one message at a time, we still have ability to 'force' it to make a long processing in the background in an async mode.

Now to be honest - we shall adjust our classic approach to use async as well, to be fair at benchmark.

Results are are very good, so it looks like both solutions are making the job.

listing 3 all async

As we can see on the listing, the actor get a new stuff called PipeTo. This allows us to use async non blocking process to get data from a slow storage and in  fact process more than 1 message at a time.

We also learned, that even if the other side use ASK, we can postpone processing and save caller reference, so we can reply later.

Last challenge in this post.

One of the service provider has a rate limiting api, where every call takes 10 seconds. They allow to reduce the time by: 

• 5 seconds when a customer makes only 30 requests per minute, 
• 0.1 per element, where request have at least 3 consolidated request elements 

So that means, we need to make a call every 2 seconds to gain some time, and if we consolidate, then we can gain more. 

To gain that we will create an internal timer, that will be executing every 2 seconds and process our data as needed.

So we will use PreStart stage (doc here) to create a timer that will allow a system scheduler to send message to our actor every 2 seconds. 

The results are good,

To make this in a standard class, the solution is possible, but will require more effort to make it done.

listing 4 store and forward