you are evicted! now! go! - scheduler has no mercy...

 In one of projects we had a challenging situation, our data process flow was interrupted by evicted pods in Kubernetes. That was occurring when our system was in a busy hour mode so incoming stream of data was up to 30MB/s. In the pod we had a C# program that was transposing incoming data for a database entity and pushing that over http request. Our scheduler was prepared to scale up and out, so we could use more hosts and pods when in the busy time, but that was not working as expected... When new pod was created all the data connections were directed to it and ..... bum eviction (to much  memory consumed) - as it was buffering all the data and not able to push it out to http.

Our precios data in transit was lost, next pod had a similar scenario - so in short words we were screw up.

We had some ideas to fix it.

  1. let's check memory usage after each packet received (huge performance drop - we had to have 4x more pods for a standard traffic and up to 6 in peek time)
  2. let's limit number incoming connections to one, so one worker node handles one stream of data (effect good, kube cluster hosts allocation 2x)
  3. GC (garbage colector) was useless in this case as all the http data was a json in a string and that was the memory consuption
  4. so maybe stop the incoming connection when we reach like 1GB of used memory - as this is the best idea - but in linux container we were getting garbage data (and on windows host this snipped was working good):


The time was ticking and the $$ was burn as well in the astronomic speed of light. 
Ok, so next step was to try to use pod liveliness probe that can be used by pods scheduler and schedule new pod if there is not enough available workers. All details here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/


With that crafted solution our pods were terminating incoming connection, flushing their internal data to the http destination and resume their work after that. 

How I will fix it today? Hmm probably I will use akka.net streams as they will give us a backpressure signal so the incoming data will be rated and pods scheduler will have some time to prepare new workers. Akka streams with tcp example here: https://getakka.net/articles/streams/workingwithstreamingio.html


Comments

Post a Comment

Popular posts from this blog

when a new guy joins a team...

Image
Not so long ago, I was browsing merged pull requests to check quality of the work that was done. I don't know why, but in 202X C# is rather a modern language and some language syntax shall be adjusted to newer standards. When I was starting my developer journey - I was told to always add a code that looks similar to the one that already is there, unless it is total crap and adding new stuff needs more effort today, to not to die tomorrow. But in this case, we have a project that is created in in 202X and there is a lot of good looking and working lines of code around, so anyone can follow it. First time, when I was a student and helped my colleague, I was in shock that the teacher recognized my help, I turned all red and asked him "how you were able to recognize that?". He said that most of us have a kind of style that identifies us as developers, so this is spacing, the way how we organize braces (even when autoformated), how we write and solve problems,  and finally var
Read more

FizzBuzz - my first interview task on whiteboard

Image
 I think we all know the FizzBuzz task, when we a function shall print fizz if the number is divided by 3, buzz if by 5 and fizzbuzz when by both. Yesterday I did a small research about how many different implementations we can have using C# and there were many. Ones looks very common with "if"   statement tree, some of them were using LINQ and looking hard to understand what is going on in a first instance. StackOverflow has this one question with a bunch of examples  Writing FizzBuzz , that I will use to check how those solutions work from a performance perspective, code complexity and human understanding. listing 1 fizzbuzz question As we can see, OP introduced a tuple to solve the problem, which in my opinion gives a lot of opportunists for extensions with a little code changes.  The metric looks good, but if we move the list declaration outside the function (so it is not recreated every time the function is called we shall be good) To do benchmark I will convert
Read more

are YOU a garbage collector?

Image
 If I will ask you: "Are you a garbage collector" - I bet I will hear: " NO ", so I ask you: show me your code.... It is unbelievable that so many developers are pushing changes with commented out code, because  there could be a chance to re-use that piece. In fact that time never comes to me and the lines stays there forever. Clean code means no rubbish commented code - we have git to remember that for us. Now, when you need to comment a few lines in a row... there are better solutions, but if someone is lazy and slumsy then we can see diffs on code review like that: Now this is a mystery for me, when I am able to see that this is going to the main branch as the guys who is called code reviewer is there only to click Approve -  I am truly believing that it is not you my reader. I need to admit that, when I was a kid, together with grandma we were collecting garbage as a way to get some $$ for a life and then every piece we found had a material value and were able t
Read more