Running Ephemeral Dataproc Clusters on Airflow using GCP Composer
What is Dataproc? Dataproc is Google Cloud’s fully managed service for running Apache Spark, Hadoop, and other open-source data processing tools. It excels at handling large-scale data processing and pipeline operations. Ephemeral Dataproc clusters are temporary clusters that: Are created on-demand when processing is needed Automatically terminate once their tasks are complete Help optimize costs by only running when necessary Can be managed through automation tools like Airflow or Terraform What is Cloud Composer?...
Socket IO & Adapter When Using Multiple Servers
Why Websockets? Limitations Of HTTP With HTTP, client requests a resource and the server responds with the requested data. It is a unidirectional communication - the data must be first requested by the client. A workaround for this limitation was the HTTP long polling. With long polling, client makes a http request with a long timeout period and the server uses that time period to keep sending data to the client....
Docker: Images & Layers
What Is Docker? Docker is a tool to build, deploy and run applications, on any platform irrespective of the infrastructure, with the use of containers. A container is a unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. Since docker packages all the dependencies of an application together with the required version, it solves the problem of having missing dependencies or version conflicts on developer’s local environment or production environment....
Node JS: Event Loop
Despite being single threaded, Node-Js allows asynchronous programming and non-blocking operations by using Event loop. Node has an event driven architecture and code is executed is in the form of callbacks. The Event Loop schedules the callbacks to be run in the single thread at a given point. Libuv is the library that provides the implementation of event loop and thread pool in Nodejs. When a node process starts execution, the top level code is executed first, and all the callbacks are registered....
JS Engine And Execution Context
Javascript engines are programs that convert Javascript code into native machine code. These engines are embedded into the browser for runtime compilation and execution of the code. Google chrome uses V8 engine, Safari uses JavaScriptCore, and Firefox uses SpiderMonkey. To transform and run code faster, modern javascript engines use Just In Time (JIT) compilation, which is a combination of both interpretation and compilation. How JS Engine Works Parser...