How it works

*Scalability:separates topics into partitions.because Kafka is distributed,you can scale up by adding new nodes to the cluster *High throughput:it is capable of handling massive volumes of incoming messages at a high velocity per second *low latency: Latency refers to the amount of time required to process each message

0
Mahmoud valizadeh
Why Kafka
Random
M
picture loading error handler
60 thought(s)60

Microservice

microservices
Kafka
Mahmoud valizadeh
Get

Explore more quotes

Email:70 valizadeh@gmail com111746 1g3scst.71hujif6r
Set Of Practices increase speed scale technology ignostic principles and architetural patterns
Mahmoud valizadeh
Micro Big Or Small No Universal Measure does on thing scope of functionality Identify Subdomain and each subdomain is one microservice
Mahmoud valizadeh
The microservice architectural style is an approach to developing a single application as a suite of small services each running in its own process and communicating with lightweight mechanisms
james lewis and martin fowler
service Independantly deployable component (IDC) Interopability Message Base Communication service oriented Architecture
Mahmoud valizadeh
some generally accepted attributes that make up a microservice: *Small and single in purpose *Communicate via technology agnostic protocols *Support continuous integration *Independently deployable.
Mahmoud valizadeh
Pros independantly Deploy_A microservice can be developed by a fairly small team_can be written in different languages_Easy to understand and modify for developers_The code is organized around business capabilities_Better fault isolation: if one microservice fails, the other will continue to work
Mahmoud valizadeh
Cons Due to distributed deployment, testing can become complicated and tedious_When number of services increases, integration and managing whole products can become complicated_Developers have to put additional effort into implementing the mechanism of communication between the services
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 19ktwmm.59lkcblnmi
The simplest way to render multiple web applications into a single one is by using HTML iframes.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1oevsa.yvth9kvs4i
but iframe has one url point by contrast microservice has multiple instances in the network.!!!! we need a mechanism to route a single URL to instances of the correct web application. This can be done by introducing a router
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1vl3sr0.5qv76kzkt9
Microservices have to interoperate with multiple microservices in order to complete any complex functionality. For this purpose, services use an inter-process communication protocol. The solution approach is to leverage the RPI for any inter-service communication and collaboration.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 sm7v9l.8adejexw29
gRPC is a language agnostic, high-performance Remote Procedure Call (RPC) framework.In gRPC, a client application can directly call a method on a server application on a different machine as if it were a local object, making it easier for you to create distributed applications and services.
Mahmoud valizadeh
benefits Of grpc: *Modern, high-performance, lightweight RPC framework. *Contract-first API development, using Protocol Buffers by default, allowing for language agnostic implementations *Tooling available for many languages to generate strongly-typed servers and clients.and ...
Mahmoud valizadeh
gRPC ideal for: *Lightweight microservices where efficiency is critical. *Polyglot systems where multiple languages are required for development. *Point-to-point real-time services that need to handle streaming requests or responses.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 11qa5mc.4m1wbnvcxr
By default, gRPC uses Protocol Buffers, Google’s mature open source mechanism for serializing structured data (although it can be used with other data formats such as JSON).
Mahmoud valizadeh
Rest Vs RPC REST is resource-oriented _ RPC is action-oriented. Rest Supports HTTP methods GET, POST, PUT, PATCH, and DELETE. RPC only supports GET and POST requests. Rest It provides flexibility for hardware architecture_rpc no flexibility in RPC for hardware architecture
Mahmoud valizadeh
Rest Vs RPC Rest It supports hypermedia and hyperlinks while RPC It does not supports hypermedia and hyperlinks REST allows to specify Content-types or accept headers While RPC Require payloads of a few data types as XML for XML-RPC.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1y7nhrm.9si024vx6r
Thechnologies : REST gRPC Apache Thrift
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1b888oe.d7egiafw29
Microservices Service Registry A service registry is a database used to keep track of the available instances of each microservice in an application. The service registry needs to be updated each time a new service comes online and whenever a service is taken offline or becomes unavailable.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 wmdp6y.n4fhl4n29
Self-Registration With self-registration, each microservice is responsible for adding itself to the service registry when it comes online. for this exposes a rest api for registration send post request for remove send delete As a fail-safe, service instances can send a heartbeat
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 14eaxa1.7vygo2bj4i
Third-Party Registration Third-party registration offers a more loosely coupled solution: a service registrar (also known as a service manager) is responsible for registering each new instance as it comes online and de-registering it when it becomes unavailable.
Mahmoud valizadeh
Discovering Services In order to send an API request to a service, the client or API gateway must know the location of the service that it’s addressing. service discovery is closely linked to load balancing of requests to instances of each microservice. *load balancing requests across service instances
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1yy2mrl.xb1uac3di
Client-Side Discovery When making a request to a service, the client obtains the location of a service instance by querying a Service Registry, which knows the locations of all service instances.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1ff2ng8.ivknr35wmi
Server-Side Discovery When making a request to a service, the client makes a request via a router (a.k.a load balancer) that runs at a well known location. The router queries a service registry, which might be built into the router, and forwards the request to an available service instance.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1n7a028.mm6kihpvi
Service Discovery Tools Docker ZooKeeper Hysterix HashiCorp Consul GRPC Eureka
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 byxm41.a9qqv3g14i
Circuit Breaker Pattern With the help of this pattern, the client will invoke a remote service through a proxy. This proxy will basically behave as an electrical circuit breaker.
Mahmoud valizadeh
Circuit Breaker Pattern Number Of Failures ↑ Then the circuit breaker trips for a particular time period. With this, all attempts request for period of time will fails.After the timeout expires it allows for some requests. if succeed=> it will goes to normal state Else=>the timeout period begins again.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 xz07y0.b1e02otj4i
As a library, the CB pattern can be found on Hystrix (Java), Polly (.Net), Resilient4j (Java), Akka (Scala), Rubyist (Go), pybreaker (Python), JRugged, and Javaslang. Apart from those
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1yphbbu.ndkkjra4i
Api Gateway the API gateway locate between the client apps and the internal microservices. It is working as a reverse proxy and routing requests from clients to backend services. It is also provide cross-cutting concerns like authentication, SSL termination, and cache.
Mahmoud valizadeh
Api Gateway Handling application-specific authorization in the gateway _Restricting requests to specific endpoints become in this way possible. The drawback of implementing authorization in the API gateway is that it can be only role-based access to endpoints.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 2d536u.dsz5n7mn29
Scalability is simply measured by the number of requests an application can handle successfully. It can be measured by the number of requests and it can effectively support simultaneously. Once the application can no longer handle any more simultaneous requests, it has reached its scalability limit
Mahmoud valizadeh
Horizontal scaling and Vertical scaling both involve adding resources to your computing infrastructure, you must decide which is right for your application. Scaling horizontally and scaling vertically are similar in that they both involve adding computing resources to your infrastructure.
Mahmoud valizadeh
Horizontal VS Vertical Horizontal scaling means scaling by adding more machines to your pool of resources (also described as “scaling out”), whereas Vertical scaling refers to scaling by adding more power (e.g. CPU, RAM) to an existing machine (also described as “scaling up”).
Mahmoud valizadeh
Vertical scaling keeps your existing infrastructure but adding more computing power.
Mahmoud valizadeh
if you have get millions of request, in that case having one server won’t be enough, because even the hardware has maximum capacity limitations. In this case we need to do horizontal scaling or scaling out.
Mahmoud valizadeh
Scaling horizontally gives you scalability but also reliability because you will have more redundancy and mostly its the preferred way to scale in distributed architectures.
Mahmoud valizadeh
horizontal scaling When stateless then its easy.just need a load balancer when statefull then its hard
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1f1gyev.jgr946xbt9
Kafka is a publish/subscribe messaging platform with built-in support for replication, partitioning, fault tolerance, and better throughput. It's an excellent choice for applications that need large scale data processing. incorporates : *fault-tolerant storage *stream processing capabilities
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 dnqqqy.xg1gs3jtt9
Features: 1)It can publish and subscribe streams of data. 2)It's capable of handling a vast number of read/write operations per second. 3)It can persist data for a particular period. 4)It has the ability to grow elastically with zero downtime. 5)It offers support for replication, partitioning, and fault-tolerance.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 k2fdy6.3e40w6zuxr
*High performance:provides high throughput with low latency and high availability *Highly reliable:in kafka the messages are durable even after they have been consumed.it can load balance consumers in the event of a failure *Durability:stores the messages on the disk,as opposed to in memory
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 bosikw.j4o08s38fr
use cases for Kafka: *Application activity tracking: You can leverage Kafka to publish all events (user login, user registration,etc.) that occur in your application to a dedicated Kafka topic.Then consumers subscribe to the topics and process the data for monitoring,analysis, etc.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 81djzl.5w5rpy14i
use cases for Kafka: *Log aggregation:You can publish logs to Kafka topics and then aggregate and process them when needed. Kafka can collect logs from various services and make them available to the consumers in a standard format (JSON). *Real-time data processing: applications need real-time data processing.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1kv2w1.wtn81l9pb9
use cases for Kafka: *Operational metrics: Kafka can aggregate the statistical data collected from several distributed applications and then produce centralized feeds of operational data.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 by8kf3.h31lfecdi
Kafka Components Architecture: 1)Kafka Topic:A Topic is a category/feed name to which records are stored and published.Each topic has a name that is unique across the entire Kafka cluster. 2)Kafka Cluster:A Kafka cluster is a system that consists of several Brokers, Topics, and Partitions for both.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1e1jyew.oqxzx2mx6r
*Kafka Producer:A Kafka producer serves as a data source for one or more Kafka topics and is responsible for writing, optimizing, and publishing messages to those topics. A Kafka producer can connect to a Kafka cluster through Zookeeper. Alternatively, it can connect to a Kafka broker directly.
Mahmoud valizadeh
Kafka Producer Serialize data & Compresses data & load balances data & among kafka brokers throught partitioning
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 i7mpzk.o7rphnnrk9
*Kafka consumer: Kafka Consumer to subscribe to Kafka topics and receive messages from these topics. *Kafka ZooKeeper: A Kafka Zookeeper manages and coordinates the Kafka brokers in a cluster. It also notifies producers and consumers in a Kafka cluster of the existence of new brokers or the failure of brokers.
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 piq2ad.eysxzgp66r
*Kafka Broker:A broker is an individual person who arranges transactions between a buyer and a seller for a commission when the deal is executed. A Kafka cluster has exactly one broker that acts as the Controller.(control events,monitor)
Mahmoud valizadeh
Email:70 valizadeh@gmail com111746 1cqrv6.qbggxbhuxr
*Kafka broker : *A Kafka broker receives messages from producers and stores them on disk keyed by unique offset. *A Kafka broker allows consumers to fetch messages by topic, partition and offset. *Kafka brokers can create a Kafka cluster by sharing information between each other directly or indirectly using Zookeeper.
Mahmoud valizadeh