kafka installation: install jdk download kafka binary create "data" folder for zookeeper and kafka data => kafka , data=>zookeeper go to config/zookeeper.properties change it e.g datadir=c:/kafka/data/zookeeper go to config/server.properties changelog.dirs=c:/kafka/data/kafka

Set Of Practices increase speed scale technology ignostic principles and architetural patterns

Micro Big Or Small No Universal Measure does on thing scope of functionality Identify Subdomain and each subdomain is one microservice

The microservice architectural style is an approach to developing a single application as a suite of small services each running in its own process and communicating with lightweight mechanisms

service Independantly deployable component (IDC) Interopability Message Base Communication service oriented Architecture

some generally accepted attributes that make up a microservice: *Small and single in purpose *Communicate via technology agnostic protocols *Support continuous integration *Independently deployable.

Pros independantly Deploy_A microservice can be developed by a fairly small team_can be written in different languages_Easy to understand and modify for developers_The code is organized around business capabilities_Better fault isolation: if one microservice fails, the other will continue to work

Cons Due to distributed deployment, testing can become complicated and tedious_When number of services increases, integration and managing whole products can become complicated_Developers have to put additional effort into implementing the mechanism of communication between the services

The simplest way to render multiple web applications into a single one is by using HTML iframes.

but iframe has one url point by contrast microservice has multiple instances in the network.!!!! we need a mechanism to route a single URL to instances of the correct web application. This can be done by introducing a router

Microservices have to interoperate with multiple microservices in order to complete any complex functionality. For this purpose, services use an inter-process communication protocol. The solution approach is to leverage the RPI for any inter-service communication and collaboration.

gRPC is a language agnostic, high-performance Remote Procedure Call (RPC) framework.In gRPC, a client application can directly call a method on a server application on a different machine as if it were a local object, making it easier for you to create distributed applications and services.

benefits Of grpc: *Modern, high-performance, lightweight RPC framework. *Contract-first API development, using Protocol Buffers by default, allowing for language agnostic implementations *Tooling available for many languages to generate strongly-typed servers and clients.and ...

gRPC ideal for: *Lightweight microservices where efficiency is critical. *Polyglot systems where multiple languages are required for development. *Point-to-point real-time services that need to handle streaming requests or responses.

By default, gRPC uses Protocol Buffers, Google’s mature open source mechanism for serializing structured data (although it can be used with other data formats such as JSON).

Rest Vs RPC REST is resource-oriented _ RPC is action-oriented. Rest Supports HTTP methods GET, POST, PUT, PATCH, and DELETE. RPC only supports GET and POST requests. Rest It provides flexibility for hardware architecture_rpc no flexibility in RPC for hardware architecture

Rest Vs RPC Rest It supports hypermedia and hyperlinks while RPC It does not supports hypermedia and hyperlinks REST allows to specify Content-types or accept headers While RPC Require payloads of a few data types as XML for XML-RPC.

Thechnologies : REST gRPC Apache Thrift

Microservices Service Registry A service registry is a database used to keep track of the available instances of each microservice in an application. The service registry needs to be updated each time a new service comes online and whenever a service is taken offline or becomes unavailable.

Self-Registration With self-registration, each microservice is responsible for adding itself to the service registry when it comes online. for this exposes a rest api for registration send post request for remove send delete As a fail-safe, service instances can send a heartbeat

Third-Party Registration Third-party registration offers a more loosely coupled solution: a service registrar (also known as a service manager) is responsible for registering each new instance as it comes online and de-registering it when it becomes unavailable.

Discovering Services In order to send an API request to a service, the client or API gateway must know the location of the service that it’s addressing. service discovery is closely linked to load balancing of requests to instances of each microservice. *load balancing requests across service instances

Client-Side Discovery When making a request to a service, the client obtains the location of a service instance by querying a Service Registry, which knows the locations of all service instances.

Server-Side Discovery When making a request to a service, the client makes a request via a router (a.k.a load balancer) that runs at a well known location. The router queries a service registry, which might be built into the router, and forwards the request to an available service instance.

Service Discovery Tools Docker ZooKeeper Hysterix HashiCorp Consul GRPC Eureka

Circuit Breaker Pattern With the help of this pattern, the client will invoke a remote service through a proxy. This proxy will basically behave as an electrical circuit breaker.

Circuit Breaker Pattern Number Of Failures ↑ Then the circuit breaker trips for a particular time period. With this, all attempts request for period of time will fails.After the timeout expires it allows for some requests. if succeed=> it will goes to normal state Else=>the timeout period begins again.

As a library, the CB pattern can be found on Hystrix (Java), Polly (.Net), Resilient4j (Java), Akka (Scala), Rubyist (Go), pybreaker (Python), JRugged, and Javaslang. Apart from those

Api Gateway the API gateway locate between the client apps and the internal microservices. It is working as a reverse proxy and routing requests from clients to backend services. It is also provide cross-cutting concerns like authentication, SSL termination, and cache.

Api Gateway Handling application-specific authorization in the gateway _Restricting requests to specific endpoints become in this way possible. The drawback of implementing authorization in the API gateway is that it can be only role-based access to endpoints.

Scalability is simply measured by the number of requests an application can handle successfully. It can be measured by the number of requests and it can effectively support simultaneously. Once the application can no longer handle any more simultaneous requests, it has reached its scalability limit

Horizontal scaling and Vertical scaling both involve adding resources to your computing infrastructure, you must decide which is right for your application. Scaling horizontally and scaling vertically are similar in that they both involve adding computing resources to your infrastructure.

Horizontal VS Vertical Horizontal scaling means scaling by adding more machines to your pool of resources (also described as “scaling out”), whereas Vertical scaling refers to scaling by adding more power (e.g. CPU, RAM) to an existing machine (also described as “scaling up”).

Vertical scaling keeps your existing infrastructure but adding more computing power.

if you have get millions of request, in that case having one server won’t be enough, because even the hardware has maximum capacity limitations. In this case we need to do horizontal scaling or scaling out.

Scaling horizontally gives you scalability but also reliability because you will have more redundancy and mostly its the preferred way to scale in distributed architectures.

horizontal scaling When stateless then its easy.just need a load balancer when statefull then its hard

Kafka is a publish/subscribe messaging platform with built-in support for replication, partitioning, fault tolerance, and better throughput. It's an excellent choice for applications that need large scale data processing. incorporates : *fault-tolerant storage *stream processing capabilities

Features: 1)It can publish and subscribe streams of data. 2)It's capable of handling a vast number of read/write operations per second. 3)It can persist data for a particular period. 4)It has the ability to grow elastically with zero downtime. 5)It offers support for replication, partitioning, and fault-tolerance.

*Scalability:separates topics into partitions.because Kafka is distributed,you can scale up by adding new nodes to the cluster *High throughput:it is capable of handling massive volumes of incoming messages at a high velocity per second *low latency: Latency refers to the amount of time required to process each message

*High performance:provides high throughput with low latency and high availability *Highly reliable:in kafka the messages are durable even after they have been consumed.it can load balance consumers in the event of a failure *Durability:stores the messages on the disk,as opposed to in memory

use cases for Kafka: *Application activity tracking: You can leverage Kafka to publish all events (user login, user registration,etc.) that occur in your application to a dedicated Kafka topic.Then consumers subscribe to the topics and process the data for monitoring,analysis, etc.

use cases for Kafka: *Log aggregation:You can publish logs to Kafka topics and then aggregate and process them when needed. Kafka can collect logs from various services and make them available to the consumers in a standard format (JSON). *Real-time data processing: applications need real-time data processing.

use cases for Kafka: *Operational metrics: Kafka can aggregate the statistical data collected from several distributed applications and then produce centralized feeds of operational data.

Kafka Components Architecture: 1)Kafka Topic:A Topic is a category/feed name to which records are stored and published.Each topic has a name that is unique across the entire Kafka cluster. 2)Kafka Cluster:A Kafka cluster is a system that consists of several Brokers, Topics, and Partitions for both.

*Kafka Producer:A Kafka producer serves as a data source for one or more Kafka topics and is responsible for writing, optimizing, and publishing messages to those topics. A Kafka producer can connect to a Kafka cluster through Zookeeper. Alternatively, it can connect to a Kafka broker directly.

Kafka Producer Serialize data & Compresses data & load balances data & among kafka brokers throught partitioning

*Kafka consumer: Kafka Consumer to subscribe to Kafka topics and receive messages from these topics. *Kafka ZooKeeper: A Kafka Zookeeper manages and coordinates the Kafka brokers in a cluster. It also notifies producers and consumers in a Kafka cluster of the existence of new brokers or the failure of brokers.

*Kafka Broker:A broker is an individual person who arranges transactions between a buyer and a seller for a commission when the deal is executed. A Kafka cluster has exactly one broker that acts as the Controller.(control events,monitor)