Table of Contents
A river of data
The river of data flows continuously. In all processes, we have a large amount of data that arrives constantly, growing, and in real time. This data can flood decisions if we are not prepared for the flood.
Analyze Data: The client senses they can obtain value from the data to respond to their business needs, but how can they achieve it? The crux of the matter is to use this data in real time to know what decisions to make when we analyze it.
Social networks are a clear example of this constant and growing flow of data. Your brand may be suffering a reputational crisis, and to detect it in real time and provide an immediate solution, it is necessary to know the opinion of the users. For this, it is essential to analyze the comments that arrive, for example, through many tweets.
Clear the data: the water is dirty
The river of data could be clearer. It is of great importance to know the state of the data. Sometimes the data is “dirty.” It is necessary to clean the data of all information outside my goal.
Sometimes, when the water runs dirty, not only do you have to clean up the data that is arriving, but you also have to complete it with additional related information. The objective is to enrich the data that comes to us in real-time to be able to work with it.
To carry out this cleaning and the subsequent enrichment of the data, the data must be filtered as it arrives. We use Apache Storm, a real-time processing framework tool, to do this. This process can also be done through scalable tools like Spark or Flink. The framework is used to clean, filter, and pipeline our river of data.
Organize the data: make me a lake to swim in.
The river flows clean now, but it cascades down. We need to build a lake to swim in, a place where we can analyze and work with the valuable data that arrives to draw conclusions and make decisions based on them: a panel with graphs, some statistics…
To create an interface with which to start working in Business Intelligence, we can use Elasticsearch. Elasticsearch is a NO-SQL database with a high statistical capacity to offer actionable data. It is open source and allows all kinds of data aggregations to make statistics.
Finding non-sql database experts is difficult, but the benefit is huge. Elasticsearch is a document database that supports coordinates and, more importantly, scales horizontally very well.
We can also combine Elasticsearch with Kibana to obtain graphs and perform all kinds of analysis, taking full advantage of Elasticsearch’s statistical capabilities since they are perfectly integrated.
Manage data: water overflow.
At this point, we can already offer valuable information to the client. But when the data flow grows suddenly, what do we do if the river is very swollen? When the system gets small unpredictably, we need a dam. With it, we can contain the data and manage the information more orderly.
In programming, we can manage the data using “message queues.” Thanks to Kafka, an Apache project, we can better distribute information. It is a trusted place that distributes information logs and also allows replicas.
Kafka supports data ingestion well and is also supported if you need it. We can use it to dampen the pressure of the water so that it does not overflow suddenly.
Organize the data: water leaks.
When system architecture issues arise, or business needs change, the water can begin to leak elsewhere. The idea is to use the technology proposed by Docker to adapt or transfer our system to the new source or origin of the data, correct it, or modify it simply.
More and more people are already using these types of containers in production to make and deliver software. Docker makes it easy to build, distribute, and scale software. If inside the container, everything is connected and works correctly; the environment is indifferent, be it Ubuntu, RedHat, or any other.
With a single command, we can build a whole system with a lot of components in a relatively simple way. Using Docker allows us to start the procedure again from new data.
system visualization
All of this flow can originate from any top provider or data grabber. Seeing the project in perspective, and taking a good satellite photo of the information, allows us to follow its course and redirect it according to the client’s needs.
How good our system is will depend on cleaning, channeling, and analyzing the information and how we present it to the users.
Also Read : Natural Language Processing, What Is It?
