Business Requirement
Application performance data was required to initiate corrective action. The ideal solution would capture and analyze data such as:
- Number of daily searches for a particular vehicle by single or multiple users, invalid searches, and searches for a particular time frame
- Number of failed responses and reasons
- Number of instances where response time exceeded the norm
- API response time details
- Number of successful/unsuccessful logins
The requirement also included analysis of user experience based on backend API success rate. All of this had to be achieved with minimum latency and performance impact.
QBurst Solution
The project involved development of a solution on Elastic Stack to analyze application usage data in real time. The solution analyzes logs and converts data into intuitive visualizations, enabling senior management to comprehend trends with ease. The solution facilitates real-time search and monitoring, visual cues on custom alerts, and graphical reports on pre-defined queries.
We developed an Angular dashboard that displays specific requirements outlined by the client. The log ingress was handled using Filebeat, a lightweight process residing within the vehicle dealership application. Once the data is in motion, it is sent to Logstash for processing.
Information from plaintext logs is extracted in Logstash, where all the fields necessary for fulfilling the requirements are parsed and tagged. Additional processing needed to mask sensitive information (such as userID) and derive additional parameters (for example, number of times a particular date range is searched) from existing ones is also done here. After processing is complete, the data is sent to Elasticsearch where it is indexed for reference by Kibana. We provided security for the entire dashboard using X-Pack security features. Visualizations in Kibana used for the ELK implementation included bar chart, pie chart, numeric metrics, tables, and time series.
The solution captures insights from almost any type of structured/unstructured data source and serves as an end-to-end solution for analytics, logging, search, and visualization.
