Overview
Our client is one of the largest auto parts distribution and marketing organizations in the world, marketing the Auto Parts sellers. With store and service center locations throughout North America and Europe, the client is a source for quality parts and service for over 4,600 parts stores and 2,500 certified service centers. Client engaged KTree for Elasticsearch Consulting to make their ominous data searchable, analyzable and presented in better way.
Requirements
- Design an ElasticSearch Cluster which can hold billions of records/documents with out fail and data search would be faster.
- Design dashboards to view net sale status Daily, Weekly, Monthly, Yearly.
- Extract information from Millions of XML files and CSV files and index to ElasticSearch.
- Needed to Implement elasticsearch python Client to get snapshots automatically from cluster periodically.
- Transform the data in to full form and Index data to elasticsearch.
Centralize logs from all application/web-servers to ElasticSearch Index.
Challenges
- Extract Historical Sales Transactions data From Relational Database Management System
- Integrate ElasticSearch to a third party Applications to filter data from Applications to Cluster directly.
- Implement automatic cluster restart scripts for ElasticSearch.
- Implement dashboards to visualize the data in a quick way.
- Automatic Snapshot from Cluster with fail over mechanism at specified time & removing snapshots by keeping one week backups in repository.
- Remove duplicates in elasticsearch while indexing data.
Results
Solution-1
Fast and Accurate Search:
A powerful search capability is furnished that instead of searching the text directly, searches an index instead. As we have had millions of transactions to take into consideration, the search function is catered to browse large number of product descriptions for the best match for a specific phrase – suitably the optimal item matched to the query is depicted as result.
Dashboards For Data Visualization:
scheme-free approach of elastic search, allowed us to calculate the aggregations within seconds of time, which has been quite a considerable improvement over writing queries on other business intelligence tools. With Kibana, as Data Visualization engine , real-time summary and charting of streaming information allowed us to and slice and dice your data as we see fit.
Cluster Creation and Management:
Elasticsearch is distributed by nature: it is perfectly adept with managing multiple nodes so as to provide high availability, scalability and agility. We set up a multi-node production cluster big enough to search all the data that has been collected and to ensure that data is safe from hardware failure.Maintenance of clusters and shards is a routine activity we do as as to have supreme stability.
The entire creation and management of clusters was carried out after thorough research and analysis and brimming on the facts that ES is equipped with ability to hold millions and trillions of data, which can be used to provide stellar customer experience.
Solution-2
- Processed millions of records from historical data to database & from there using ElasticSearch river concept we have indexed data to Elastic-search with a blazing speed. Later data is available for search & visualizations
- Chosen Good hardware Configurations in order to made an effective cluster to hold such kind of huge data & make them available for search. This can help us to achieve fail over and high throughput from Elastic Cluster. Designed cluster configurations by Considering data growth in future.
- Log Assessment and Management:
Integration with Logstash facilitated in processing logs and other event data from a variety of stream systems. Logstash helped not just in parsing and normalizing this data and converge on a common format, but also assisted in inserting it into our analytics datastore. Efficient management and tracking of daily logs. - Third-party Application Framework:
As elasticsearch offers wide range of API for other frameworks to integrate with, we ensured it manages the data repository in the most effective sense possible. - Eliminate Data Redundancy:
Making the most of primary key in elasticsearch, data duplication is avoided during indexing.
Moreover to circumvent redundancy, only one shard is used from each shard group (primary and replicas) - Implemented elasticsearch python Client to get snapshots automatically from cluster periodically.
Solution-3
- Implemented log-stash shippers for each application server logs from that logs are pushing to elasticsearch so that we can analyze the logs in a quick way.
- Like traditional databases we can define a primary key for elasticsearch also but we don’t have any combination key for elasticsearch. By choosing this Primary key correctly we can remove duplicates from data and it will be helpful to update data at any time in efficient way.
- Processed Historical data of XML files using Java Clients and push to Database from there we are indexing to elasticsearch so parts search is available for Client by hitting elasticsearch rather that going to each XML file. The search would be perform from millions of transactions even though response will get in milliseconds of time.
- ElasticSearch have vast API’s for all opensource frameworks which can integrate with third party Applications (like ERP, Java, PHP, Python, Perl etc..)
- Implemented Kibana Dashboards by integrating with elasticsearch to view sales, parts data quickly and perform aggregations with in minutes of time. Designed monthly, weekly, daily dashboard to view sales data quickly.
- Implemented automatic snapshots sweeping from repository if it is older than 7 days.
Gallery
Read more about KTree
KTree, a Global IT Company
What and how the processes are structured in KTree?
KTree's processes follow CMMi to ensure consistent, high-quality outcomes. With a focus on quality coding , code reviewing, and performance and load testing in our QA processes, we deliver reliable, scalable, and high-performing applications to meet our clients' needs.
Where is KTree's Web and Mobile application Development done?
KTree's web and mobile application development primarily takes place at our Hyderabad development center. Hyderabad is not only abundant in IT talent but is also known for its rich heritage and exotic food.
How KTree makes outsourcing project a positive experience?
KTree delivers high-quality applications that meet our clients' needs, by prioritizing collaboration, communication, visibility, traceability, and requirements engineering. While we charge a little premium for our services, we believe that it is important to provide the necessary resources and expertise to ensure project success.
What is Success mantra of KTree, as one of the respected web development company?
The success mantra of KTree as a respected web development company is combining Agile methodology with strong documentation, powerful web and mobile frameworks, and a solid KTree team to deliver great web and mobile applications within a specified timeframe.
Any other Differentiation which KTree has, which it can offer?
KTree's tremendous knowledge and expertise, gained from working on hundreds of web and mobile application projects, gives us a significant advantage that allows us to provide a head-start on projects, saving our clients many man-months.
What Development methodology KTree follows and why?
KTree's development team follows the Scrum framework of the Agile methodology, which improves Agility through its simplicity and flexibility. This allows us to adapt to changing project requirements and deliver high-quality products within the specified timeframe.
Leverage the expertise of a Top Application development company for your enterprise web and mobile application needs.
Contact our solid KTree team to get started.
