Oct 02, 2023
No image
Service categories
Service Lines
Big Data
DevOps
Domain focus
Commerce
Subcategories
Big Data
Data Analytics
Data Discovery
Data Mining
Data Quality
Challenge
Blackbird needed a system that could re-scrape all records within < 4 hours of time windows, while each scrape record takes no more than 10 seconds.
The solution should be scalable and adaptable to changing business needs.
Blackbird expects high-quality DevOps, SecOps, and DataOps services to be provided by Intellicoworks, with a focus on delivering bespoke solutions that meet Blackbird's specific requirements.
Blackbird needed a system that could re-scrape all records within < 4 hours of time windows, while each scrape record takes no more than 10 seconds.
The solution should be scalable and adaptable to changing business needs.
Blackbird expects high-quality DevOps, SecOps, and DataOps services to be provided by Intellicoworks, with a focus on delivering bespoke solutions that meet Blackbird's specific requirements.
Solution
Provided high-quality DevOps, SecOps, and DataOps services that focused on delivering bespoke solutions to meet Blackbird's specific requirements.
Implemented real-time monitoring for cron jobs that tracked the success and failure of requests per second. Developed a notification system that alerted the team when a cron job failed, enabling them to take immediate action and prevent any significant issues.
Created a system that allowed for re-scraping all records within < 4 hours of time windows. Optimized the scraping process to reduce the time it took to scrape each record to less than 10 seconds.
Utilized PHP Laravel, AWS Services, and SSH Shell technologies to develop a scalable and adaptable solution.
Provided high-quality DevOps, SecOps, and DataOps services that focused on delivering bespoke solutions to meet Blackbird's specific requirements.
Implemented real-time monitoring for cron jobs that tracked the success and failure of requests per second. Developed a notification system that alerted the team when a cron job failed, enabling them to take immediate action and prevent any significant issues.
Created a system that allowed for re-scraping all records within < 4 hours of time windows. Optimized the scraping process to reduce the time it took to scrape each record to less than 10 seconds.
Utilized PHP Laravel, AWS Services, and SSH Shell technologies to develop a scalable and adaptable solution.
Results
We worked closely with Blackbird and we were successful in delivering a real-time analytics platform for hardware connectors.
Hardware connectors can process more than a million records per day while keeping infrastructure costs to less than $1K per month.
The project has been running smoothly for over 5 years, with real-time monitoring and efficient re-scraping capabilities.
Intellicoworks' bespoke approach and use of cutting-edge technologies allowed for seamless integration with Blackbird's existing systems.
The successful implementation of the real-time monitoring and re-scraping system has enabled Blackbird to deliver accurate and timely data to its customers.
We worked closely with Blackbird and we were successful in delivering a real-time analytics platform for hardware connectors.
Hardware connectors can process more than a million records per day while keeping infrastructure costs to less than $1K per month.
The project has been running smoothly for over 5 years, with real-time monitoring and efficient re-scraping capabilities.
Intellicoworks' bespoke approach and use of cutting-edge technologies allowed for seamless integration with Blackbird's existing systems.
The successful implementation of the real-time monitoring and re-scraping system has enabled Blackbird to deliver accurate and timely data to its customers.
No image
