Tailor Med
A fully automated ETL pipeline that drives data-driven decisions
Context:
TM is a startup with a big, very complex solution with multiple outcomes, where the focus is on a financial aspect of treating. Source of the information for the company was data files generated by the hospital once a day. These files transfer from site to the data storage for further processing and operating. Processes of reading took more than a day, which blocked company build accurate to real time data KPI.
Business Challenge:
TM has multiple sites where they source their data. Each site has its own way to compose the data: file extension and structure. Before the project on adding each site to the system their developer had to add scripts to the system that might support the site, and deploy it. This process could take a week or weeks. In the long run, when TM is added by 10-20 sites per week - the process could become a bottleneck, which might block from scaling the business.
Technological challenge:
TM has some sites with files around 20-40Gb, before the project the company was reading the full file at once, which means, they have to open the file which puts the entire file to the RAM. In cloud calculations it might be an issue to order such powers. Also, the process could take more than 10hours.
Solution:
We used the AWS platform, and focused on the server less approach Lambda to reach the goals. The tool improved all the part of the system that we were expecting: time and power consumption, complexity to maintain it. Some files, with size over 20Gb were added to the system within around 10 hours, during the process TM had to rent expensive powers to have enough RAM memory to read the file. We built a system, totally configurable from Front-End, that delegated the process of onboarding new sites from the RnD team to the operations team. Result After the tool was ready AWS Lambda never takes more than 200Mb RAM, and the process over the biggest file takes not more than 45 minutes. Our solutions reduced company expenses by 10%, integration time with the new site reduced x3. Number of exceptions reduced to <1%.