How Amount Modernized Its Data Environment to Reduce Customer Onboarding Time

Client Overview

Amount, a leading-edge provider of software based financial solutions located in Chicago Illinois has developed and launched an application platform that seamlessly enables banks and financial institutions to traverse the digital divide, unify data, and gain insight to improve the consumer experience. Built on the AWS cloud, this team of software developers makes digital financial experiences that align with the way consumers live, empowering banks and financial institutions with a cloud-based lending solution for personal loans, credit card payments, deposits, and point-of-sale lending. Their suite of turnkey solutions and customizable interfaces enable established brands to accelerate digital transformation in a fraction of the time it would take banks and financial institutions to build similar in-house technology solutions.

The Challenge

With rapid success and growth brought the difficulties of larger volumes of data, along with the increasing complexity of how data could be utilized across the organization. Furthermore, the tools selected early in the startup cycle became inadequate to deal with, especially with the higher demands and challenges that came with an increasing data platform environment. Amount knew they needed to find a better set of tools and processes to meet the immediate needs of the data team who were looking for a sizable reduction in overall management over-head.

Additionally, Amount was using a third party for the big data processing, but their existing hosted solution could not handle the anticipated scale, and the lack of reliable support increased the cost of ownership and unpredictable uptime and runtime of the solution, leading to missed SLA’s. Amount was also using Luigi, a well-known tool built by Spotify for their data pipelines, but the increasing usage and the effort to manage the tool internally was putting unnecessary demands on the data team. A more modern approach that would include a migration path from their diverse set of data sources in AWS such as Aurora and Postgres RDS into their S3 based data lakes, would form the foundation of the requirements on the data side. In addition, reduced management overhead and infrastructure reusability were driving demands from the DevOps side of the business. The ideal solution would be deployable and rely on their existing Elastic Kubernetes Service (EKS) to provide the necessary flexibility, while addressing overall security and regulatory requirements, and data isolation across the organization.

The Solution

As they kicked off their investigation, the client leveraged long-time partner Uturn Data Solutions to evaluate a number of different industry tools and to build their future data platform roadmap. With Uturn’s assistance, Amount identified Apache Airflow, a well-known industry standard for data orchestration, as the primary tool for their data pipeline strategy. Astronomer was quickly identified as an option that would meet Amounts requirement for a managed Airflow solution. Additionally, AWS EMR was identified as the best alternative for the third-party big data hosted solution. The AWS EMR solution would provide better predictability for cost, uptime, and runtime, allowing Amount to meet their planned SLA’s within the budget, and on time.

With the previously mentioned challenges of regulation, security, and data isolation, along with the migration requirements for the current data pipelines, Uturn was contracted to facilitate the implementation, including an integration of Astronomer’s Airflow service, and an AWS EMR solution for a production deployment. The partnership with Uturn would allow the client to be tactically hands-off, while ensuring that the implementation requirements and strategy were addressed properly. Uturn first validated that all previous identified requirements were correct and finalized before the production solution was developed. Secondly, Uturn reviewed security constraints and network limitations to make sure the solution complied with Amount’s data access restrictions. In order to reduce the deployment time, a stand-alone EKS cluster was created to house the Astronomer and Airflow. All of the workloads running in third-party big data solution were then migrated to run in EMR. This allowed Uturn to catalog any changes that might be required for the future production deployment, and the Terraform design would provide a goforward template to on-board new tenants to the solution.

With the previously mentioned challenges of regulation, security, and data isolation, along with the migration requirements for the current data pipelines, Uturn was contracted to facilitate the implementation, including an integration of Astronomer’s Airflow service, and an AWS EMR solution for a production deployment. The partnership with Uturn would allow the client to be tactically hands-off, while ensuring that the implementation requirements and strategy were addressed properly.

Uturn first validated that all previous identified requirements were correct and finalized before the production solution was developed. Secondly, Uturn reviewed security constraints and network limitations to make sure the solution complied with Amount’s data access restrictions. In order to reduce the deployment time, a stand-alone EKS cluster was created to house the Astronomer and Airflow. All of the workloads running in third-party big data solution were then migrated to run in EMR. This allowed Uturn to catalog any changes that might be required for the future production deployment, and the Terraform design would provide a goforward template to on-board new tenants to the solution.

Post the production deployment, Amount’s Data and DevOps teams immediately noticed a positive impact on the business through a combination of increased productivity, reduced onboarding time for new clients, and an overall reduced infrastructure management effort. In addition, there have been numerous client wins, including a testing and approval process implemented by the data team, and the roll out of eight complete and separate Airflow installations completed by their Systems Reliability Engineering (SRE) team.