In the evolving landscape of technology, the integration of real-time data pipelines and edge computing is reshaping how brick-and-mortar stores operate. Engaging actively with this gradual evolution is Suhas Hanumanthaiah, a data architect who has actively worked to impact the operational efficiency of Grocery Outlet Inc. During his tenure at Grocery Outlet, Hanumanthaiah played an important role in managing the real-time pipeline of sales data, transitioning from stores to Cloud Data warehouse - AWS Redshift. He also contributed to the digital transformation project that migrated AWS Redshift data to Google BigQuery Lakehouse. This shift optimized read queries to reduce cost by 1 TB. Further, a significant aspect of Hanumanthaiah's work and digital transformation involved integrating SAP systems into the new data architecture. He meticulously mapped data from SAP HANA to over 300 data warehouse tables, ensuring that the transition from legacy systems like AS400 to SAP-based systems did not disrupt the flow of critical business information. One of the notable achievements in this process was the development of an ELT (Extract, Load, Transform) framework tailored for the data warehouse migration. This framework facilitated the efficient handling of Change Data Capture (CDC) from SAP, a method essential for real-time data replication. By optimizing the CDC process, Hanumanthaiah achieved a substantial cost saving of $10,000 in Google BigQuery billing. This was accomplished by redesigning how CDC data was processed, reducing unnecessary data scans, and improving query performance. In addition to cost savings, Hanumanthaiah addressed the challenges posed by the highly normalized SAP HANA data model. Complex process calculation queries were prone to timeouts in the middleware layer, affecting the performance of critical applications. Through persistent optimization efforts, he enhanced query performance. Hanumanthaiah's expertise also extended to eCommerce data integration. He modeled data from SAP to meet the requirements of eCommerce vendors such as Doordash, Instacart, Uber Eats, and the Grocery Outlet App. This integration was vital for expanding the company's digital footprint and ensuring that online platforms had access to accurate and timely data. Speaking of integration, he also developed a repeatable ELT framework for the CDAP pipeline and accelerated the development of pipelines to support 600+ tables in less than 8 weeks. Furthermore, he was instrumental in building high-impact reports, such as Sales Margin analyses, leveraging the new source systems, and developing a framework to track if SAP SLT, an ELT from SAP fails to replicate data into Google Cloud. This included defining standard operating procedures (SOP) and disaster recovery. He also developed an eCommerce interface at a fast pace with key data quality checks and supported business continuity. Beyond technical implementations, Hanumanthaiah collaborated with technical vendors to optimize the usage of cloud resources. He fine-tuned MicroStrategy VLDB (Very Large Database) settings for AWS Redshift, achieving a 15% improvement in query execution times. These enhancements contributed to more responsive analytics and reporting capabilities. Reflecting on the challenges faced during these projects, Hanumanthaiah emphasizes the importance of building connectors capable of managing CDC when replicating data from SAP HANA to cloud-based data warehouses or lakehouses. He suggests that, in the absence of such connectors, employing an intermediary transactional cloud database can be effective. This approach allows real-time applications to rely on the cloud transactional system, while the lakehouse serves as a repository for aggregated business reporting. Hanumanthaiah also notes that constructing real-time pipelines can become complex when dealing with high-velocity data. He advocates for thorough testing of connectors and interface technologies under simulated large loads to ensure reliability. Additionally, he points out that replication technologies often lack native error tracing, highlighting the need for strong monitoring and testing. In summary, Suhas Hanumanthaiah's contributions to integrating real-time data pipelines and edge computing in retail have led to operational improvements at Grocery Outlet Inc. His work exemplifies how the data architecture is undergoing a modern makeover, enhancing customer experiences. This story was distributed as a release by Kashvi Pandey under HackerNoon’s Business Blogging Program. This story was distributed as a release by Kashvi Pandey under HackerNoon’s Business Blogging Program.
