Data Systems
Serverless AWS pipelines (S3, Lambda, Athena, Redshift), distributed Spark and Hadoop ETL across 60+ Linux nodes, containerized RAG and vector DB on on-prem GPU servers, and Linux systems administration with Bash and Puppet automation.
Core Technology Stack
Tools and platforms used across production roles and lab projects.
Cloud & Storage
Big Data
Linux & Systems
Automation & DB
Serverless AWS Data Pipelines
Architected serverless pipelines on AWS (S3, Lambda, Athena, Redshift), provisioned as code to ingest org-wide digital assets at scale: websites, PDFs, and documents.
Built an Adobe API integration into the pipeline to evaluate WCAG 2.1 compliance, generating accessibility metrics and reports for all digital assets automatically.
View AWS Data Systems RepositoryPipeline Architecture
Provisioned as Code
All AWS resources (S3, Lambda, Athena, Redshift) were provisioned as code, making the pipeline reproducible and version-controlled rather than manually configured.
Automated WCAG Compliance
Built an Adobe API integration into the pipeline to evaluate WCAG 2.1 compliance on every digital asset, generating accessibility metrics and reports automatically.
Org-Wide Asset Ingestion
The pipeline ingests websites, PDFs, and documents at org scale using S3 event triggers into Lambda, with Athena for querying and Redshift for structured analytics.
On-Prem Cluster · 60+ Linux Nodes
… and 45+ more DataNodes
Distributed Hadoop & Spark Cluster
Set up and configured an on-prem cluster across 60+ Linux nodes, enabling HDFS-based storage and Spark jobs for ETL workflows, joins, and aggregations.
Also designed distributed data pipelines on AWS using S3, Glue, Athena, Redshift, Spark, Lambda, and CloudFormation. Benchmarked cost and latency trade-offs across storage types and compute models.
View Scalable Data Systems RepositoryPySpark on Delta Lake
Engineered PySpark feature pipelines on Delta Lake with temporal aggregated features using economic (Inflation, Interest Rates), climatic (Weather), and geopolitical indicators.
Cost and Latency Benchmarking
Benchmarked cost and latency trade-offs across storage types and compute models: S3, Athena, Redshift, and Spark, to inform architectural decisions for the lab pipeline.
Hadoop/Spark Cluster Setup
Set up and configured the on-prem cluster across 60+ Linux nodes, enabling HDFS-based storage and Spark jobs for ETL workflows, joins, and aggregations.
AWS Distributed Pipelines
Designed distributed data pipelines on AWS using S3, Glue, Athena, Redshift, Spark, Lambda, and CloudFormation for the Scalable Data Systems Lab project.
On-Prem Linux Infra & Vector DB
Containerized and deployed the RAG service and vector DB on on-prem Linux GPU servers using Docker and Kubernetes, cutting cloud API spend while scaling to demand.
Built weekly ETL workflows and pipelines for doc parsing: schema mapping, metadata enrichment, and indexing into a NoSQL vector database to support RAG. This made siloed data across Windchill, Confluence, and ServiceNow accessible via a unified RAG layer for Agentic AI.
On-Prem Deployment Stack
Cross-System Unified Data
Developed microservices in Java, TypeScript, and Python to make siloed data across Windchill, Confluence, and ServiceNow accessible via a unified RAG layer for Agentic AI.
Spring Boot RPC
Built a Spring Boot microservice that used RPC to invoke methods on a legacy Java server monolith, safeguarding from failure propagation while making siloed data available downstream.
Performance and Root-Cause Analysis
Cut chat latency from 90 to 20 sec by profiling compute, memory, and I/O bottlenecks on Linux GPU hosts, tuning threading and batching for 4x RAG throughput.
Linux Systems Administration
Provisioned and maintained Linux systems across Jetson, Raspberry Pi, and on-prem GPU servers. Wrote Bash automation for deploys and diagnostics, and profiled performance issues down to the kernel level.
Performance and Root-Cause Analysis
Cut chat latency from 90 to 20 sec by profiling compute, memory, and I/O bottlenecks on Linux GPU hosts using Linux perf tools, NVIDIA NSight, and TensorRT Profiler, debugging issues from the application layer down to kernel and driver level. Tuned threading and batching for 4x RAG throughput.
Linux Admin
Provisioned Ubuntu Linux across Jetson and Raspberry Pi: kernel and driver config, systemd services, package and user management, SSH and network setup, with Bash automation for deploys and diagnostics.
Automation Tooling
Used Puppet for configuration management and Git for version control. Wrote Python and Bash tooling for operations tasks. Managed cron and systemd services for recurring job scheduling.
Networking and Security
Configured TCP/IP, DNS, DHCP, SSH, and firewall settings as part of Linux system setup. Applied OS hardening and managed user and group permissions across systems.