Principal Software Engineer, Data Engineering

We are seeking a Principal Software Engineer, Data Engineering to serve as a technical leader and domain expert across multiple Data Engineering teams. This role is critical for driving strategic transformation and defining foundation technology, specifically focusing on high-scale data products. The successful candidate will operate with a Builder Profile, driving velocity by identifying and executing key opportunities across the company.

This role will build the data layer for a Unified Consumer Data Plane across Credit Karma and TurboTax. Building creating the unified data plane brings together unified identity and attributes, a canonical catalog of products and offers and behavioral and activity context so that AI/ML models, AI agents, personalization and marketing teams can build on one trusted, real-time data foundation. You will design for scale (from hundreds to thousands of attributes),across clouds (GCP and AWS) to control cost, faster time-to-data, and governance by design (data quality, lineage and fine-grained access control).

This is an individual contributor level role with no direct reports or managerial duties. You will apply master level software engineering and industry best practices to solve hard problems end to end, working strategically, addressing complex issues, and leading technology decisions that have significant long-term impact.

Responsibilities

Technical Craft and Architectural Leadership

• Lead the development and implementation of the Consumer Platform data strategy, aligning with the target state architecture. Drive significant technology initiatives end-to-end, including across the horizontal layers of the architecture.

• Provide a birds-eye architecture view and give early visibility into Technical Decisions, clarifying the rationale and principles for those decisions.

• Drive the design and implementation of durable, robust, scalable, and secure technical designs, implementing them effectively to ensure high availability and optimal performance of applications.

• Act as a role model in full cycle engineering, demonstrating deep expertise across the software lifecycle, including design, implementation, and operationalization.

• Engage in 70% hands-on development in all phases of the software life cycle.

• Lead the initial phases of critical projects, such as proof of concepts or pilot implementations, and engage in live coding sessions and collaborative problem-solving.

Enterprise Data Strategy, Quality, and Efficiency

• Drive the data strategy across the consumer platform, including defining the technical strategy for data modeling, ingestion, curation, and quality standards across consumption points and patterns.

• Drive the consumer platform analytics pipeline and establish the target state for real-time analytics.

• Define and enforce the technical and operational standards required for high velocity, establishing clear technical paved paths and standardizing technology direction.

• Reduce rework and overhead by providing directions on smart execution for migration-specific initiatives and addressing infrastructure overhead (e.g., EMR Upgrades and image management for EMR serverless).

• Ensure designs are scaled to handle future data volumes with consideration of infrastructure cost optimization.

• Define approaches for how to avoid incidents and position for 'game day' preparedness, including addressing the lack of Data Quality (DQ) monitoring for upstream pipelines and E2E testing and quality checks.

• Make data backed decisions and drive the right level of instrumentation and Experimentation.

AI Native Applications and Innovation

• Possess the knowledge required for building AI native applications.

• Guide the applicability of AI to customer problems through a deep understanding of the value and limitations of AI technologies.

• Champion new solutions, emerging technologies, and AI to solve complex problems and create new opportunities, including leveraging AI and Agentic experiences to improve customer experiences.

• Provide direction on managing the multiples of AI tooling silos to reduce inefficiency and inconsistency.

• Maintain a passion for continuous learning, experimenting, and applying cutting-edge technology and software paradigms.

Leadership, Mentorship, and Collaboration

• Influence technology and product direction, typically driving work across multiple scrum teams.

• Act as the bridge between product and data engineering, enabling teams to work closely with Product Development teams. Partner with other groups for cross-functional design, development, and integration.

• Provide clear business context, including a mind map of business flows, and clarify who are the consumers of the data and how the data is being consumed.

• Provide comprehensive technical mentorship through personalized skill enhancement with clear milestones.

• Conduct detailed architecture and code reviews, including interactive methods like paired programming, to enforce standards and share fundamental and advanced design principles.

• Accelerate learning and growth by actively keeping engineers updated on the latest industry trends, tools, technologies, and frameworks.

• Contribute to standards, patterns, and best practices that improve the engineering community.

Strategic and Leadership Skills

• Strong mentoring skills and ability to influence and communicate effectively with both technical and non-technical people.

• Excellent interpersonal and communication skills, with the ability to communicate clearly, succinctly, and persuasively to all levels.

• Ability to operate at varying levels of abstraction from business strategy to product strategy to high level technical design to detailed technical design to implementation.

• Demonstrated experience driving technical strategy in high scale data environments.

• Ability to synthesize achievable solutions from diverse inputs, alternative sources (build / buy / partner) and complex data.

• Strategic mindset: influences the team and platform’s short and long-term strategy for maximum business impact.

Qualifications

• BS or MS in Computer Science, Data Engineering, or a related field.
• 15+ years of software engineering experience, with a strong background in data engineering.

• 9+ years professional working experience with various big data technologies, including Hive, Hadoop, Kafka, Columnar Databases, and Graph Databases. Strong experience with stream processing or near real-time data ingestion is important.

• Expertise in developing Software for Java (Spring & Springboot), Scala for Spark streaming & Spark applications, or other JVM based languages.

• Expert professional experience with SQL, XML, JSON, YML, very strong Python and Linux.

• Strong understanding of extensible data modeling and designing for large scale data environments.

• Strong background using cloud platforms such as AWS, Azure, or GCP (AWS expertise specifically noted). Demonstrated experience operating high scale services in the cloud, designing highly resilient services, and building recovery mechanisms.

• Expertise with tools and frameworks such as Docker, Spark, Scala, Jupiter Notebook, Databricks Notebooks, Kubernetes, Feature Management Platforms, and SageMaker.

• Strong background in adoption of AI skills in both traditional and Generative-AI. Knowledge of AI / ML, Generative AI, and experience with growth-focused AI/ML applications.

• Expert knowledge of software development methodologies and practices, including Agile Development, SCRUM, and/or Extreme Programming methodologies.

…