As a Data Engineer at Inito, you will play a pivotal role in designing and implementing robust data pipelines that serve as the backbone for our data-driven decision-making process. Working closely with Business Analysts, Data Scientists, and Machine Learning Engineers, you will ensure the integration, transformation, and delivery of data across our platforms, creating a single source of truth that supports complex analytics and machine learning models. We are seeking a passionate and skilled individual who is adept at leveraging cutting-edge technology to solve complex data challenges and enhance our business capabilities.

Responsibilities

• Design, develop, and maintain robust data pipelines using Airflow, DBT, and Python within a BigQuery environment.
• Implement and maintain data integrations with various sources and targets using tools like Airbyte, Elementary, and Metaflow, with a focus on optimizing data storage and retrieval in BigQuery.
• Write advanced SQL queries and scripts for data manipulation and retrieval in BigQuery.
• Develop and optimize data models and schemas in Preset and other visualization tools, tailored for performance in BigQuery.
• Utilize Pyspark for big data processing tasks, ensuring compatibility with BigQuery's infrastructure.
• Work with APIs, Docker, and GCP services to enhance and automate data workflows, particularly focusing on integration with BigQuery.
• Apply best practices in continuous integration and continuous deployment (CI/CD) environments using testing and pipeline validation techniques.
• Employ asynchronous architectures, queues, caching strategies, and distributed computing to enhance system performance.
• Write custom code in Python and utilize Jinja within DBT for dynamic pipeline generation, specifically designed for efficiency in BigQuery.
• Adopt design patterns and engineering best practices to ensure code scalability and maintainability.
  
  

Key Outcomes

• Develop a Unified Data Foundation - Spearheaded the creation of a single source of truth that centralizes our data assets, ensuring consistency and reliability across the board.
• Expertise in Cutting-Edge Technology - Utilize the state-of-the-art DBT stack and automate reports, optimizing our data processing workflows for efficiency and accuracy.
• Drive Impact Across Teams - Your work will directly influence the success of our Business Analysts, Data Scientists, and Machine Learning Engineers by providing them with the crucial data they need to excel.
• Optimize Business Logic - Design and manage data tables that support and enhance business logic functions, delivering tailored data solutions that drive strategic decision-making.
  
  

This job was posted by Manikanta Reddy from Inito.