Data Engineer

Data Engineer

PGP Glass Pvt. Ltd. | Vadodara, GJ, IN

Posted a month ago

Apply Now


Purpose of the Job

The Data Engineer extracts data and implements machine learning models into production by utilizing state of the art tools/algorithms and methodologies following DevOps and a test driven development process. Data Engineering is the key role for bridging the gap between insight and actions. Data Engineers work in close collaborations with the data scientists and guide them to focus not only on model performance but also delivery stability, reproducibility and scalability of a software product.

The Data Engineer believes in a non hierarchical culture of collaboration, transparency, safety, and trust. Working with a focus on value creation, growth and serving customers with full ownership and accountability. Delivering exceptional customer and business results

Essential Qualifications:
Bachelors or masters in Computer Science or Engineering, BCA / MCA / B.Sc / M.Sc

Essential Experience:
Collaborate with Data Scientists to test and scale new algorithms through pilots and later industrialize the solutions at scale to the comprehensive fashion network of the Group
Influence, build and maintain the large scale data infrastructure required for the AI projects, and integrate with external IT infrastructure/service to provide an e2e solution
Leverage an understanding of software architecture and software design patterns to write scalable, maintainable, well designed and future proof code
Design, develop and maintain the framework for analytical pipeline
Develop common components to address pain points in machine learning project, like model lifecycle management, feature store and data quality evaluation
Provide input and help implement framework and tools to improve data quality
Work in cross functional agile teams of highly skilled software/machine learning engineers, data scientists, designers, product managers and others to build the AI ecosystem within the Group
Deliver on time, demonstrating strong commitment to deliver on the team mission and agreed backlog

Skills & Competencies required
Ability to transform proof of concept machine learning models into scalable solutions
Well versed with data handling techniques Data cleansing and feature scaling
Ability to apply software development best practice into machine learning project, including unit test, DevOps integration, release management, test driven development, etc.
Ability to automate development process of machine learning project by leverage state of art tools and technology like container, continuous integration and delivery, orchestration tools etc.
Familiar with at least one of major cloud solutions, preferably Azure. Able to recommend and select right services available in cloud to address technical problem
Good understanding and experience with software design principle and design pattern
Good understanding of Agile and scrum methodology, keep team focus on deliver business value

Work experience:
experience:~2+ years of industry exposure in Data Engineering projects

Required to travel to sites as per business needs

Any (prefer Manufacturing, Logistics); willingness to learn manufacturing systems (OT systems and data stores)
Functional Knowledge
Azure Data Factory & Data Components like
such as Azure Data
Lake, Azure SQL Database, Azure SQL Warehouse, SYNAPSE
Azure Analytics services like Databricks (ADB)/
Azure Data Lake(ADL).
DevOps and continuous integration
Data pipeline health monitoring
Software development best practices
Python/R Programming
Distributed computing
Spark and Hadoop eco system
Visualization using Power BI or any other
Data structure and algorithm
RDBMS and NoSQL databases, good skill with SQL