We have requirement for Senior Data Engineer with experience in Microsoft Azure Cloud. In this role, you will be responsible for building real-time/batch data ingestion pipelines integrating different data sources like RDBMS, APIs, plant legacy systems, historian systems, storage systems etc.
- Design, develop, and launch extremely efficient and reliable data pipelines.
- Solve issues in the existing data pipelines and build their successors.
- Build modular pipeline to construct features and modelling tables.
- Maintain data warehouse architecture and relational databases.
- Monitor incidents by performing root cause analysis and implement the appropriate action.
- Create, document and monitor highly readable code.
- Masters/Bachelor’s degree in Computer Science or any other related field with minimum 7-10 years of IT experience.
- Minimum 4 years’ experience in designing, building and operationalizing medium to large scale data integration (structured & unstructured) projects with Azure Data Lake, Data Warehouse, BLOB Storage, RDBMS using Azure integration services.
- Minimum 1 project implementation of real-time data integration & processing.
- Minimum 4 years of Hands-on Experience in following big data technologies in the Microsoft Azure cloud is must: Data Lake Store, Data Warehouse, Data Catalog, Data bricks, Data Lake Analytics, Stream Analytics, Event Hub, IoT Hub, Data Factory, and Logic Apps is must.
- Hands-on experience in CI-CD implementation with Azure DevOps.
- Hands-on experience in Programming languages Python & .Net is must.
- Hands-on experience in SQL/T-SQL and query optimization with SQL DB and DW.
- Minimum 2 years of designing and developing reports and Dashboards using Power BI/Tableau/Qlik/SSRS
- Added advantage:
- Data engineering with Spark framework.
- Real-time data ingestion using Spark streaming
- Data integration with other cloud services like AWS SQS, AWS S3, AWS Glue and Google cloud platform etc.
- Prior knowledge in relational SQL and NoSQL databases such as Neo4j and MongoDB, Python, UNIX or statistical software.