Azure Data Engineering

 Job description

Are you passionate about building scalable data solutions in the cloud? We are looking for an experienced Data Engineer skilled in (AWS/Azure) technologies to join our dynamic team. This role requires expertise in designing, developing, and optimizing data pipelines, models, and cloud-based data architectures.

Your Future Employer: A leading global analytics and digital solutions company empowering businesses with cutting-edge cloud and data engineering capabilities.

Responsibilities:

1. Designing and implementing scalable data pipelines using (AWS/Azure) services.

2. Developing and optimizing ETL workflows using (AWS Glue/Azure Data Factory).

3. Building and maintaining data models using SQL, (Redshift/Azure Analysis Services).

4. Automating CI/CD pipelines leveraging (AWS DevOps/Azure DevOps).

5. Ensuring data governance, security, and compliance best practices.

6. Collaborating with business and technology teams to define data requirements.

Requirements:

1. Bachelor's degree in Computer Science, IT, or a related field.

2. 5+ years of experience in data engineering with expertise in (AWS/Azure) platforms.

3. Strong experience with SQL, Python, Spark, and cloud-based data technologies.

4. Hands-on experience with (AWS Glue, Redshift/Azure Data Factory, SSAS).

5. Knowledge of CI/CD pipeline automation using (AWS DevOps/Azure DevOps).

Whats in it for you?

1. Opportunity to work with cutting-edge cloud and data technologies.

2. A collaborative and innovative work environment.

3. Career growth and professional development opportunities.

Reach Us: If you think this role is aligned with your career, kindly write an email along with your updated CV to shreya.mohan@crescendogroup.in for a confidential discussion on the role.

As an Azure Data Engineer, the candidate is expected to be specializing in designing, implementing, and managing large-scale data solutions on the Microsoft Azure cloud platform. They possess expertise in various aspects of data engineering like data storage, data integrations, analytics etc using Azure data services. Ideal candidate will have a strong background in data engineering, Azure cloud services and data processing technologies.

# Requirements:

1. 5+ years of experience in data engineering, preferably on Microsoft Azure. This will increase with senior positions as other DE roles we have.

2. Strong knowledge of Azure cloud services, including Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Storage.

3. Experience with data processing technologies, such as Apache Spark, Apache Hadoop, and SQL.

4. Strong understanding of data modeling, data warehousing, and data governance.

5. Experience with data security, compliance, and regulatory requirements.

6. Strong programming skills in languages such as Python, Scala, or Java.

7. Experience with agile development methodologies and version control systems such as Git

8. Azure certifications, such as Azure Data Engineer Associate or Azure Solutions Architect expert.

9. Experience with containerization technologies such as Docker.

10. Knowledge of data visualization tools, such as Power BI, Tableau etc

11. Experience in machine learning and AI technologies will be added advantage.

Soft Skills:

Problem-solving: Solve complex data problems effectively.

Communication: Communicate clearly with both technical and non-technical team members.

Attention to detail: Pay close attention to detail to ensure accuracy.

Adaptability and learning: Stay up to date with new technologies and trends.

Teamwork: Collaborate well with others for project success.

Leadership and mentorship: Take on leadership roles and mentor junior team members for growth.

Create and maintain data pipelines that efficiently move data from various sources to storage and processing systems

Extract, transform, and load (ETL) data using tools like Azure Data Factory

Work with Azure SQL Database, Azure Data Lake Storage, Azure Cosmos DB, and Azure Blob Storage to design scalable and performant data storage solutions

Use technologies like Azure Databricks and Apache Spark to process and analyze large volumes of data

Implement data governance procedures and ensure compliance with security policies

Skills:

Proficiency in SQL and other data query languages

Experience with Azure data storage solutions

Knowledge of data integration tools

Familiarity with data processing frameworks

Understanding of data modeling and schema design principles

Ability to work with large datasets and perform data analysis

Strong problem-solving and troubleshooting skills

Etl, Azure Cosmo Db, Spark, Python, Sql, Azure, Data Engeneering

We are seeking a skilled Azure Data Engineer to join our team and play a key role in designing, implementing, and managing our data solutions on the Azure cloud platform. The ideal candidate will have hands-on experience with Azure data services and a strong background in data engineering and analytics.

Responsibilities:

Design, develop, and deploy scalable and efficient data pipelines on the Azure cloud platform using services such as Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.

Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and translate them into technical solutions.

Build and maintain data warehouses, data lakes, and other data storage solutions on Azure.

Implement data governance and security best practices to ensure data quality, privacy, and compliance.

Optimize data processing and storage for performance, cost, and scalability.

Monitor, troubleshoot, and optimize data pipelines and systems for reliability and efficiency.

Stay up-to-date with the latest Azure data services and technologies and recommend best practices and optimizations.

Requirements:

Bachelor's degree in Computer Science, Engineering, or related field.

A bachelors degree in Computer Science or related field with 6-10 years of technology experience

Strong experience in System Integration, Application Development or Data-Warehouse projects, across technologies used in the enterprise space

Software development experience using:Object-oriented languages (e.g. Python, PySpark,) and frameworks

Database programming using any flavours of SQL

Expertise in relational and dimensional modelling, including big data technologies

Exposure across all the SDLC process, including testing and deployment

Expertise in Microsoft Azure is mandatory including components like Azure Data Factory, Azure Data Lake Storage, Azure SQL, Azure DataBricks, HD Insights, ML Service etc.

Good knowledge of Python and Spark are required

Good understanding of how to enable analytics using cloud technology and ML Ops

Experience in Azure Infrastructure and Azure Dev Ops will be a strong plus

Proven track record in keeping existing technical skills and developing new ones, so that you can make strong contributions to deep architecture discussions around systems and applications in the cloud (Azure)

Characteristics of a forward thinker and self-starter

Ability to work with a global team of consulting professionals across multiple projects

Knack for helping an organization to understand application architectures and integration approaches, to architect advanced cloud-based solutions, and to help launch the build-out of those systems