Data Engineer

Rendlen Solutions

多个城市，巴基斯坦

Apply Now

发布 Aug 22, 2024 643 查看 Report Job

PKR. 150,000 - 200,000/Month

The primary focus will be in collecting, storing, processing and analysing of huge sets of data with choosing optimal solution to use and creating data pipeline architecture in TugstenPlus platform as well as optimizing data flows. The Candidate will support our dev team on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. The ideal Candidate must have strong experience using a variety of data analysis methods, a variety of data tools, building and implementing data pipeline:

Work with business cases to identify datasets and design data pipeline architecture
Analyse data from sources to choose optimal structure and storage architecture
Assess the consistency and accuracy of new data sources
Monitor performance of data pipeline execution and fix any issues
Define technical requirements for third-party applications to pull data.

Core Accountabilities

Analyse business cases and identify data sources (internal/external) with solution to collect them
Create and maintain optimal data pipeline architecture
Assemble large, complex data sets that meet functional/non-functional business requirements
Design/Build an ETL pipeline
Keep data secure and control role-based access to collected data
Work with stakeholders to assist with data-related technical issues and needs

Skills Requirement

Experience building and optimizing data pipelines, architectures and data sets.
Knowledge of various ETL techniques and frameworks
Experience with integration of data from multiple data sources
Proficient understanding of distributed computing principles
Proficiency in using query languages such as SQL and experience working with relational/non-relational databases
Experience working with and creating data architectures, data models, data warehouses/data lakes
Minimum of 3 years’ experience in area of data engineering/analysis

Technical Skills

Experience with Databricks (Azure, AWS) and lakehouse approach
Experience with relational SQL and NoSQL databases, including Microsoft SQL Server, Mongo DB
Strong knowledge and experience using object-oriented languages (Python, etc.) and SQL language (Spark SQL) to manipulate and transform data