Data Engineer - Python

Data Engineer – Python

Asad Hameed

•

We are looking for an experienced and skilled Data Engineer – Python to join our team at Devigital Systems. This hybrid position is ideal for someone with substantial experience in developing and managing data processing pipelines using Python and expertise in AWS services. You will be responsible for designing, implementing, and optimizing data workflows to manage large datasets and support analytical capabilities across the organization.

As a Data Engineer – Python, you will work closely with data scientists, analysts, and other stakeholders to ensure that our data infrastructure is scalable, secure, and effective in providing the data solutions needed for the success of the organization.

Key Responsibilities

  • Data Pipeline Development:
    • Design and build scalable, reliable, and efficient data pipelines using Python and key AWS services such as S3, Lambda, Glue, Redshift, EC2, and Athena.
    • Automate and streamline data processing tasks, data transformations, and loading from different sources.
  • AWS Services Management:
    • Work with AWS services including S3, Lambda, Glue, Redshift, RDS, Kinesis, and API Gateway to optimize data processing and storage workflows.
    • Architect serverless Lambda functions for seamless, on-demand data processing and integrate these functions with other AWS services.
  • Data Integration and ETL Processes:
    • Design and implement ETL (Extract, Transform, Load) processes to aggregate data into data lakes and data warehouses.
    • Develop mechanisms for data integrity, validation checks, and monitoring tools.
  • Data Warehousing & Storage:
    • Optimize cloud-based storage solutions and improve the retrieval, security, and performance of data across storage platforms like AWS S3 and Redshift.
    • Work with large datasets efficiently to ensure data is clean, consistent, and ready for reporting and analysis.
  • Collaboration with Teams:
    • Partner with data scientists, data analysts, and business teams to understand and develop solutions for data challenges.
    • Maintain close communication with stakeholders to ensure the development and optimization of systems that meet their data needs.
  • Performance & Cost Optimization:
    • Ensure that the data infrastructure is both scalable and cost-effective.
    • Troubleshoot performance issues with pipelines and storage solutions, ensuring they are running optimally.
  • Documentation and Best Practices:
    • Maintain detailed documentation on data pipeline configurations, service architecture, and development processes.
    • Ensure all processes are followed in accordance with AWS best practices and organization-specific guidelines for performance, security, and compliance.
  • Continuous Improvement & Monitoring:
    • Stay up to date with the latest AWS features and data engineering trends to improve service delivery.
    • Monitor, troubleshoot, and optimize data workflows to ensure a reliable and stable data environment.

Required Skills and Qualifications

  • 5+ years of experience as a Data Engineer or similar role with substantial experience in Python programming and AWS ecosystem.
  • Proficiency in Python:
    • Proven track record of developing efficient, scalable Python code for data processing, automation, and pipeline development.
  • AWS Expertise:
    • In-depth knowledge and experience with AWS services including S3, Lambda, Glue, Redshift, Athena, EC2, and Kinesis.
    • Hands-on experience in managing data pipelines and using cloud resources efficiently.
  • ETL and Data Processing:
    • Demonstrated experience in ETL design and development, and managing workflows involving the transformation and loading of data.
  • Data Warehousing & Databases:
    • Knowledge of AWS Redshift and other cloud-based data warehousing solutions.
    • Experience working with relational (SQL) and NoSQL databases.
  • Big Data Tools:
    • Familiarity with big data technologies such as Apache Spark, Hadoop, Kafka, or other parallel processing tools is a plus.
  • Snowflake Expertise:
    • Strong understanding of Snowflake cloud data warehouse platform, including optimizing data pipelines for loading and querying data.
  • Data Integration & Security:
    • Ability to work with various data sources (APIs, third-party services) and ensure data security and consistency across systems.
  • Additional Skills (Preferred):
    • Familiarity with CI/CD practices and tools.
    • Experience with containerization tools like Docker or Kubernetes.
    • Knowledge of infrastructure-as-code tools like Terraform, CloudFormation, or AWS CDK.
    • Data visualization experience with tools like Tableau or Power BI is a plus.
    • Machine Learning knowledge is advantageous.

Education

  • Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or a related field, or equivalent work experience.

Benefits of Working at Devigital Systems

  • Competitive Compensation Package:
    Enjoy an attractive salary and benefits tailored to your skills and experience.
  • Professional Development Opportunities:
    Access to continuous learning opportunities and career growth in an evolving field of data engineering and cloud technology.
  • Flexible Work Environment:
    Hybrid working model providing flexibility in balancing office and remote work.
  • Collaborative Culture:
    Join a forward-thinking team that thrives on collaboration, idea-sharing, and innovative problem-solving.
  • Comprehensive Benefits:
    Benefit from medical coverage, performance-based bonuses, and a range of company-provided benefits.

How to Apply

Interested candidates are invited to submit their CV along with relevant qualifications and work experience to the following email address

Asad Hameed
Latest posts by Asad Hameed (see all)
Apply for this job
Submission
Gender?

Leave a Comment