Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

c20wSmJNbGR1K0diM0diNllJcGJwTTJ2Y1E9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

Multimedia Solutions Corp.

Join us for a Digital Marketing Internship Job at Multimedia Solutions Corp.

Join us for a Digital Marketing InternshipAre you interested in helping to transform the way people communicate?At Multimedia Solutions...  ...Internships are unpaid; Stipends available for summer internships with longer commitments. Schedule MSC... 

JMJ PHILLIP

EXECUTIVE SEARCH CONSULTANT Job at JMJ PHILLIP

We are seeking a dynamic and driven Executive Search Consultant to play a key role in identifying and placing top executive talent for a wide range of clients. This is a client-facing, strategic position where you will lead search engagements, manage candidate pipelines... 

The MPH Team - Compass Florida.

Real Estate Virtual Assistant Job at The MPH Team - Compass Florida.

Were looking for a driven Transaction Coordinator assistant to manage contracts throughout the entire home-selling and buying process. Youd be responsible for overseeing transactions from start to finish to ensure offers and counteroffers are approved for a smooth and... 

CPC Clinical Research

Drug Safety Associate II Job at CPC Clinical Research

 ...Salary: $64,000 - $76,000 We are looking for an experienced Drug Safety Associate II to join our team! The Drug Safety Associate II...  ...clinical safety and pharmacovigilance regulations. Remote applicant are welcome to apply. Drug Safety Responsibilities... 

Blue Sky Hospitality Solutions

Hotel Rooms Inspector Job at Blue Sky Hospitality Solutions

 ...individual conducts detailed inspections, supports housekeeping staff, and ensures that guests have a positive and comfortable stay at hotel. Inspect guest rooms and public areas for cleanliness, presentation, and compliance with Hilton brand standards. Report and...