Data Analytics at Income Insurance is the enterprise team that champions and develops AI/ML solutions. We are focusing on in-house solution development as well as partnerships. To ensure the successful execution of Income’s enterprise strategy to build Analytics as a product through creating impactful AI/ML solutions, driving innovation and cultivating AI culture, we are seeking a highly skilled and experienced Data Engineer to join our team.
The selected candidate will focus on in-house developed solutions deployment and maintenance. You will play a crucial role in understanding requirements from business stakeholders, implementing changes on existing pipeline, migrating codes from Cloudera to AWS, optimizing data workflows, and ensuring seamless integration and running of data and analytics pipelines on the AWS platform.
Responsibilities:
- Take the ownership of key data pipelines in AI /ML use cases, from design and implementation to optimization and maintenance.
- Serve as a technical specialist within a data science team. Design, develop, and operate data pipelines to provide high-quality datasets for analytical and AI / ML use cases.
- Collaborate with other data engineers, analysts, data scientists, product specialists, and other stakeholders to build well-crafted, pragmatic, and robust engineering solutions that meet business requirements.
- Stakeholder management and engagement. Proactively engage with stakeholders to understand their data needs and translate them into technical requirements for data pipelines in AI / ML.
- Maintain documentation of processes, code changes, and data flows.
- Demonstrate a strong understanding of data privacy regulations such as PDPA to ensure zero breaches.
- Manage existing runs and deployment of AI / ML model pipelines to ensure efficient data processing and model training.
- Foster an innovative and growth-oriented mindset, continuously seeking opportunities to enhance data engineering processes and drive improvements across the organization.
Requirements:
- At least 5 years of experience in data engineering with relevant experience in the big data ecosystem.
- A bachelor’s degree in computer science or equivalent.
- Proficient in SQL, Python, and Spark. Experience with streaming frameworks like Kafka and Spark Streaming is a plus.
- Proven experience in implementing MLOps practices on AWS.
- Experience with Data Warehouses such as Redshift, BigQuery, Snowflake, Hive, and S3.
- Experience in leveraging DevOps frameworks to orchestrate and monitor production deployments – Jenkins, Git/AWS CodeBuild, AWS CodeDeploy.
- Experience in developing data pipelines leveraging AWS services such as Glue/EMR, Lambda, CloudWatch, and Athena.
- Intermediate-level knowledge of Linux commands and shell scripting.
- Good understanding of the data science model development and deployment process.
- Proven record of designing and implementing large-scale data processing pipelines.
- Passionate about technology and always looking to upskill based on new developments in data engineering practice.
- AWS certifications will be a plus.
