Data Engineer (Remote - California Residents Only)

Summary

Identified Talent Solutions is seeking a Data Engineer to join its Infrastructure Team. The role focuses on managing and scaling AWS infrastructure, developing data analytics and engineering tools, leading ETL/ELT processes, optimizing data pipelines, and improving data governance and documentation.

Responsibilities

• Manage, refine, and enhance AWS cloud services infrastructure which include the following: EC2, VPC, RDS, ECS, CloudWatch, CloudFormation, CloudTrail
• Transfer for sFTP, S3, Lambda, Secrets Manager, and Route 53
• Lead ETL/ELT processes which include the development, refinement, and implementation of data loading and processing tools, including Snowflake, Airflow, Python, SQL, dbt, and shell scripts
• Review, redesign, and expand existing analytics and data processing architecture to create an optimal data pipeline(s)
• Identify, design, and implement internal process improvements:
• Automating manual processes, optimizing data delivery, designing infrastructure for greater scalability and automation
• Maintain and update documentation of data architecture from both the macro view (i.e. architectural diagrams) and the micro view (i.e. script level tasks in Airflow)
• Lead meetings, research processes, collect data, analyze information
• Collaborate with the key stakeholders of your projects
• Develop and maintain expert knowledge of our platform and organization
• Continuously improve data pipelines and architecture by staying updated on industry trends and best practices

Skills

• SQL: 2 years (Required)
• Cloud infrastructure: 2 years (Required)
• US work authorization (Required)
• Data Engineer - 2 years
• Remote/Virtual - All candidates must live in California
• The ideal candidate for this team has extensive experience building out complex data architectures, including the testing, maintenance, and refinement of all data pipelines
• They have also experience successfully scaling up an organization's data intake to terabytes and petabytes of data, as well as optimizing data delivery and automating manual processes
• This person is comfortable working with structured and unstructured data and can troubleshoot data loading and processing tools via SQL, Python, shell scripting, AWS, etc, and take on a leadership role in special projects when needed
• Extensive experience developing robust data documentation and data governance protocols is required for this role
• But you must be able to proactively and successfully partner and collaborate with other subject matter experts on a project management basis
• You must be comfortable leading and project managing work with many unknowns and ambiguous solutions, as well as carry a deep knowledge and/or curiosity about the needs and behaviors of students, educators, and parents, and a passion for educational equity
• Build processes supporting data transformation, data structures, metadata, dependency, and workload management
• Advanced proficiency with cloud analytical tools: Snowflake, Redshift, Hadoop, Spark, Kafka, etc
• Advanced proficiency with ETL tools like Matillion, dbt, Talend, etc
• Advanced proficiency with data pipeline and workflow management tools: Airflow, Azkaban, Luigi, etc
• Advanced proficiency building and optimizing data pipelines, cloud architectures, and data sets
• Advanced proficiency with scripting languages like Python, R, Scala, etc
• Advanced proficiency developing cloud infrastructure in AWS or currently has or is in the process of obtaining AWS
• Certification as a Solutions Architect - Associate
• Expert SQL knowledge (SQL Server, PostgreSQL, MySQL, etc.) and understanding of relational databases, query authoring and optimization, as well as working familiarity with a variety of databases
• A successful history of manipulating, processing and extracting value from large structured and unstructured datasets
• Strong decision-making skills and collaborative spirit with the ability to take abstract brainstorming and generate concrete proposals for action
• Advance projects without detailed supervision, balancing multiple responsibilities, and providing colleagues with actionable proposals for advancing collective efforts
• Thrive in a fast-paced environment with changing priorities and deadlines
• Juggle multiple projects of various scopes with ease and grace
• Meticulous attention to detail
• Excellent verbal communication skills; ability to communicate with various levels of professionals
• Strong organizational, project, and time management skills
• We are interested in hiring qualified candidates who are eligible and authorized to work in the United States
• At this time, we are not able to sponsor visas
• As a result, we cannot hire applicants that currently, or in the future, require immigration sponsorship for work authorization (i.e., H1B or F1 Student Visa)

Qualifications

Must Haves

• SQL: 2 years (Required)
• Cloud infrastructure: 2 years (Required)
• US work authorization (Required)
• Data Engineer - 2 years
• Remote/Virtual - All candidates must live in California
• The ideal candidate for this team has extensive experience building out complex data architectures, including the testing, maintenance, and refinement of all data pipelines
• They have also experience successfully scaling up an organization's data intake to terabytes and petabytes of data, as well as optimizing data delivery and automating manual processes
• This person is comfortable working with structured and unstructured data and can troubleshoot data loading and processing tools via SQL, Python, shell scripting, AWS, etc, and take on a leadership role in special projects when needed
• Extensive experience developing robust data documentation and data governance protocols is required for this role
• but you must be able to proactively and successfully partner and collaborate with other subject matter experts on a project management basis
• You must be comfortable leading and project managing work with many unknowns and ambiguous solutions, as well as carry a deep knowledge and/or curiosity about the needs and behaviors of students, educators, and parents, and a passion for educational equity
• Build processes supporting data transformation, data structures, metadata, dependency, and workload management
• Advanced proficiency with cloud analytical tools: Snowflake, Redshift, Hadoop, Spark, Kafka, etc
• Advanced proficiency with ETL tools like Matillion, dbt, Talend, etc
• Advanced proficiency with data pipeline and workflow management tools: Airflow, Azkaban, Luigi, etc
• Advanced proficiency building and optimizing data pipelines, cloud architectures, and data sets
• Advanced proficiency with scripting languages like Python, R, Scala, etc
• Advanced proficiency developing cloud infrastructure in AWS or currently has or is in the process of obtaining AWS
• Certification as a Solutions Architect - Associate
• Expert SQL knowledge (SQL Server, PostgreSQL, MySQL, etc.) and understanding of relational databases, query authoring and optimization, as well as working familiarity with a variety of databases
• A successful history of manipulating, processing and extracting value from large structured and unstructured datasets
• Strong decision-making skills and collaborative spirit with the ability to take abstract brainstorming and generate concrete proposals for action
• Advance projects without detailed supervision, balancing multiple responsibilities, and providing colleagues with actionable proposals for advancing collective efforts
• Thrive in a fast-paced environment with changing priorities and deadlines
• Juggle multiple projects of various scopes with ease and grace
• Meticulous attention to detail
• Excellent verbal communication skills; ability to communicate with various levels of professionals
• Strong organizational, project, and time management skills
• We are interested in hiring qualified candidates who are eligible and authorized to work in the United States
• At this time, we are not able to sponsor visas
• As a result, we cannot hire applicants that currently, or in the future, require immigration sponsorship for work authorization (i.e., H1B or F1 Student Visa)

Benefits

• Remote work arrangement for candidates who live in California