Data Pipeline Software Developer

Location: Vancouver, British Columbia, Canada

Department: Engineering

Type: Full Time

Min. Experience: Experienced

We’re looking for a Data Pipeline Software Developer with a minimum of three years experience to take our analytics engine and data pipelines to the next level. We are looking for someone with production experience in development and maintaining analytics and ETL pipelines on the Hadoop ecosystem on-premise or in the cloud (AWS, Google, Azure etc.)

We’re an enterprise marketing analytics company based out of Railtown, in Vancouver. We’re an unusual company, being completely private, bootstrapped, and profitable. We’ve experienced high growth over the past years and the team is more than 40 people strong.

OUR TECHNOLOGY

Here’s what you’ll be working with:

We’re currently collecting and analyzing tens of millions of records and terabytes of data everyday. We do this with a system built on AWS, that distributes its workload over thousands of EC2 instances, in addition to many other services. We collect and analyze data via our legacy pipeline developed in Ruby, with our latest analysis processes being developed in Scala using Apache Spark and running on Amazon EMR.

We’ve been delighted with the success we’ve been having with Spark so far, and want to bring on a talented developer who has experience developing with distributed data analysis engines like Spark or Flink. We’re scaling up our operation, and we plan to implement a collection of batch and streaming data pipelines on AWS using technologies such as Docker (on ECS), Spark (on EMR), AWS Batch, and DynamoDB streams. You will be instrumental in evaluating, designing, implementing and deploying pipelines with these new technologies.

Your main duties and responsibilities:

  • Plan, design, and coordinate the development of software components for our data collection and data aggregation system using Ruby, Scala, and other AWS services.
  • Rigorous and thorough research and evaluation to improve and evolve STAT’s data collection systems and data aggregation pipelines to be more reliable, faster, and cost efficient.
  • Work with the Data Services team to manage the operation of STAT's data collection systems ensuring that they are running smoothly at all time, upgrading, and developing maintenance procedures as needed.
  • Draft technical documentation of our software, development environments, production environments, and procedures. This documentation is primarily for in company use by our development team.
  • Participate in team level meetings and discussion for software design, software development, and code review.
  • Discuss progress on work being undertaken by yourself and the Data Services team on a regular (daily and weekly) basis with the rest of the development team and participate in the ongoing refinement of our software development process.

REQUIREMENTS

You’ve got experience

You should have at least three years of experience in Data Infrastructure and Analytics engineering and should be comfortable working with a team in an agile process. This role is very hands-on and will be focused on system architecture, software design, research, testing, and development.  

You’ve got education

At a minimum, you should have a post-secondary diploma or equivalent degree in computer science, engineering or a related discipline.

You’ve got skills

We are looking for people who are confident working in our technical landscape, including distributed technologies and software-as-a-service. In order to be eligible for this role, we need you to display competency in the following areas:

  • Experience in building data pipelines using AWS EMR, AWS Data Pipeline, ECS, SQS or similar technologies
  • Scala or other common data pipeline languages like Java, Haskell etc.
  • Knowledge of Hadoop ecosystem and similar ETL based frameworks
  • Ruby or other scripting languages like Python, Go etc.
  • SQL databases like MySQL, PostgreSQL, Microsoft SQL Server, etc.
  • NoSQL databases like MongoDB, DynamoDB, Redshift, etc.
  • Caches like Redis, Memcache, etc.
  • Queue Management Systems like AWS SQS, Resque-Redis, ZeroMQ, Celery, etc.
  • TCP/IP networking and HTTP protocol
  • BDD or TDD development in an Agile-like process
  • CI, provisioning and deployment using Travis, Chef and capistrano respectively
  • Source control like Git, SVN

PERKS

A few side benefits of working here:

Three week paid vacation, plus everyone gets an extra two-week paid vacation leading up to New Year’s.

  • Shorter summer hours to take advantage of that elusive Vancouver sunshine.
  • Free gym memberships.
  • Medical package including dental, optometry, prescriptions, massage, and physio.
  • A relaxed and welcoming work environment

CULTURE

Our Development culture at STAT is an open and collaborative agile like workflow. We believe that the best way to solve challenging engineering problems is to bring the most inventive ideas together through discussion and then implement solutions through an iterative process that involves constant communication and feedback.

We believe the best way to foster growth and creativity is to bring multiple minds together on an issue and then make one individual responsible for ultimately solving it. We are never hesitant to ask for help/advice and never too busy to offer the same.

We're a small, highly focused team of experts, doing world-class, global-scale data mining and classification. We're also building interfaces around this data that allow our enterprise customers to leverage the insights we provide in order to grow their businesses. Our customers love us.

Our founder & CEO is a software developer at heart, so we have a development-focused culture. We're a 100% revenue-funded private company with a strong sense of craft. We are focused on delivering excellent work, and having a great time while doing it.

Forward this Position
Recipient email address (one)
Your name
Your email address
Enter a message (optional)
Human Check*
Apply for this Position
* Required fields
First name*
Last name*
Email address*
Location
Phone number*
Resume*

Attach resume as .pdf, .doc, or .docx (limit 2MB) or paste resume

Paste your resume here or attach resume file

Cover Letter
In 150 characters or fewer, tell us what makes you unique. Try to be creative and say something that will catch our eye!*