Data Pipeline Engineer

Data Pipeline Engineer – London

We’re working with a fantastic technology company, working in the Arts and Media sector to help creatives monetise their art and reach as wide an audience as possible.

Data is increasingly becoming a large part of their service, so we’re looking for an experience Data Pipeline engineer to join the team.

Starting remotely, with time in the office in the future, we’re looking for someone with a broad data engineering skillset who will be able to take large volumes of web, social media and blockchain, and move it into a workable format for further analyst.

You will:

* Write data pipelines in Python or R to extract data from blockchain & service APIs into relational databases.
* Develop tools to monitor existing data pipelines and web scrapers.
* Improving the data quality of the databases by working on hotfixes with the help of QA & fellow data engineers/data scientists.
* Deploying machine learning models on a secured public facing API via Kubernetes/Kubeflow/Ray Serve.
* Writing and securing public facing APIs on Kubernetes to allow external partners to access our services.
* Work with data scientists in deploying the following on Kubernetes/AWS cloud

* Building datasets and training models
* prediction performance monitoring of models already deployed and serving.
* online machine learning pipelines

* Monitoring load on databases and suggesting ways to optimise SQL queries.
* Mentor and develop junior members of the team.

We’re looking for:

* Good experience with Python/R and SQL.
* Building and deploying data/machine learning pipelines.
* Deploying jobs and services on Kubernetes.
* Working with AWS EC2, RDS, S3.
* GitHub – PRs, Git Flow etc.
* CI/CD – Concourse/GitHub Workflows/GitOps
* Backend API development for model/data serving.
* Familiarity with machine learning, blockchain and data science would be a benefit.
* Motivation to work with dirty data and to clean it.

On offer for the right candidate is a generous salary in line with your experience, as well as a good benefits package with everything you’d expect from a leading client. You’ll have opportunities to work on fascinating data challenges and new, emerging technologies.

For more information, apply now!

Data Pipeline Engineer – London

stream is acting as an Employment Agency in relation to this vacancy.


We use cookies to improve your experience on our site find out more