• Full Time (allows remote)
  • London
  • No. of Vacancies: 3
  • Experience: 3 - 5 Years
  • Posted 7 months ago

Python Developer – Web Scraping position

HENI is looking to increase its data-gathering capability and we are looking for Python Developers (Junior through to Senior) to help us do this. As a Python Developer/Web Scraper the role is split into three main areas:

  • Maintenance of existing crawlers
  • Setting up new crawlers
  • Scraping the NFT/Blockchain market



From a tooling perspective, we use Python (Scrapy) to set up the spiders. We have an AWS based infrastructure and data is stored in S3 buckets as well as SQL based databases (MySQL and PostgreSQL).


You must have experience with:

  • Scrapy (Python)
  • SQL
  • Pandas (Python)


Key Responsibilities:

  • Writing and maintaining software for digital data collection for hundreds of websites
  • Developing software to allow for scaling of data gathering across thousands of sources and across multiple team members
  • Working with and deploying data pipelines (e.g. Scrapy, written in Python), process and cleaning the data, and storing accurately into our database
  • Setting up scrapers, fixing any bugs or issues when websites change


Need to know information:

  • This role can be fully remote, anywhere across the globe.
  • We are looking for people who can do this full time (40 hours per week).
  • We can offer a competitive salary or hourly rate. It’s almost impossible to put a figure on this as different countries have different living costs but we are very competitive.
  • If you are based in the UK, we can offer a permanent contract. If you are based overseas, we can offer a freelance contract.

Please answer the questions below first

Please enter your personal information

only doc, pdf or docx file allowed and file size will be less than 2MB

Why do you think you are a good fit for this job?