Python Developer – Web Scraping position

HENI is looking to increase its data-gathering capability and we are looking for Python Developers (Junior through to Senior) to help us do this. As a Python Developer/Web Scraper the role is split into three main areas:

• Maintenance of existing crawlers
• Setting up new crawlers
• Scraping the NFT/Blockchain market

 

Tooling

From a tooling perspective, we use Python (Scrapy) to set up the spiders. We have an AWS based infrastructure and data is stored in S3 buckets as well as SQL based databases (MySQL and PostgreSQL).

 

You must have experience with:

• Scrapy (Python)
• SQL
• Pandas (Python)

 

Key Responsibilities:

• Writing and maintaining software for digital data collection for hundreds of websites
• Developing software to allow for scaling of data gathering across thousands of sources and across multiple team members
• Working with and deploying data pipelines (e.g. Scrapy, written in Python), process and cleaning the data, and storing accurately into our database
• Setting up scrapers, fixing any bugs or issues when websites change

 

Need to know information:

• This role can be fully remote, anywhere across the globe.
• We are looking for people who can do this full time (40 hours per week).
• We can offer a competitive salary or hourly rate. It’s almost impossible to put a figure on this as different countries have different living costs but we are very competitive.
• If you are based in the UK, we can offer a permanent contract. If you are based overseas, we can offer a freelance contract.