This is Part I of a two part series on my attempt to collect, clean, analyze and visualize data from the Ask HN: Who is Hiring? threads. This first part is a discussion of my analysis. Part II goes into the details of how I collected and analyzed this data.
Geeks love YCombinator’s Hacker News! So its hardly a surprise that when they posted the first Ask HN: Who’s Hiring thread in April 2011, job seekers and posters flocked to it in droves to find each other. This was despite HN’s unwieldy, almost unusable interface. Since that first post, 7 years ago, there has been a similar post on HN every single month. The most recent one (at the time of writing this article) from April 2018, attracted more than 700 job postings from around the world; Ask HN: Who is hiring? is stronger than ever.
Hacker News’ audience is very likely to be a decent sample set of geeks, nerds and tech entrepreneurs, who are all early adopters of latest tech trends. This led me to think that it may be interesting to analyze data from these threads to foretell how the technology landscape is shifting. More specifically, which regions of the world have been hiring for what capabilities? How rapidly is remote work picking up pace? Which technologies are becoming obsolete and which are gaining traction? Among many other, these are some interesting questions that can be answered by analyzing these posts.
My gut reaction was to see if something like this already exists. A number of interesting results pop up when you google ‘Hacker news hiring trends’:
As with almost anything on the internet these days, it turned out that a few people had already done interesting work on the same idea. The most notable among them is HNTrends which is essentially a tech-keyword frequency analysis of all Ask HN:Who’s Hiring threads. Even with it’s minimal analysis, the insights it offers are pretty fantastic. Among other things, I learned because of HNTrends that ReactJS is the most sought-after tech skill in nerd universe today and GoLang is rapidly making headway towards becoming one of the top 5 most desirable languages in geekdom.
The only problem with HNTrends is that the analysis is too generalized. For example, the technology-keyword frequency is not necessarily an accurate measure of how many jobs are posted for that particular technology in any given month. Even if some comments are criticizing a technology in comments, the ‘mentions’ there can induce a false positive bias in results.
Given these issues, as well as my desire to look into this data a bit more deeply, I decided to write some quick Python code to analyze it myself.
I visualized the data I was able to collect, using Tableau. You can view and play around with these visualizations on my public Tableau profile here.
Technology — the geographical landscape
The very first visualization I created shows the distribution of HN job posts across the world in the last 4 years. A few obvious and some not-so-obvious things to note here are:
- The US, in general, and San Francisco, in particular, is, by far, still the capital of the world for tech jobs.
- Having said the above, the geographical density of tech jobs is surprisingly high in the United Kingdom. USA is at least 40x the size of UK but the number of job posts from the US is only about 6 times as many as the number of job posts from the UK.
- In Europe, Berlin and Amsterdam (but not Paris) are right behind London but they are pretty distant second and third.
- Russia has a negligible number of job posts on hacker news which really skews the data. Russian hackers possibly consider themselves, perhaps even with good reason, to be above the rest.
Here is the list of top 20 cities in the world with most tech jobs posted on Hacker News over the last 4 years:
The trends make this picture a bit more interesting:
San Francisco is not only the biggest tech market, it’s also the fastest-growing, followed by New York and London. In Europe, Berlin seems to be emerging as the new tech destination and from the trend-line, seems poised to overtake both Boston and Seattle.
Technology Trends
Looking at the technology trends, there is a whole bunch of interesting insights. Lets look at the top 5 technologies over the years and how their demand is changing:
- The top 3 technologies: JavaScript, ReactJS and Python, are in a league of their own. They have not only been the most sought after skills over the years, the demand for these skills is also increasing at an increasing rate. The demand for React professionals has especially been exploding over the years.
- Among other popular skills, iOS is definitely the one where the demand is beginning to decline. The popularity of React (and React Native) could be one explanation for that.
Lets look at the next 10 technologies with the most jobs posted on Hacker News:
Python, React and GoLang, are the only technologies where the demand is clearly going up. For everything else, its either holding steady or going down.
Another interesting trend which was contrary to my expectations was the preference for on-site vs remote jobs. I expected that the acceptability of remote work is increasing but the trends are inconclusive at best. See for yourself:
A majority of jobs (>80%) are still on-site and the trend-line for on-site jobs has a positive slope indicating an increasing preference for it.
Finally, I wanted to see which of the jobs pay the highest. Only a few companies actually mention the pay range with their job post. So for my analysis, I only considered technologies where at least 500 jobs posts had mentioned a salary scale in order for me to remove outliers. I then used the average for minimum and maximum salary. Here is what that visualization looks like:
Hardly a surprise that Data Science is almost at the top of the list and Machine Learning is among the top 10. However, Docker and Devops being in the top 10 list, took me by surprise. Also surprising is the fact that despite their popularity and demand, Javascript, React, Python and GoLang do not figure in the top 10 most highly paid jobs list.
This is the end of part I. If you would like to see a certain trend that I have missed, let me know in comments and I will add it to my analysis.
In part II I discuss how I achieved these results using Python. If you are a tech geek like myself you may be interested in that story also.
This blog was originally published on Medium.
