Magnetic’s Inaugural Hackathon

On Wednesday and Thursday, Magnetic hosted our first quarterly internal hackathon. We were lucky enough to be able to bring the whole team into the New York office, from London, Russia, and Slovakia.

Engineers gather to kick off the hackathon in Magnetic’s kitchen

The rules of the hackathon were simple:

1. Teams had 48 hours between the kickoff and presentations. Most teams (wisely) chose to spend at least some of this time asleep.
2. Each team had to have between two and four members to be eligible for winning.
3. Most importantly: have fun!

Because our normal, day-to-day work has a lot of structure, we wanted to be sure to have the hackathon be a change of pace. We explicitly did not require (nor request) that teams work on projects relevant to Magnetic’s business, nor that they use languages or frameworks currently in use or planned for implementation.

Winners

Twelve teams competed, and built projects ranging from very applicable to our business, to entirely technical in nature. The projects were judged by our CTO, James Green, and our Director of Product Management Keren Perry-Shamir, as well as by a popular vote among the engineering team.

CEO Prize: Live Targeting Now

To date, Magnetic’s data collection services write log files which are periodically gathered, compressed, and shipped to Amazon S3; from there, the files are copied down to log processing, targeting, and reporting servers, or any other system which needs log data. In all, this process adds about 15 minutes of average latency to our ability to react to new, incoming data.

Live Targeting Now, implemented by Ed Lim and James Wong, reduces this latency by prototyping a streaming logging system from our search collection service through to our audience database and targeting systems, using Kafka as the messaging layer.

Being able to repsond to events quickly will enable Magnetic’s campaigns to be more successful (we’ll be able to target users within seconds, rather than minutes, after they take a qualifying action), and it’s no surprise that this project was popular among both the Engineering team and our judges.

Ed (left) and James receive their prize, while CEO James Green applauds via Google Hangout.

Congratulations Ed and James!

Product Prize: Will It Work?

One challenge that faces our internal ad operations team on a day to day basis is understanding what impact their changes to targeting rules will have on performance of our clients’ campaigns. One a campaign is set up, we can estimate the size of its audience (how many users it will target), and obviously once it’s running we have real data for delivery, click-through-rate, and other metrics.

Will It Work? was the brain child of Stephen Henderson and Offer Sharabi, and addresses this issue by comparing the targeting rules of existing campaigns to a new campaign set up by ad operations, and provides an intutive visual dashboard of audience size estimates based on similarity of the new campaign to existing campaigns.

Will It Work? identifies similar campaigns and helps users estimate audience size.

Having a forecasting tool like Will It Work? saves our ad operations team from the uncertainty and guesswork of their current workflow, and improves our ability to launch successful campaigns on day one.

Congratulations Stephen and Offer!

Edit: You can read more about Will It Work? in Stephen and Offer’s article about their project.

Popular Vote: SKIP

Magnetic specializes in Search Retargeting, so it is no surprise that any additional insights we can gain into our search and keyword data are immensely valuable. We previously wrote about our Keyword Categorization service, a core part of this effort, and the work continues with SKIP, the Search Keyword Intent Predictor.

Michal Laclavik and Vladimir Tucek added two important features to our keyword categorization service during the hackathon: the ability to ineractively categorize keywords via a web UI, and a related terms generator based on our model and understanding of Wikipedia, one of the key sources for our keyword categorization algorithm.

Taken together, these tools help our adoperations and business development teams to understand and manipulate our search and keyword data, which is too large and unweildy to manage with the tools to which our non-technical co-workers are accustomed (spreadsheets, etc).

SKIP aids users by automatically finding related keywords using a “seed search”.

Congratulations Michal and Vladimir!

Edit: You can read more about SKIP in Michal and Vladimir’s article.

Runners Up

• Go Bidder Go, an experiment to build a request-decoding and routing proxy for our real-time bidding service, using Go and ZeroMQ
• CyCy, a C interpreter with Just-In-Time compiler written in Python, using RPython, a part of the PyPy project
• Spocular, a stateless scheduler and status dashboard for Apache Spark jobs
• MapOnMyLap, an examination of the bipartite community network of our data, which yielded human-understandable clusters of sites and users that could be used for targeting
• Wika-Predicta
• Hack-a-Snack, a tool to track and predict consumption of supplies in the Magnetic kitchen
• WkiGraph
• The “Russian Mafia” team experimented with settings for getting the most out of Apache Spark, the data processing framework we are migrating to
• The QA Force team prototyped test automation for our in-browser javascript data collection tool, an important step towards full end-to-end test automation

In the coming days, we’ll have blog posts from many of the teams giving more detail on the motivation, benefits, and implementation of each of the projects.

More to come

The most important result of the hackathon was that the team had a blast, and everyone wants to do it again. To that end, we’re planning to have hackathons quarterly, and to expand participation beyond engineering. This should ensure that we have a vast array of ideas to work from, and that the projects stay fresh and interesting.