Hackathon Trials & Tribulations
The team traveled a total of 25,000 miles for this. Two engineers flew from Nigeria, two other team members came from across the US. And me. The new guy, flying from NYC on his 4th day on the job. All to try and get a hack done in the next 24 hours at TechCrunch Disrupt. What am I doing here? I thought I was too old for this.
It all started with a cool idea - as it should. Wouldn’t it be awesome to pull someone into a meeting at a moment’s notice? “Hey Alexa, is Tolu available?” –> “Yes she is, setting up a Google Hangout now.” Or “She’s busy in a meeting”. Or “It’s after work hours in Lagos”. Sounds great, right? But as engineers, we have to figure out the nuts and bolts of actually getting this done, in less than one day.
For software engineers, there’s something incredible about a hackathon. The pressure, the cycle of asking the questions, of finding the hurdles and overcoming them. There’s a purity to it. It’s an engineering rush to build quickly, determine tradeoffs, and go with the best solution. There’s no waffling, no politics, and no ego when a team is this united in pursuit of getting something working, fast.
After a few discussions and diagrams, we settled on an architecture:
This is starting to feel pretty complex for a hackathon project. There are a lot of pipes to hook together and get working in sync. In my experience, pipes mean leaks, data incompatibility and trusting the network - which is never a good idea. On the plus side, this architecture allows the team to divide up and build in parallel so that we can make the best use of our time.
We set to work.
The questions, tradeoffs and decisions started coming fast and furious.
Does anyone know a good Node library Redis pub/sub? Should we use Heroku with Node.JS or a full blown AWS infrastructure? What is this AWS Lambda beast? Does anyone know the basics of Amazon Alexa? What the hell is an ‘utterances’ file? This is really the best part of hackathons. You come in, not knowing anything about the APIs or tools you’re going to use and you run into hurdle after hurdle to scale those walls within hours.
A few hours in and we’re making progress. But there are new complexities at every turn. This particular team has never worked together before. Can we really get this all done?
Doubts start to creep into my mind. I’m worried about failing - for my own ego, but also for my team’s. I’ve known them less than a week, but I desperately want to see them succeed. I wonder at what point I need to jump in, steer us towards something simpler, and “save the day”. Calculations are running through my head, estimating the likelihood that the Lambda system will be reliable. And how likely is the Heroku bot going to be resilient to failure scenarios? I hold my tongue for now, but look for opportunities to improve our chances of success.
We’re getting some movement. I think maybe this will work. But there’s so much left. And nothing is tested. Each change we make could bring down the system. It’s getting harder to isolate the variables when debugging as the codebase grows. Oh, and we’re getting tired.
If we’re going to finish this, we need to keep our speed up. I start looking for ways to optimize our process.
I find one. We have basic AWS Lambda code working. Updating this code requires selecting our local files, zipping and uploading it to AWS. For me, this feels like a ridiculous deploy process. And it’s messing with the entire development process. Every time we make changes to our javascript script we have to upload to Lambda to see if it works. Pssh! This is crazy in a world of continuous integration - real-time feedback cycles that I see as a necessity towards building solid products. If I’m not getting feedback in under a minute, I want to build the tools to be sure I can. I start work on a better pipeline.
Fast forward an hour and a half and I’m not making any headway improving our development environment and deploy script, which would automatically test our scripts and deploy it with one command. I look over at my teammate, Brice, and realize he’s already done 15 or 20 iterations of tweaking, compressing and hand uploading in the time I’ve spent trying to optimize our deploys. Shit. I’ve made a big mistake. I’ve wasted over 5% of my total hackathon time trying to make tools, instead of tackling hurdles that would get us closer to finishing an awesome hack. This entire time Brice was making real progress, and what was I doing? Futzing around with unnecessary tooling that wouldn’t matter in a few hours.
How did I screw up this badly?
Engineering is a discipline of trade offs. It’s not a science, not a set of best practices to be rote memorized and spouted out. It’s about considering the entire problem set, the requirements at hand and making the right tradeoffs to attempt a solution. I’d missed the mark because I hadn’t taken into account the time frame I was working in. With 24-hours to build something awesome, every wasted hour is critical, and every optimization pointless.
Luckily, engineering is a team sport and my team knew exactly what to focus on. With a few hours left to hack, I set aside my uncompleted script and got back to the real work. It was a scramble to the end, but on Sunday afternoon I watched nervously as our demo team took the stage. Ugly deploy system, leaky pipes, and questionable network be damned - in the moment, everything works perfectly. We built the right tool for the job.
