Being a productive deep learning researcher takes more than just an understanding of theory. Spending a little time setting up the right tools can pay off in just a few hours, and save you an enormous amount of time!

If you feel like using cloud GPUs adds overhead to your training or development workflow, these tools are for you. Once you get used to them, developing and training models in the cloud will feel just like doing it on your local computer — except with unlimited resources!

We’ll talk about five key tools here: remote debugging, agent forwarding, port forwarding…

Deep Learning: Need For Speed

When training deep learning models, performance is crucial. Datasets can be huge, and inefficient training means slower research iterations, less time for hyperparameter optimisation, longer deployment cycles, and higher compute cost.

Despite all that, it can be hard to justify investing too much time speeding things up, as there are many potential dead ends to explore. But fortunately, there are some quick wins available!

I’m going to show you how a simple change I made to my dataloaders in PyTorch for tabular data sped up training by over 20x — without any change to the training loop! Just a simple…

This post was first published on www.harald.co.

Reproducibility is hard.

Last week I wrote a simple Reinforcement Learning agent, and I ran into some reproducibility problems while testing it on CartPole. This should be one of the simplest tests of an RL agent, and even here I found it took me a while to get repeatable results.

I was trying to follow Andrej Karpathy and Matthew Rahtz ‘s recommendations to focus on reproducibility and set up random seeds early, but this was taking me much longer than expected — despite adding seeds everywhere I thought necessary, sometimes my agent would…

This post was first published on www.harald.co.

Some brief notes from day 3 of NeurIPS 2018. Previous notes here: Expo Tutorials Day 2.

Reproducible, Reusable, and Robust Reinforcement Learning (Professor Joelle Pineau)

I was sad to miss this talk, but lots of people told me it was great so I’ve bookmarked it to watch later.

Investigations into the Human-AI Trust Phenomenon (Professor Ayanna Howard)

A few interesting points in this talk:

• Children are unpredictable, and working with data generated from experiments with children can be hard. (for example, children will often try to win at games in unexpected ways)
• Automated vision isn’t perfect, but in many cases it’s better than the existing baseline (having a human record certain…

This story was first posted on www.harald.co.

Just a brief highlight from day two: Professor Michael Levin’s incredible talk on What Bodies Think About, summarising 15 years of research in exploring the hardware/software distinction in biology. Hello, Cronenberg World.

A brief introduction on how the brain is far from the only place where computation happens in biology. Experiments with regenerative flatworms show that memories persist even when their heads are removed and grow back. Butterflies can remember experiences they had when they were caterpillars.

Then the key bit of the talk: reverse engineering bioelectric signals to trigger high-level anatomical subroutines…

This story was first published on www.harald.co.

Monday was the first day of the core conference, following the Expo on Sunday. There were a number of really interesting tutorials. Here’s a brief summary of the three I managed to attend.

Scalable Bayesian Inference (Professor David Dunson)

This tutorial explored one main question: how do we go beyond point estimates to robust uncertainty quantification when we have lots of samples (i.e. ’n’ is large) or lots of dimensions (i.e. ‘p’ is large)?

The introduction was Professor Dunson’s personal ode to Markov Chain Monte Carlo (MCMC) methods, and in particular the Metropolis-Hastings algorithm. …

This story was first published on www.harald.co

Today was NeurIPS Expo, the zeroth day of this year’s Neural Information Processing Systems conference in Montréal. The Expo is a day with content from industry right before the rest of the conference. Below are some highlights from a few of the sessions I managed to attend.

The Montréal Declaration

An initiative of the University of Montréal, the Declaration “aims to spark public debate and encourage a progressive and inclusive orientation to the development of AI”.

Unlike the Asilomar AI Principles, which were set by experts in the field, the principles of the Montréal Declaration are…

This post was first published on www.harald.co.

It’s easier to read than to remember what you’ve read. I used to struggle to remember what a book was about, even just a few years after reading it.

I don’t have this problem anymore. In a few recent conversations about books, people have asked me how I manage to remember so much about books I read a long time ago. I’ve also been surprised at how clear the structure of certain books is in my mind.

I don’t think it’s because my brain has got better at remembering things. I think I’ve…