jack

Do you want a Time Travel Debugger

2021-09-30T16:02:39Z

I never got around to talking about what my current work do: http://undo.io There was some previous discussion on the topic on facebook: https://www.facebook.com/jack.vickeridge/posts/10103938681712440

What is a Time Travel Debugger

It records everything that happens in a program's execution, so you can step backwards as well as forwards, or rewind execution and then replay it again more carefully. Or you can "replay" it backward, e.g. going to the end of time, seeing your program crashed with a null pointer and then setting a watchpoint on that pointer and reverse-continuing until you find out where the pointer was set to that value.

There's two main modes of use, using it like a debugger sitting in front of a program, or using a companion recorder (which is actually an executable with much of the same code but packaged differently) to record your program in your overnight test suite, or running to replicate a bug that happens in a very long running process. Then once you've reproduced the bug once, you've almost finished, you can just load up the recording and step forward and back in a debugger until you figure out what went wrong.

That sounds impossible!

Yes, it does sound impossible, but it works.

It records literally everything the program does that interacts with the outside world in any way, e.g. any system call (including any file access, network access, gettime, even getpid, etc, etc), any instructions which write to shared memory, etc. That can get large for some programs (but customers do use it successfully!)

It saves a snapshot at several points during history (by forking the process there), so it can create the state of any point in history by forking another process from that snapshot, and playing it forward using the saved events instead of actually doing any of the things that interact with the outside world.

It does all this by rewriting the compiled program in memory, and maintaining a mapping between the rewritten memory and the original assembly. So you the user see the original source code and original assembly, with whatever level of debug info you originally compiled the program with. But behind the scenes, almost any non-trivial instruction is rewritten to do something else, to either to save the result of the instruction in the event log, or to replay the value from the event log.

That means that you can attach it to any program, compiled any way, just like any debugger can. You don't need to compile it with some magic -- people keep expecting this, and it could have been written that way, but instead, you can just connect it to any program you could attach gdb to.

Caveats

Recording multiple threads is slow, and recording multiple processes doesn't exist yet. We're working on it, but right now can help with some multithreading bugs but can't help with others.

Program execution is slower, between 2x and 10x. We are working to improve that. Replaying through execution can be faster than that (and you can usually go directly to the beginning, end, etc without any replaying).

This is all on linux only.

The interface and implementation is based on the gdb forntend/gdb server protocol. So by default it looks like debugging with gdb but with "reverse-next" as well as "next". And it works with any program which uses gdb backend, e.g. visual code, emacs, although some of those are tested extensively and some aren't.

But no linux debugger has a very good UI, so currently it is mainly used by people who have to debug using something like gdb anyway, but want to be able to solve harder bugs quicker. We are trying hard to make it easy for languages like python and java where the translation has to understand an interpreter as well as the code. This works in the sense that it can be recorded and replayed, but getting a good user experience is a lot harder.

Worth and Price

I always describe it as, the difference between "not having a debugger" and "having a debugger". If you have a debugger, maybe actually 90% of problems you can solve with print statements. But the 10% that you can't fix with print statements could take months to solve without a debugger, or hours with a debugger. It's hard to describe why you need a debugger to someone who hasn't tried using one. But almost no-one would go back to not having one.

A time travel debugger makes trivial the small proportion of issues that still feel impossible even with a debugger. You say, "yes, it fails intermittently but we don't know if we'll ever track it down unless someone wants to study the failure for nine months", but that might be only hours with the right tool.

Unfortunately, this tool takes a large amount of programmer effort to create, and is only viable if it's sold commercially. If you view it as "The 5% of bugs we have that take 9 months to track down, instead get solved in a few hours", you compare the cost to the salary for an extra programmer or two, it's very reasonable. But most people including me hate paying for tools, so it's hard to sell.

It has a great retention rate -- any companies which have subscribed to a contract, have almost always kept it, and programmers who have used it regularly (including me) are very very eager to keep having it available.

Currently there are several introductory offers. There's an educational license which is cheaper or free. There might be an offer of free licences to the right open source project if you're interested. There's a 30 day free trial, and a personal license, in the hopes people will become converts and persuade their employer to adopt it. There is standing offer that if you have an intractable hard to reproduce bug that's you'd like to see just go away, we can arrange some sort of trial to have someone come and help capture and diagnose that bug, and see if that leads to a longer term arrangement.

Ask questions in the comments. Feel free to download the trial -- if you've used gdb, it's fairly straightfoward to try out, and it's magical to see "step back, step forward".

Or if it sounds like you might be someone who would actually benefit from acquiring a license, I can put you in touch with helpful people -- we used to focus on big clients because there was a lot of shakedown, but now it works more reliably out of the box, it's plausible for a wider spectrum of companies and people.

comments

Jobs at Adder

2018-07-09T11:46:07Z

The exact combination of jobs advertised varies over time, but work are looking to hire junior, intermediate and senior software engineers and (I think) testers.

We mostly do low-and-medium-level embedded programming, with audio/video and networks. Although for the graduate software engineer jobs, probably any programming background is fine.

I'm not shilling for them as the most amazing company ever, but they've been reasonably good to work for so far.

If anyone is interested I can put you in touch.

I also wonder, is there anywhere I should be advocating they advertise to reach an appropriately diverse demographic of engineers?

comments

Do I know anyone who does UI work?

2018-07-09T11:39:40Z

Work are contracting out some UI work. Basically, there's a fairly good idea of what features are needed, but are investigating the possibility of handing off the work of what different screens there are, what options on each screen, etc, etc to someone else.

It will likely be implemented through a web-browser inside, but not be interacted with as a webpage as such.

They already have one fairly promising contact, but are interested in having at least two choices. (Pro: they will probably make a decision and not dither. Con: they may go with the existing contact.)

Do I know anyone or any company who does that sort of work?

I can't vouch for what they're like to be a contractor for, but as far as I know from the inside, they're looking to get the work done, not to cut costs. And they're not specifically looking for future work, but if they find a company they get good results from, there's other projects it might make sense to do a similar thing with.

If there is anyone I can put you in touch with the relevant people.

comments

Working without internet

2014-04-08T12:41:13Z

On Friday, I turned the internet off (well, just my main sources of procrastination) while I was working. I immediately felt massively better. I've kept it up for two days now with reasonable success -- can I keep it going?

I've tried that before, but it never worked. I think it helps when I've reached the point where I _can_ work, but am being distracted, but didn't make much difference when something was preventing me working whether I was distracted or not.

Although I'm still not working as consistently hard as when I had the goad of working closely with someone else every day.

comments

Work trip to Germany

2014-02-09T15:19:50Z

Work trip to Germany was quite good. The travelling was easy, as was spending three days in the company of people from work (one from my office, one from one of the German offices).

I coped in Germany only speaking a few tourist phrases, although I need to get better at sometimes jumping into conversations when I know the people speak English, but don't automatically switch to English when I enter the room.

I managed to spend a couple of hours walking around the old parts of the town and looking at the river, and to go to a couple of different places for meals, since most places had some vegetarian dish, but not a very interesting one.

For the first time I had to ask myself what's normal to put on expenses. I have to keep reminding myself that my time is valuable to work, and that if it's worth sending me abroad at all, it's not worth saving a few extra pounds by being extra frugal. But I have to force myself not to think "oh no, surely that's just gratuitous". I think work clothes you don't have any other use for, and meals when you're away from home are the normal standard? And beer with meals when it's cheaper than the water?

The work itself was generally a success, we did everything we intended to, and didn't screw everything up, and I've a good idea of what we want to achieve. But it still feels like banging my head on a brick ceiling that 90% of what I say we need to do is dismissed, I think I should persist with the last 10% but I'm not sure. I'm not sure how much I'm bad at sounding like I know what I'm talking about, and how much I should better have figured out which decisions were taken several months ago but I wasn't told about.

The factory was really interesting. It's like a tutorial in best practices, eg. most of the furniture is on wheels and COULD be anywhere, but there are strips of tape on the floor showing where it SHOULD be. It's about half robots and half people. And people have to ride bikes around when they're going round the whole thing.

comments

Laptop

2011-05-26T12:00:23Z

Got drunk ✓
Bought laptop ✓

comments

Resigned from work

2010-04-30T16:06:42Z

We couldn't agree anything so I've just resigned as of now, and am leaving immediately as of this evening! However, we parted very amicably so YAY even though EEK!

That's awkward because I'm jobhunting as of NOW but that's life. I feel good about it, and think and hope work do too.

(Now I'm looking for a job immediately, and don't know whether or not I will be able to do the contracting-from-Stoke option, I need to sort logistics on that immediately!)

OK, am off to London for Mum's graduation. Love to you all! J.

comments

Snow

2010-01-06T14:59:45Z

I came back to work at home because of the snow. I have to work out of Cambridge and drive home, so if the snow got really bad this afternoon, it'd be really really annoying (or dangerous). Unfortunately, a few years ago, snow was exciting and pretty, but now it's more stressful and annoying.

Two days ago, working at home would be great, but I'd just got set up to work on the desktop, and now I have to switch back to the laptop, and the transition isn't seamless yet, so every context switch feels like an annoying leap backwards.

On the other hand, English towns, trees, lanes and fields are really, really pretty in the snow.

comments