jack

I never got around to talking about what my current work do: http://undo.io There was some previous discussion on the topic on facebook: https://www.facebook.com/jack.vickeridge/posts/10103938681712440

What is a Time Travel Debugger

It records everything that happens in a program's execution, so you can step backwards as well as forwards, or rewind execution and then replay it again more carefully. Or you can "replay" it backward, e.g. going to the end of time, seeing your program crashed with a null pointer and then setting a watchpoint on that pointer and reverse-continuing until you find out where the pointer was set to that value.

There's two main modes of use, using it like a debugger sitting in front of a program, or using a companion recorder (which is actually an executable with much of the same code but packaged differently) to record your program in your overnight test suite, or running to replicate a bug that happens in a very long running process. Then once you've reproduced the bug once, you've almost finished, you can just load up the recording and step forward and back in a debugger until you figure out what went wrong.

That sounds impossible!

Yes, it does sound impossible, but it works.

It records literally everything the program does that interacts with the outside world in any way, e.g. any system call (including any file access, network access, gettime, even getpid, etc, etc), any instructions which write to shared memory, etc. That can get large for some programs (but customers do use it successfully!)

It saves a snapshot at several points during history (by forking the process there), so it can create the state of any point in history by forking another process from that snapshot, and playing it forward using the saved events instead of actually doing any of the things that interact with the outside world.

It does all this by rewriting the compiled program in memory, and maintaining a mapping between the rewritten memory and the original assembly. So you the user see the original source code and original assembly, with whatever level of debug info you originally compiled the program with. But behind the scenes, almost any non-trivial instruction is rewritten to do something else, to either to save the result of the instruction in the event log, or to replay the value from the event log.

That means that you can attach it to any program, compiled any way, just like any debugger can. You don't need to compile it with some magic -- people keep expecting this, and it could have been written that way, but instead, you can just connect it to any program you could attach gdb to.

Caveats

Recording multiple threads is slow, and recording multiple processes doesn't exist yet. We're working on it, but right now can help with some multithreading bugs but can't help with others.

Program execution is slower, between 2x and 10x. We are working to improve that. Replaying through execution can be faster than that (and you can usually go directly to the beginning, end, etc without any replaying).

This is all on linux only.

The interface and implementation is based on the gdb forntend/gdb server protocol. So by default it looks like debugging with gdb but with "reverse-next" as well as "next". And it works with any program which uses gdb backend, e.g. visual code, emacs, although some of those are tested extensively and some aren't.

But no linux debugger has a very good UI, so currently it is mainly used by people who have to debug using something like gdb anyway, but want to be able to solve harder bugs quicker. We are trying hard to make it easy for languages like python and java where the translation has to understand an interpreter as well as the code. This works in the sense that it can be recorded and replayed, but getting a good user experience is a lot harder.

Worth and Price

I always describe it as, the difference between "not having a debugger" and "having a debugger". If you have a debugger, maybe actually 90% of problems you can solve with print statements. But the 10% that you can't fix with print statements could take months to solve without a debugger, or hours with a debugger. It's hard to describe why you need a debugger to someone who hasn't tried using one. But almost no-one would go back to not having one.

A time travel debugger makes trivial the small proportion of issues that still feel impossible even with a debugger. You say, "yes, it fails intermittently but we don't know if we'll ever track it down unless someone wants to study the failure for nine months", but that might be only hours with the right tool.

Unfortunately, this tool takes a large amount of programmer effort to create, and is only viable if it's sold commercially. If you view it as "The 5% of bugs we have that take 9 months to track down, instead get solved in a few hours", you compare the cost to the salary for an extra programmer or two, it's very reasonable. But most people including me hate paying for tools, so it's hard to sell.

It has a great retention rate -- any companies which have subscribed to a contract, have almost always kept it, and programmers who have used it regularly (including me) are very very eager to keep having it available.

Currently there are several introductory offers. There's an educational license which is cheaper or free. There might be an offer of free licences to the right open source project if you're interested. There's a 30 day free trial, and a personal license, in the hopes people will become converts and persuade their employer to adopt it. There is standing offer that if you have an intractable hard to reproduce bug that's you'd like to see just go away, we can arrange some sort of trial to have someone come and help capture and diagnose that bug, and see if that leads to a longer term arrangement.

Ask questions in the comments. Feel free to download the trial -- if you've used gdb, it's fairly straightfoward to try out, and it's magical to see "step back, step forward".

Or if it sounds like you might be someone who would actually benefit from acquiring a license, I can put you in touch with helpful people -- we used to focus on big clients because there was a lot of shakedown, but now it works more reliably out of the box, it's plausible for a wider spectrum of companies and people.

Work are contracting out some UI work. Basically, there's a fairly good idea of what features are needed, but are investigating the possibility of handing off the work of what different screens there are, what options on each screen, etc, etc to someone else.

It will likely be implemented through a web-browser inside, but not be interacted with as a webpage as such.

They already have one fairly promising contact, but are interested in having at least two choices. (Pro: they will probably make a decision and not dither. Con: they may go with the existing contact.)

Do I know anyone or any company who does that sort of work?

I can't vouch for what they're like to be a contractor for, but as far as I know from the inside, they're looking to get the work done, not to cut costs. And they're not specifically looking for future work, but if they find a company they get good results from, there's other projects it might make sense to do a similar thing with.

If there is anyone I can put you in touch with the relevant people.

Work trip to Germany was quite good. The travelling was easy, as was spending three days in the company of people from work (one from my office, one from one of the German offices).

I coped in Germany only speaking a few tourist phrases, although I need to get better at sometimes jumping into conversations when I know the people speak English, but don't automatically switch to English when I enter the room.

I managed to spend a couple of hours walking around the old parts of the town and looking at the river, and to go to a couple of different places for meals, since most places had some vegetarian dish, but not a very interesting one.

For the first time I had to ask myself what's normal to put on expenses. I have to keep reminding myself that my time is valuable to work, and that if it's worth sending me abroad at all, it's not worth saving a few extra pounds by being extra frugal. But I have to force myself not to think "oh no, surely that's just gratuitous". I think work clothes you don't have any other use for, and meals when you're away from home are the normal standard? And beer with meals when it's cheaper than the water?

The work itself was generally a success, we did everything we intended to, and didn't screw everything up, and I've a good idea of what we want to achieve. But it still feels like banging my head on a brick ceiling that 90% of what I say we need to do is dismissed, I think I should persist with the last 10% but I'm not sure. I'm not sure how much I'm bad at sounding like I know what I'm talking about, and how much I should better have figured out which decisions were taken several months ago but I wasn't told about.

The factory was really interesting. It's like a tutorial in best practices, eg. most of the furniture is on wheels and COULD be anywhere, but there are strips of tape on the floor showing where it SHOULD be. It's about half robots and half people. And people have to ride bikes around when they're going round the whole thing.

We couldn't agree anything so I've just resigned as of now, and am leaving immediately as of this evening! However, we parted very amicably so YAY even though EEK!

That's awkward because I'm jobhunting as of NOW but that's life. I feel good about it, and think and hope work do too.

(Now I'm looking for a job immediately, and don't know whether or not I will be able to do the contracting-from-Stoke option, I need to sort logistics on that immediately!)

OK, am off to London for Mum's graduation. Love to you all! J.

Wednesday was our work summer outing, which turned out to be very fun. We went an activities centre little sailing like at Mepal. (I unsurprisingly keep a romantic attachment to sailing)

Sometimes such things turn out to be just a bit complicated, and I think it may have been for people who'd never been sailing at all, but I was familiar enough from a day or two I spent on a school trip; not of how to sail effectively, particularly, but enough to not worry if my life jacket was sufficient, or if I could recover from a capsize.

We were half a dozen of us sailing, and about the same doing archery+climbing. There was one instructor for sailing, and six little one-person boats (toppers). Nowadays those are mass-produced plastic hulls with mast+sail, boom, keel, and rudder fitted on. They're intended for kids, but can hold an adult if you can duck under the boom.

( Read more... )

It's funny, if my fingernails are just ever so slightly too long, most of the time I don't notice, but just occasionally it jumps out at me and once I've noticed it's like fire on my skin, I want to tear the ends off right then and I have to deal with it immediately.

At home, cutting fingernails always feels like a minor chore, something I'll do but not something I'll want to do. Especially as nothing I do once every few weeks gets into a regular rhythm. However, I've kept a spare pare of clippers in my draw at work[1] and at work, it feels relaxing and cathartic to snip snip snip the ends away!

But then I remembered the Dilbert cartoon. It's something like Alice listening to clipping sounds from Wally's cubicle, and screaming "AGH! It's so annoying. Yet quiet enough I'll look neurotic if I say anything! It's driving me mad!" and the last panel is Wally telling Dilbert about his new screensaver, "Wally's 101 annoying cubicle sounds".

Is fingernail clipping so annoying? I can see why it might be in principle, but it doesn't feel it. It's very quiet -- practically not showing up against the background -- and lasting only a few minutes and only every few weeks. Work isn't so silent that noise would stand out. And we're spaced so only about one person might hear.

[1] along with emergency: toothbrush, alcohol and tissues. I've yet to stash shoes and underwear.

Our building has two flights of stairs. Any more, and using the lift would seem entirely justified. When it only had one, I never bothered. Now I do either, depending what I feel like at the time.

But oddly, I go down the lift, but almost never come up in it. Which is backwards, because the lift naturally rests at the bottom, so going down you have to wait for it to come up. I think going up feels more like cheating :) Or maybe because standing around in the lobby feels stupid, but at the top its on its own corridor.

This week has been tiring, as I've suddenly realised there's a few things at work and home I want to do before I head off for xmas.

On Tuesday I finally played Sebby's underground -- it was quite fun. I probably want to play it for a time other than the first and see what it's like when I know what I'm doing, but congratulations, I'm impressed :)

Tonight was work's Christmas dinner. Thank you Becky for giving me moral support -- and your dress was fabulous :) I hope you had a good evening. Work functions can never be amazing, but it was fun. I'm kind of pissed off -- I ended up worrying the entire evening about arranging a taxi because the people we were supposed to be sharing with weren't there anywhere, but I had to try to find them because I didn't want to leave them in the lurch, and we swapped with two other people, and didn't realise that Panther couldn't put that on broadcom's tab, so had to spend ten minutes whining about it, and then ask me to pay and reimbursed -- I just hope we're not charged for the taxi that isn't there. If I just took a taxi myself, I wouldn't care, but it seems perverse to have one provided and it then be more pain than it's worth. Still, kudos to Her than Organised it, it went very well. There was a casino and I won $6,500, but not real money :)

Tomorrow night I hope to go to Susan's, and then on Saturday to S&M's, and thence to my parent's in Worcester for visit, returning to Cambridge in December.

Have a good Christmas to everyone I won't see! I will still be on em

Perhaps running a book on more things would encourage accurate assessment?

Boss: How long do you think this'll take?
Coder: Four to two weeks.
Boss: It should only take one week!
Coder: Two-to-one says you're wrong.
Boss: Ummm... maybe you're right.

Coder: Hey, who do you think's going to break the build next?

OK, probably not. But for today, last week's bug:

1-1: My misunderstanding of the spec.
1-1: Verilog bug.
2-1: Compiler bug.
9-1: Something else.

ETA: Outcome, my error, by the compiler doing something acceptible but not what I thought. So that could have arguably paid off three of the above.

You: But those don't add up to 1!
Me: Congratulations, you've discovered economics :)

Hah, finally I'm lying outside in the fresh air and sun working. Except I'm not, I'm telling you all about it. But there's no other distractions so once I turn the browser off again I'll probably get more done than I did all morning :)