I wrote tests and all I got was this lousy non-determinism (In python)

[jack]

You know that weird feeling where your tests sometimes pass and sometimes don't, and you eventually realise they're not deterministic? But it took a while to notice because you kept changing things to debug the failing tests and only slowly realised that every "whether it succeeded or not" change didn't follow changing the code?

In this case, there were some failing tests and I was trying to debug some of them, and the result was the same every time, but only when I ran a failing test by itself and it passed did I realise that the tests weren't actually independent. They weren't actually non-deterministic in that the same combination of tests always had the same result, but I hadn't realised what was going on.

And in fact, I'd not validated the initial state of some tests enough, or I would have noticed that what was going wrong was not what the test *did* but what it started with.

I was doing something like, there was some code that loaded a module which contained data for the game -- initial room layout, rules for how-objects-interact, etc. And I didn't *intend* to change that module. Because I'm used to C or C++ header files, I'd forgotten that could be possible. But when I created a room based on the initial data, I copied it without remembering to make sure I was actually *copying* all the relevant sub-objects. And then when you move stuff around the room, that (apparently) moved stuff around in the original copy in the initialisation data module.

And then some other test fails because everything has moved around.

Once I realised, I tested a workaround using deepcopy, but I need to check the one or two places where I need a real copy and implement one there instead.

Writing a game makes me think about copying objects a lot more than any other sort of programming I've done.

Comments

[mtbc]

Could switch to a purely functional language. (-: This afternoon a colleague tracked down an integration test issue: a Java test was leaving something behind in the database that it turned out confused an apparently unrelated Python test. Too much non-determinism about.

[jack]

I have considered it, I feel like more experience with a functional language is a useful experience for most programmers to have and I don't have much. I do prefer more-functional functions where possible when working in other languages. But when I tried to look at it once or twice before, it felt like I was just introducing all the same mutability the hard way, or that I knew how I wanted the code to be optimised in practice, but there was no way of expressing that.

Is there any language you'd recommend as being worthwhile to work with (in terms of being mature enough to be useful, or easy to learn if you're not used to functional languages, or anything else)?

[andrewducker]

I'd go with every test recreating everything from scratch.

And also as much as possible being immutable.

[jack]

"I'd go with every test recreating everything from scratch."

How specifically? It seems that the example tests I've seen do the same thing I thought was natural: call "import modulename" at the top of the file or maybe in a setUp function, and don't sprinkle "reload modulename" for every module used before every test. Nor that the test suite seem to provide that as standard (I guess?) Is that in fact necessary/recommended? Or just because I apparently was doing something hinky? I didn't REALISE I was doing something hinky, that was the problem.

[andrewducker]

I'm not familliar with the language you're using, but NUnit (for instance) has the ability to run each test class in its own appdomain, so that even statics aren't shared.

If you're using static (module-level?) variables, and that's not possible, then you're in trouble, certainly.

[jack]

Sorry, I should have been a bit more precise. I'm using Python, and the problems seem specific to the way python does stuff, probably that I'm just unfamiliar with it, but possibly that it's just unhelpful.

Something like:

import helpers
import unittest

class TestBlah(unittest.TestCase)
    def test_blah_1(self):
        blah = helpers.create_blah_object()
        self.assertEquals(blah.foo,"foo")
        blah.foo = "baz"
        self.assertEquals(blah.foo,"foo")
    def test_blah_2(self):
        blah = helpers.create_blah_object()
        self.assertEquals(blah.foo,"foo") # Fails!
        blah.foo = "baz"
        self.assertEquals(blah.foo,"foo")

Because create_blah_object accidentally returned an object from helpers, rather than a new object that was a copy of it, which was then modified, as the content of the helpers module just exists in the normal namespace and can be modified.

It might well be better to run the script multiple times running only one test each time, but I hadn't realised that level of caution would be necessary. I'm not sure if that's a standard option for running python tests or not. I didn't INTEND to use any module/global/static variables, but I hadn't realised the apparently obvious way of doing things did implicitly.

[andrewducker]

I don't know much about Python, but I think I've got this straight:

You've got a "helpers" class which returns an object, but it's not a _new_ object each time, it's a single object which gets returned every time (i.e. "create_blah_object" should be renamed "_return_instance_of_blah_object").

Which means that changing it causes those changes to break things.

In that case I'd either actually create a new Blah every time that method is called, or if that's not how it's supposed to run in production, I'd isolate the tests from each other, possibly by running each test separately from the script (although that will incur Python startup overhead).

Writing code so it's easily testable is _hard_. It's the main reason we use Dependency Injection for our C# code at work.

[jack]

"In that case I'd either actually create a new Blah every time that method is called"

Yes, that's clearly what *should* happen, and now I know it's what was causing the other tests to fail I changed it so it works. But I hadn't realised "other tests failing for no reason" could be caused by this mistake, so it took some time to realise that it was the problem.

"Writing code so it's easily testable is _hard_. It's the main reason we use Dependency Injection for our C# code at work."

It is indeed, although I feel like I've got a lot better at it.

I'm not sure this was a testing-mistake as much as a mistake-mistake which happened to show up in testing. Is there way dependency injection would have helped in this case?

[andrewducker]

Finally getting back to this (Don't have DW access from work, and was busy most evenings, sorry!)

The advantage of Dependency Injection, when it comes to tests, is that it tends to push you towards a stateless design - or, at least, one where state is immutable by default, so you know when you have to work around it.

Although, in this case, your design should have been doing the right thing, and only didn't because of an error.

[jack]

I've mixed opinions of dependency injection. It seems like, "this class should talk to an instance of a class with such-and-such interface" is much better than "this class always talks to this particular instance of such-and-such class and there's no way to ever change that".

But that still couples the class tightly to the *interface* of the other class, and it's better if as much as possible is exposed in ways that any code could instantiate an instance of the first class and use its functionality without needing a specific other class to connect to it.

Dependency Injection

[andrewducker]

If you've got "Car" object and the "Repair" method on it needs to call a "Mechanic" object which can then "Fix" it, then your Car has a dependency on something which has a "Fix" method. Using an interface to specify that helps, I'd have thought, as now multiple different classes can be used, in different situations. But not using DI - or, indeed, not using interfaces, doesn't mean that you don't need the mechanic object to implement that "Fix" method, because your Car object is calling it. I can't see how you can get around that.

If your only objection is that the object which creates the Car needs to know what kind of Mechanic to pass in to the Car, then I'm a bit confused - because that's now how I'm used to using Dependency Injection.

We're using Castle Windsor (in C#). Where I can specify something like:

var utilitiesAssembly = AllTypes.FromAssemblyContaining(typeof(ConfigurationProvider)).IncludeNonPublicTypes();

container.Register(utilitiesAssembly.Where(e => e.Name.EndsWith("Provider")).WithService.FirstInterface());

And that will go off and find my MathsProvider, my TextProvider, my DogProvider, and my CthulhuProvider classes, and register them as the default provider for their first interface.

And I can set up my ChthuluProvider (which, obviously, is my main class, from which all other things are controlled) to either have a "IDogProvider" property, or take a "IDogProvider" in its constructor.

Then I can say var myCthulhu = container.Resolve()

and it will resolve the ICthulhuProvider to CthulhuProvider, notice that that needs an IDogProvider, resolve that to a DogProvider, notice that that needs an IMathsProvider, resolve that to MathsProvider, and so on, until the entire chain is resolved.

And none of those classes know anything about dependency injection. Cthulhu just knows he needs a dog for his terrible plans, and so gets one. And the dog needs maths for its strange angles, so it gets that.

And now, when I'm writing unit tests I can use a mocking utility (like RhinoMocks) to create alternate objects which respond in specific ways when they're called. And Cthulhu has no idea that his Dog is now actually a cardboard cutout designed to test what happens if things get really fractal.

Re: Dependency Injection

[jack]

Re: the second half of your post, OK, thank you for introducing me to that. I am not following your example, but I think I see the sort of thing it's doing and why it's valuable.

Re: Dependency Injection

[jack]

Re the first half of your post. I'm not sure I chose the best example here, and I'm finding it hard to follow the mechanic/car/fix example, it's too much of a metaphor and too far from an example I'd expect to see in actual code. (This is my bugbear about many object oriented tutorials too -- using examples of real world objects, not things you mgiht actually use in a computer program :))

Can I try a slightly different example and see if it makes more sense.

Suppose you have a "PersonalisedMarketingEmail" class with a "send" function that sends an email to a particular customer. It maybe needs access to a database class (to get "badly mangled version of customer's first name" and "offensive guess at customer pronouns" etc) and an EmailSender class (which actually talks to the OS or deals with SMTP or whatever).

One way is that the "send" function or the "PersonalisedMarketingEmail" class instantiates those other classes itself. This exhibits all the problems that DI fixes.

Another possibility is that whatever calls "PersonalisedMarketingEmail::send" provides a pointer to instances of those classes. This is a lot better because if the caller wants to provide a slightly different version of them, it can do so, and PersonalisedMarketingEmail doesn't need to know anything about it.

This is what I think of as dependency injection, although it sounds like from a C# background it encompasses something broader?

But my point is that it would be even better if PersonalisedMarketingEmail didn't have a "send" function, it had a "construct" function that returned the text of the email (and maybe some header fields). And the calling function then called EmailSender::send on that email text.

And (possibly, this is more arguable) "PersonalisedMarketingEmail::construct" accepted a struct containing some relevant info about a customer that could have come from a database but could have come from elsewhere.

This has lots of advantages (and some costs). It means you don't need to bake in the assumption that it sends an email, if you want to show the text on screen or print it to a printer or scan it for combination of phrases that shouldn't be used together, you can use the same class in the same way. If you want to send an email, but batch up several and do that later, you can. If you change the name or interface to "EmailSender" you need to change the calling function but not the PersonalisedMarketingEmail class. If you want to use a different email sending class with a send function with a slightly different name or signature, you don't need to do anything particular. All of those make tests easier, but they also make the rest of the programming easier.

I think I think in those terms because I come from a C++ background where mocking is really fucking tricky. But also because, it makes the object more useful in many ways, not only for testing. I'd describe it as "glue logic is hard to test and should be moved as far up the call stack as possible." But I suspect, that's a general principle which is useful, but often it's not possible to apply, and accepting a DI/mocking solution as necessary is the best trade-off (?)

But I don't know enough about the languages-with-working-reflection side of it to be sure, does that make sense to you?

Re: Dependency Injection

[andrewducker]

One way is that the "send" function or the "PersonalisedMarketingEmail" class instantiates those other classes itself. This exhibits all the problems that DI fixes.

Another possibility is that whatever calls "PersonalisedMarketingEmail::send" provides a pointer to instances of those classes. This is a lot better because if the caller wants to provide a slightly different version of them, it can do so, and PersonalisedMarketingEmail doesn't need to know anything about it.

Those are two options. The third is that the dependency is injected at instantiation time. Which is what my example above was trying to lay out.

That when you create a PersonalisedMarketingEmail object you do so using the DI framework, and it works out that it's dependent on EmailSender and then passes that into the constructor.

I don't speak Python, but with a little Googling and some guesswork I reckon that you'd have something like:
class PersonalisedMarketingEmail:
def __init__(self, EmailSender)

And then you'd tell the DI to give you a PersonalisedMarketingEmail and it would magic one up, complete with EmailSender.

Which seems to be something like what Spring Python does:
http://springpython.webfactional.com/1.1.0/reference/html/objects.html
(see 2.2.1)

Mocking in C# is fairly simple with RhinoMocks.
Fairly good example of how simple it is here:
https://www.hibernatingrhinos.com/oss/rhino-mocks

Re: Dependency Injection

[jack]

OK, I think I'm starting to see.

It sounds like the point is, in that approach, you always know "the" version of the EmailSender interface which is appropriate? And the centralised framework instantiates one when it's needed?

But it implicitly don't expect that the code that instantiates/calls a PersonalisedMarketingEmail object usually gets any benefit of being able to pass in a different EmailSender object, or of accessing PersonalisedMarketingEmail in any different other way?

I infer that in practice, you find that the centralised framework dependency injection model works better for most classes? I am trying to figure out whether that's different from my experience because of language differences, or because it solves problems I'm used to living with or used to not encountering, or because the benefits I imagine of a more pure-function approach are usually inconvenient or not applicable.

[ewx]

You've probably discovered a bug: create_blah_object is misnamed, in that it doesn't (always) create an object.

[jack]

Uh, yes, after I spent all evening figuring out the problem, I constructed the illustrative example so the problem was super-obvious. If it had been that obvious originally hopefully I would not have had to spend time figuring it out.