I think Taras might be interested in this stuff too; I recall he always wanted to do infeasible path elimination in Dehydra, and a solver like this would make it a lot easier.

Another good thing about having a real solver is that it easily takes care of another detail I papered over in my work on this so far. In the real Firefox code, NS_SUCCEEDED is not a function, but rather:

#define NS_SUCCEEDED(_nsresult) (NS_LIKELY(!((_nsresult) & 0×80000000)))

So the code sequence actually looks more like:

rv1 = rv
t1 = rv1 < 0
t2 = !t1
t3 = __builtin_expect(t2)
if (t3)

I just coded up the constraints directly from the C semantics (and assuming __builtin_expect is a no-op) and MiniZinc was able to solve it correctly. Pretty cool! Here’s my code:

var int: rv;
var int: rv1;
var int: t1;
var int: t2;
var int: t3;

predicate succ(var int: rv) = rv >= 0;
predicate fail(var int: rv) = rv = 0;

predicate true_if_not(var int: lhs, var int: rhs) =
lhs = 1 /\ c_false(rhs) \/ lhs = 0 /\ c_true(rhs);

constraint
rv1 = rv /\
true_if_ltz(t1, rv1) /\
true_if_not(t2, t1) /\
t3 = t2 /\
c_false(t3) /\
fail(rv);

solve satisfy;

http://research.microsoft.com/projects/z3/
http://yices.csl.sri.com/
http://www.cs.nyu.edu/acsys/cvc3/

CVC3 has the advantage of being open source; not sure about the license details for the others. I haven’t completely understood your example, but if I recall our nifty automated deduction class correctly, I think the prover should be able to handle your formulas mostly just using uninterpreted functions.

How about trying a constraint programming system?

I haven’t done any work with them before, although I’ve read a little. But I managed to hack up something fairly quickly that seems to give a solution to one of your examples. The code is in MiniZinc, something I hadn’t heard of before today!

var int: rv;
var int: tmp;
var int: iftemp;

predicate successful(var int: returnvalue) =
returnvalue = 0;

predicate ns_succeeded(var int: value, var int: check_result) =
(successful(value) /\ check_result = 1) \/
(not successful(value) /\ check_result = 0);

constraint
tmp = rv /\
ns_succeeded(tmp, iftemp) /\
iftemp = 0 /\
not successful(rv);

solve satisfy;

To run it, I downloaded the G12 MiniZinc distribution, then I run it with mzn2fzn outparms.mzn && flatzinc outparms.fzn. The output is below.

iftemp = 0;

rv = -2097152;

tmp = -2097152;

But if I change the last not successful(rv) to just successful(rv), then no solutions are found. This means that rv must be not successful. i.e. a failure code.

In terms of automating this, you should be able to do the same work in an embeddable constraint programming library, like Gecode. Gecode has a FlatZinc interpreter, so you can move from handcoded MiniZinc examples to automation step by step.

Cheers
Rich

That isn’t enough for the case analysis side of your problem, of course. Sounds fun.