In test-http-dns-fails, the test performs two HTTP requests and makes sure that:

1. The response callback is not called.
2. The error event is emitted.

Here's the relevant code:

function httpreq(count) {
  if (1 < count) return;

  var req = http.request({
    host: 'not-a-real-domain-name.nobody-would-register-this-as-a-tld',
    port: 80,
    path: '/',
    method: 'GET'
  }, function(res) {
    resDespiteError = true;
  });

  req.on('error', function(e) {
    console.log(e.message);
    assert.strictEqual(e.code, 'ENOTFOUND');
    hadError++;
    httpreq(count + 1);
  });

  req.end();
}

httpreq(0);


process.on('exit', function() {
  assert.equal(false, resDespiteError);
  assert.equal(2, hadError);
});

The problem with this test is that DNS services providers such as level3 answer queries for non-existing hostnames with the IP of their search portal. Currently, the machine that runs our Jenkin's SmartOS node is setup to use this DNS service, and thus the test fails. It doesn't fail all the time because this node is also configured to use another DNS service that does not reply with redirects. If level3's DNS server is commented out from /etc/resolv.conf, the test works as expected.

We could change the configuration of the Jenkin's SmartOS node, but it will still fail if we move the Jenkins node to another machine with the same DNS config issues.

It seems these days I want to solve every testing issue with LD_PRELOAD, but it could be a good solution to test DNS failures properly. We could override getaddrinfo using LD_PRELOAD to make sure the dns lookup fails when we invoke the test. This method doesn't work on Windows, but we might be able to do something equivalent with SetWindowsHookEx.

What do you think?