I’m still working ferociously on getting ready for the FogBugz World Tour. We won’t have the final list of cities until we can get hotel meeting rooms confirmed in each city, but it looks like the first round will hit about 20 cities in the US and Canada plus a few stops in New Zealand, Australia, and Europe. In any case, I’m looking at about six weeks on the road.

Fog Creek’s office manager Liz is planning all the logistics for the event. Today she faxed around about 80 RFPs to hotels to get meeting rooms lined up. (There’s a huge business opportunity there–getting hotel event coordinators onto the Internet). I have to buy a bunch of gear:

• a very light, bright, robust video projector
• a very portable PA system and a couple of lavalier mics
• a subnotebook that can serve as my primary desktop while I’m on the road but which won’t break my back

Is there such a thing as a video projector which connects to the PC using Cat-5 instead of video cable, so I don’t have to carry a 50 foot video cable around?

What’s a good subnotebook? Does anyone have any experience (good or bad) with the new flash-drive based notebooks?

What about the portable PA systems? Is there something self-contained that fits in the overhead bin on a plane?

Steve from Richmond, VA, wrote in to ask a couple of questions:

“With software development, is it better to get something out there with customers and then continually improve or build the best wiz-bang software and then start marketing?”

That depends. You want to avoid the Marimba Phenomenon, where you get so much publicity in the early days that everybody checks out your underwhelming offering and decides that you’re never going to have something worth looking at. (I should rename this the live.com phenomenon, in honor of Microsoft’s horrible launch of live.com. Or the Zune phenomenon.) On the other hand, small startups are unlikely to have the problem of too much attention, so most companies with a 1.0 product can certainly get real customers with their earliest usable versions and build from there.

Steve is starting a company with a software developer.

“Assuming the software developer is getting paid at a reduced rate, but concurrently with his development. If you were giving also giving him some equity in your company, would you make that equity contingent on phases of the software getting done, the entire software getting done or vesting over time.”

The standard solution is to vest over time — anywhere from four to seven years — with unvested shares being forfeited if he leaves for any reason. If the software doesn’t get done, you fire him and he loses the unvested shares — it’s not necessary to make the vesting contingent specifically on finishing the software (besides, “finishing” software would be too hard to define in a contract).

You can set it up either as normal vesting (where he gets, say, 20% of the shares every year) or reverse vesting (where he gets the shares up front, but you have the right to repurchase them for a penny, and this right evaporates by 20% every year). Reverse vesting is preferable for tax reasons, because at the time you give him the shares, they’re worth a lot less, so there’s less income and more capital gains, which are taxed at a lower rate.

The Business of Software conference coming up at the end of October is new this year, but it’s got a pretty phenomenal line-up of speakers:

• Guy Kawasaki practically invented the field of high tech evangelism working at Apple in the 1980s, and has written some of the best books about marketing I’ve read (my personal favorite is The Macintosh Way.)
• Rick Chapman wrote the encyclopedia of software marketing, The Product Marketing Handbook for Software, and the hilarious In Search of Stupidity, which makes the quite convincing argument that success in software is a matter of just not doing stupid things. (My foreword to In Search of Stupidity).
• Hugh MacLeod does those great cartoons, which is reason enough to listen to him, but he’s also the marketing strategist for the only Web 2.0 wine company.
• Tim Lister needs no introduction to this crowd. He’s the author of Peopleware, the first and last thing you have to read about managing software teams. That’s really all you need to know.
• Eric Sink, moderator of our own Business of Software forum, co-founder of SourceGear, the classic small software company success story.

Also speaking: Dan Nunan, Jennifer Aaker, Jeffrey Pfeffer, Bill Buxton, and me. Register here.

Our systems administrator Michael Gorsuch explains it: “So, yes, even though the SQL Server Mirroring technology sounds like an ideal fit at first, it is easy to see how it doesn’t really suit our needs.”

Back in 2001, I wrote: “When you’re working on a really, really good team with great programmers, everybody else’s code, frankly, is bug-infested garbage, and nobody else knows how to ship on time. When you’re a cordon bleu chef and you need fresh lavender, you grow it yourself instead of buying it in the farmers’ market, because sometimes they don’t have fresh lavender or they have old lavender which they pass off as fresh.”

We do it every summer… an open house at the Fog Creek offices in New York City.

Come on by next Thursday, July 19, 2007, between 5:00 pm and 7:00 pm to see our offices and meet the Fog Creek team including this years’ interns, a.k.a. the caribou, the software management trainees, and me. It’s always a fun chance to meet other software developers in the New York area and geek out about impedence mismatch. Bring your boss to prove how private offices, 30″ LCDs, and Aeron chairs are possible. Drink free wine. Show off your new iPhone. No RSVP necessary.

Fog Creek Software is at 535 8th Ave., on the 18th Floor.

Maxim Shemanarev: “The amazing thing is there is no rocket science! Nothing to patent! All information is publicly available and/or deducible from what we see. You only need to use a bit of your engineering intuition plus common sense. So, it goes. You can download an application with full sources in the end of the article and play with it, but not now, please. Now be patient to read rather a long story.” Wow!

Mike P. asks:

“On your recent post you mention that FogBugz is now going to be hosted and that you hired a sysadmin who converted your systems from NetBSD to Linux. Then you go on to say that you bought a Dell server, W2K3 and W2K5 SQL Server. I don’t understand. Are you or are you not runing FBOD with the LAMP stack?”

I was not very clear. We have a lot of servers. Some of them were running NetBSD, and are now running Linux. Others are running Windows Server 2003. The Linux servers tend to be used for routers and assorted network services like DNS. The Windows servers are either working as web front ends (IIS) or database back ends (MS SQL 2005).

FogBugz On Demand is, indeed, on the Windows servers. We try very hard to stick to a single hardware architecture so any server can be swapped with any other in case of problems, and parts can be moved hither and yon as needed. We weren’t really able to do this perfectly because Dell switched from SCSI to SAS hard drives when they went from the 8th generation to the 9th generation servers, but mostly we use a common platform so it’s easy to keep spare parts on hand. (In fact, one backup server even has an extra NIC in it so it can replace the router if it comes to that).

I’m happy to announce that FogBugz On Demand is now available. This is a professionally-hosted version of FogBugz 5.0, previously only available as a download.

Selling web software has always been a slightly strange aspect of the way Fog Creek operates. Since FogBugz is a web-based project management tool, why should customers have to download it and install it on their own servers?

In the past, we did it that way because we’re a small company. With just a handful of full time employees, we didn’t really have the resources to be a reliable service provider that customers could trust with their mission-critical data.

To prepare for FogBugz On Demand, we’ve done a lot of hard work over the past year.

First, we hired a professional system administrator. He has pored over every inch of our hosting infrastructure, patching and testing and improving reliability everywhere. He upgraded all the NetBSD servers to Linux, installed a bunch of new hardware, and added lots and lots of automated monitoring.

In general, we decided to use high-end components for our hosting architecture: Dell PowerEdge 2950 Servers with SCSI RAID, Windows Server 2003, and SQL Server 2005. Yep, that’s an expensive way to do it. Since FogBugz runs fine on LAMP (Linux / Apache / MySQL / PHP), we could have gotten a bunch of cheap boxes, used all free software, and saved some money in exchange for some level of headaches. Indeed, most hosted services really should be built on LAMP. In our case, though, the cost of those Microsoft licenses and those extremely reliable Dell servers can be spread out over quite a few paying customers, so for us the cost difference per customer is really inconsequential. And we’ve been running IIS and Microsoft SQL Server here for six years without data loss, so that’s what we know and trust. But to be honest, if we ever get to the point where we’re racking up 10 new servers at a time, we’ll almost certainly switch to LAMP. I’m still gonna buy Dell servers and SCSI hard drives, because frankly, the small extra cost over cheapo white boxes is well worth it in reliability.

We made changes to the FogBugz code base itself to make it work better in a multi-hosted environment. The biggest surprise was how much work it took so that every user sees things in their own time zone. We also put a lot of work into the accounting and billing system (FogBugz On Demand is $21 a month, with no commitment). We implemented a database-backed DNS system so that each On Demand customer gets their own domain (you.fogbugz.com).

The biggest change was bringing up a second data center. I can’t tell you how scary it is to be responsible for our customers’ mission-critical data, so I didn’t want to have any single point of failure, no matter how fortified it is.

Our first data center has been with Peer 1 Network in New York’s financial district. Peer 1 is a Canadian backbone provider where we’ve been since the beginning of 2003. To take advantage of their backbone, we put our second data center in their new Los Angeles facility. This new data center is pretty much an exact replica of what we have in New York.

To some extent, by using Peer 1 for our second facility we are, technically, putting all our eggs on one backbone. But it’s a pretty darn reliable backbone and an excellent company. We actually investigated a couple of other colo providers and even went so far as to build out a facility in Chicago (with an unnamed provider). But shortly before we launched, they had a six hour outage, and in the aftermath of that, we discovered that their network connectivity was inadequate and their concept of building reliable systems did not use the same definition of “reliable” as we do. So we gave up on them, shipped all the servers from Chicago to LA, and went with the tried and true Peer 1.

Rather than setting up Los Angeles as a mere backup, we decided it would be completely live. Half our customers will be hosted from Los Angeles, and half from New York. That way we know at any time that both data centers are working and set up correctly, and we don’t have to wait until a massive failure to discover the problems with the backup data center.

Copies of the database backups are maintained in both cities, and each city serves as a warm backup for the other. If the New York data center goes completely south, we’ll wait a while to make sure it’s not coming back up, and then we’ll start changing the DNS records and start bringing up our customers on the warm backup in Los Angeles. It’s not an instantaneous failover, since customers will have to wait for two things: we’ll have to decide that a data center is really gone, not just temporarily offline, and they’ll have to wait up to 15 minutes for the DNS changes to propagate. Still, this is for the once-in-a-lifetime case of an entire data center blowing up, not just for occasional outages: each data center already has incredible backbone connectivity, UPSs, backup diesel generators, and so forth (Peer 1 survived that huge blackout during the summer of 2003 while many of their competitors were winking out).

To implement this warm backup feature, I wrote a SQL mirroring application that implements transaction log shipping: basically, it does an incremental backup in one city, compresses that backup, ships it to the other city, uncompresses it, and applies it to the warm backup database. Right now, we’re log shipping twice a day, so you might lose a day of work if an entire city blew up, but in a couple of weeks, we’ll implement a system that does more continuous backups, and we expect that the warm backups will never get more than 15 minutes behind.

FogBugz On Demand has actually been in beta since April, and in fact we have been hosting FogBugz trials on line since 2000, without ever losing anyone’s data. (The first FogBugz 1.0 trial server, believe it or not, was a Thinkpad laptop with a broken screen plugged into our office T1!) So I’m pretty confident now that our little company can do a pretty good job of hosting FogBugz for you.

There’s a scene in the movie 1408 that features a stack of Joel on Software (the book) along with John Cusack and Samuel L. Jackson.

The Business of Software Wiki: “The point of this wiki is to bring under one roof as much useful, quality information as possible about the business of software, whether it’s microISVs selling desktop software, Web 2.0 sites or even the big enterprise kind of outfits.”

Hmm, gee, that phrasing is a little bit awkward.

Click:

“The point of this wiki is to bring under one roof as much high quality, useful, qualityinformation as possible about the business of software, whether it’s microISVs selling desktop software, Web 2.0 sites or even the big enterprise kind of outfits.”

That’s better!

You can edit it too.