Data Format Documentation

I've posted two documents. Please take a look and suggest improvements/corrections:

File Formats - This document is intended to discuss the nitty gritty of file formats. It starts with a recap of the format published in Challenge #26 and used GDW's TRADER application for the Apple II, attempts to document the format used by Joe Fugate for the GEnie data upload, loosely specify the "sec" files found on the Internet, and define two canonical T5 data file formats, as used by the site. We'll see if they catch on.

Second Survey Data - This recaps the definition of the fields describing a world. For many of the fields this is covered in the Traveller 5 Core Rules but this document goes into more detail on some specifics hidden in the data (like sophont populations) and legacy fields.

I aim for these to be "live specifications" - that is, updated over time as new formats emerge. Right now they try to balance being rather formal specifications with being readable by non-programmers. We'll see how things go.








Ω

Image Regression Tests and Text Quality

I've started to put together a quick and dirty image regression test - an HTML page that will load a set of image pairs and generate a third image showing differences between the two so it's easy to spot changes - for example, if I intentionally moved a glyph up a few pixels (good) or introduced a bug so that gas giants stopped rendering (bad). If the images match perfectly, nothing shows up in the diff.

Check it out: http://travellermap.com/test/ImageTest.html

Call for input: what other URLs would be good to have in a corpus of tests as reference images?

While building this I compared my development machine vs. the live site, and was struck once again by how badly the text rendering is done on the server since it stands out in the image diffs. I was able to make a one line change which dramatically improves the text and glyph quality, so if you've saved off any bitmaps recently you probably want to redo them. It's embarrassing that I didn't do this earlier!

I also enabled CORS headers for image generating APIs, but that's probably not of use to anyone unless you're using WebGL to render map tiles or some such.

Ω

Stellar Data

A week or so ago I slipped out a file format page, adapting Marc Miller's article in Challenge #26 describing the first standard sector data file format. It's pretty bare bones at the moment - just what's in the article so it is preserved for posterity - but I plan to grow this over time to document other sector data file formats that have appeared over the years. Call this "Version 1.0".

(Honestly, posting it was inspired by site users wondering what the various base codes in the data meant. But everyone seems to love file format discussions...)

Then, inspired by Yet Another Thread on sector data file formats on CotI I decided to finally tackle stellar data.

• Stellar data now output by SEC.aspx API when present in source data. The following grammar is used:

// Basic:
//   system     ::= star ( w+ star )*
//
// Extended: (Malenfant's Revised Stellar Generation Rules)
//
//   system     ::= unit ( w+ companion )*
//   companion  ::= near | far
//   near       ::= unit
//   far        ::= "[" system "]"
//   unit       ::= star | pair
//   pair       ::= "(" star w+ star ")"
//
//   star       ::= type tenths w* size main?
//                | dwarf
//                | unknown
//   type       ::= "O" | "B" | "A" | "F" | "G" | "K" | "M"
//   tenths     ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
//   size       ::= "D" | "Ia" | "Ib" | "II" | "III" | "IV" | "V" | "VI" | "VII"
//   dwarf      ::= "DB" | "DA" | "DF" | "DG" | "DK" | "DM" | "D"
//   unknown    ::= "Un"
//
//   main       ::= "*"
//
//   w          ::= " "

• "Un" in stellar data is from Mendan 0221 in Challenge #46 (by Mike Mikesh). That could be a typo, but I'm interpreting it as "Unknown" (mysteries abound!). Note that the world is an Amber zone, so it's a navigational hazard at least!
  
• Like other SEC.aspx output, the data is normalized - that is, it's been parsed and then re-output. In some cases this may filter the data. You get what you pay for. In particular, although Malenfant's revised stellar generation data is now consumed by my site, only basic star data is output. Anyone have a link to Malenfant's rules?
  
• SEC.aspx output now has a sector header, similar to the GEnie data. The field lengths are NOT guaranteed to remain fixed, however. Consumers that rely on fixed width fields (rather than Regular Expression or Grammar-based parsers) should parse this header to determine the field lengths.
  

 1-25: Name
27-30: HexNbr
32-40: UWP
   42: Bases
44-68: Codes & Comments
   70: Zone
72-74: PBG
76-77: Allegiance
79-98: Stellar Data

• I tidied up some bogus stellar data: Night (Core 0839) "K8 VI M4" to "K8 VI M4 D" (presumably a truncation); and Tender Mercy (Ley 0314) "2 V A6 D" to "K2 D M6 D" (ditto); in both cases I used older GEnie data as the source
• While I was there, I fixed a couple of O/0 (oh/zero) typos: Kaskar (Vland 1933) stellar "MO V K6 D" to "M0 V K6 D", and Zdienjbats (Far Frontiers 1801) PBG "BO4" to "B04". The latter actually inhibited the world from showing up on the map - oops!

That's it for now! Ω

Browser Compatibility Changes

Testing in Apple's new Safari 3 Beta for Windows*, I found some issues with the map. They turned out to be caused by some layout hacks I had in the page code to handle the dynamic layout of the page.

The page is composed of resizing (map) and docked (controls, footer) boxes containing other resizing elements (label controls, search results). Ideally, each element should be sized by the content - i.e. the area taken up by the footer is defined by the size of the elements in the footer, and so on. The page did that using old-school HTML <table> elements, but had some issues I'm still not sure of. (Short version: a relatively sized div set to overflow: scroll, inside a relatively sized table cell makes the cell expand if the content overflows, even if the cell and div size correctly with non-overflowing content.) To overcome the issues I had to have some script that would fix the sizes of some elements when the page size changed. These hacks - often browser specific - worked in IE6, IE7, Firefox 1.5 and Firefox 2.0, Opera 9, and Safari 2. But they caused Safari 3 to stop performing layout correctly - everything looked great, but you couldn't drag the map!

So... not wanting to layer on further hacks I redid the page entirely using modern CSS, and the end results were beautiful. Worked like a charm with IE7, Firefox 1.5 and 2.0, Opera 9 and Safari 2 and 3. Unfortunately, they didn't work in IE6. After much teeth gnashing I found a workaround. (Short version: even in standards compliance mode IE6 can't correctly size elements having top and bottom or left and right defined; however, you can use CSS expressions to compute the width or height.)

(Aside: when CSS was young and fresh, Microsoft took the interpretation of "width includes margins, borders and padding" whereas everyone else took the interpretation of "width excludes margins, borders and padding", and it was the latter which eventually became codified in the CSS standards. You can tell that Microsoft was in the "web developer" mindset rather than the "web designer" mindset. When you are designing for the look of content, you want to frame items (text, images, etc) so the exclusive width makes sense. However, when you are developing a UI, you often compose and nest relatively sized boxes. The CSS notion of "width: 100%" when nesting items is absolutely useless when combined with exclusive margins, padding or border, since the element will end up offset and overlapping its parent's right edge!)

Anyway, the map it all appears to work now... but it's probably broken in some obscure browser. If the map suddenly broke for you (boxes everywhere! oh the huge manatee!) I have the retro version available here: http://www.travellermap.com/index_classic.htm - I'll try to keep it working, but I might break it by accident with future changes to underlying shared components such as styles or code. And please let me know what platform and browser you're using - I might be able to bludgeon the page into shape.

* Safari for Windows is a subtle but effective ploy by Apple to encourage Windows-using web developers who haven't shelled out for a Macintosh to do more testing on their platform. Worked for me!

[Update: January 2008]
Given that the old version works in browsers as old as IE5 and no-one has screamed and complained, the "classic" version of the page is no longer available.
Ω

Fixed cache-control headers for images

If you use IE you may have noticed that when zooming in then back out the map image tiles would re-load from the server (that is, it'd be slow). Firefox (or another Mozilla-derived browser) were zippy. I was generating the HTTP headers incorrectly which caused IE (compliant) to re-request the images each time.

FYI, the fix was in the ASP.NET code for Tile.aspx:

Response.Cache.SetCacheability( HttpCacheability.Public ); // was doing this
Response.Cache.SetExpires( DateTime.Now.AddDays( 1 ) ); // was doing this
Response.Cache.SetValidUntilExpires( true ); // wasn't doing this

I was also misinterpreting the meaning of Response.Cache.VaryByParams (which populates the HTTP "Vary:" header). I was interpreting the semantics as "re-request if these params are different" rather than correct semantics: "re-request even if these params are the same". So I removed calls to that, and everything appears happier now.

So what does that mean? In IE, try zooming all the way in to 64 pixels/parsec then out to 1/32 pixels/parsec, using the mouse wheel. Then, once all the tiles are cached, you can repeat the zoom in/zoom out without waiting for the images to re-load. Makes a great demo! Ω