Starbugs

The night was clear, we would not be playing Bomberman in the VLT control room that night. Clear skies and a sub-arcsecond seeing meant we would have a full batch of data to process every hour or so until the next morning. Once the calibrations had finished, the telescope operator launched the first observation. I re-compiled the whole processing software once more, just to be sure we had not forgotten anything, ran a series of unit tests for good measure, and waited in front of my screen for the first incoming set of frames to appear on the local disk.

First batch of sixty frames was completed after exactly sixty minutes. As the machine started doing its number-crunching, everybody in the room turned to me, waiting for the first processed image to come out. It took a good fifteen minutes for all algorithms to run through the set: calibrate all frames, remove the infrared sky, take into account bad and crazy pixels that have been hit with cosmic rays during the observation, register all frames to a common position, and finally stack them to a single image. The final result appeared on the screen above me and I could see smiles all around. It seemed the results were up to what my customers were expecting.

Now we had a clear image of a set of bright object against a dark background. In order to assess how much infrared light is emitted by each object, it needs to be calibrated. Somewhere on the image is a standard: a star with precisely known photometry in the wavelength we had been observing. Compute how many photons were received in this image from this star and you can deduce the magnitudes for all other objects present on the same frame.

I checked once more the final frame position on the sky and then launched the photometry calibration routine. The standard star was found and identified by name, its photometry computed by integrating all received light in a small surrounding radius, and then all objects in the frame were suddenly known by magnitude rather than number of photons. Perfect score! With a sigh of relief, I finally pushed myself away from the desk and reached for some water. The memory routines had done their job, we did not crash in flight by lack of RAM this time. Eleven more hours to go and then we could all go to sleep.

Next incoming data batch was processed just fine. Another image emerged. And then another one. It seemed everything was working perfectly fine.

Around midnight, something weird happened: the result image was correctly processed but photometry calibration failed because it found no standard star in the frame.

The star database we had was pretty simple: a simple text file containing named columns: first the star name, then its position on the sky as Right Ascension and Declination (a couple of angles), and then its magnitude at various wavelengths. Something like:

# Name | Ra         |  Dec      | Sp |  J     |    H   |   K
AS01-0 | 00 55 09.9 |  00 43 13 | -- | 10.716 | 10.507 | 10.470
AS03-0 | 01 04 21.6 |  04 13 39 | -- | 12.606 | 12.729 | 12.827
AS04-1 | 01 54 43.4 |  00 43 59 | -- | 12.371 | 12.033 | 11.962
AS05-0 | 02 30 16.4 |  05 15 52 | -- | 13.232 | 13.314 | 13.381
AS05-1 | 02 30 18.6 |  05 16 42 | -- | 14.350 | 13.663 | 13.507
AS07-0 | 02 57 21.2 |  00 18 39 | -- | 11.105 | 10.977 | 10.946
AS10-0 | 04 52 58.9 | -00 14 41 | -- | 11.349 | 11.281 | 11.259
AS13-1 | 05 57 10.4 |  00 01 38 | -- | 12.201 | 11.781 | 11.648
AS13-1 | 05 57 09.5 |  00 01 50 | -- | 12.521 | 12.101 | 11.970
AS13-3 | 05 57 08.0 |  00 00 07 | -- | 13.345 | 12.964 | 12.812
AS15-0 | 06 40 34.3 |  09 19 13 | -- | 10.874 | 10.669 | 12.628
AS15-1 | 06 40 36.2 |  09 18 60 | -- | 12.656 | 11.980 | 11.792
AS15-2 | 06 40 37.9 |  09 18 41 | -- | 13.711 | 12.927 | 12.719
AS15-3 | 06 40 37.9 |  09 18 19 | -- | 14.320 | 13.667 | 13.415
AS16-0 | 07 24 15.3 | -00 32 50 | -- | 14.159 | 14.111 | 13.305
AS16-1 | 07 24 14.3 | -00 33 05 | -- | 13.761 | 13.638 | 13.606
AS16-2 | 07 24 15.4 | -00 32 49 | -- | 11.411 | 11.428 | 11.445
AS16-3 | 07 24 17.2 | -00 32 27 | -- | 13.891 | 13.855 | 13.818
AS16-4 | 07 24 17.5 | -00 33 07 | -- | 11.402 | 11.106 | 11.043

J, H, K are infrared bands corresponding to a relatively narrow wavelength range.

Something went wrong in the star-loading routine, so I loaded the whole set into memory once more and dumped it back to a text file to plot it. The results were not particularly obvious:

Somebody in the room came up to the screen and asked what we were looking at. I said: "these are the positions of all known infrared standards we have. For some reasons we cannot find tonight's star in here."

Looking at it again, I found our star. It was not in the right position. It should have been below the x axis but had shifted symmetrically above it. Looking at the data set again, the Declination was indeed negative: something like -00 14 41, but it was plotted on the wrong side of the x axis.

And then it dawned on me: the star was plotted at +00 14 41 instead of -00 14 41.

How do you read numeric data in C? Using scanf(). When you scanf() for "-00", what do you think ends up in memory? Zero. Positive zero, since it is technically the same as negative zero. Except the angle has now been flipped around the x axis.

Right: plotting a denser set of stars revealed a clear white patch for Declinations between zero and minus one. I had just forgotten to take into account the first character as a sign since scanf() does not make any difference between "00" and "-00". Once I corrected the database-loading line, everything fell into place and photometry computations could take place as expected.

Interestingly enough, it seems the same bug hit a large number of GPS devices over the past years. The German C't magazine told the story a few years back about somebody who planned a bike tour around Bordeaux and ended up with intermediate points in the middle of the ocean. Bordeaux is located around longitude zero (Greenwhich), so you do have data points located at an angle that starts with -00. In effect, you could see all points correctly plotted on the map except for the ones located between zero and minus one degree, which flipped over the other side of the meridian. As soon as I saw the map I knew exactly what had happened.

At least the guy was clever enough not to bike into high waters. It could have been worse: though probably related to time manipulation errors rather than angles, you may want to read how F22 Raptors spontaneously rebooted upon crossing the international date line:

F22 Raptor gets zapped by international date line

There are some assumptions you should not make about handling time in software. Some of them are presented in this blog article:

Falsehoods programmers believe about time

Time and angles can be tricky scalars.