Inclusion of CSO Tau Fits for Determining FCFs

During the first few months of 2013 the WVM at JCMT had several periods of instability where it was unreliable for determining the value of tau. Because of this, members of the science team collaborated to produce a collection of smooth polynomial fits to the tau values from the 225-GHz tau meter at the CSO for the affected nights, which can be used to perform extinction correction in place of the WVM tau values.

In the latest version of starlink (which you can get by rsyncing from /stardev at the JAC) makemap will now first check to see if the date of the observation is one of the (somewhat sporadic) dates when the WVM was unstable. This occurs as long as the ext.tausrc parameter is set to "auto," which it is by default. If the date is one of the affected dates, makemap will look for an available fit to the CSO data in the file specified by the parameter ext.csofit, which is set up by default to refer to the collection of CSO fits produced at the JAC. If makemap cannot find a fit for an observation in the specified path it will print a warning that no fit or WVM data is available and refuse to reduce the observation, though this shouldn't ever happen in normal operation.

If you have observations taken between January 19 and May 14 of this year, using the latest version of starlink rsynced from /stardev at the JAC will help ensure that you get the best extinction data available.

The graph below shows a comparison between the FCFs derived from WVM data and those derived using the new CSO tau fits over the period of January-May 2013. The blue diamonds represent FCFs from the WVM, and the tan circles are FCFs from the CSO tau fits.

Removing linear striations in maps

You've probably seen several maps that show the faint linear striations visible in the following 850 map.

Well, in at least some cases, these striations seem to be due to bad bolometers that have a very different common-mode signal to the others. The map-maker has an algorithm that looks for and rejects such bolometers, but by default the rejection criteria are fairly  weak, so few samples are rejected. Changing the  com.corr_abstol value from its default of 0.2 to 0.8 produces the following map:

The striations are much less noticeable (both these images use the same grey scale). The down-side to this is that since more bolometer samples are rejected, there are fewer samples to form the map. For the first map, only 1.6 % of the available samples were rejected due to common-mode mis-match, leaving a total of 65.4 % available for the map. For the second map, 11.1 % were rejected due to common-mode mis-match, leaving 55.9 % available for the map. 

Going even further, the following map was created with com.corr_abstol set to 0.97 and  com.gain_fgood set to 0.5:

Here, 38.5% of samples were rejected due to common-mode mismatch, leaving only 28.6 % available for the map. You can see that the striations have been almost completely removed, but at the expense of higher noise.

Automatic FLT masking

The estimation and subtraction of low frequency noise in each bolometer (the FLT model) very often causes ringing around bright sources. This ringing can be very bad for the first few iterations but often improves if sufficient iterations are performed. Masking the FLT model consists of excluding known bright sources from the estimation of the low frequency noise, and thus reducing the ringing. The ability to do this using the "flt.xxx" set of configuration parameters has been around for while, but sadly the ability to do automatic   FLT masking has not been available - an external mask image needs to be supplied. By comparison, the AST model can be masked automatically using the ast.zero_snr and ast.zero_snrlo parameters. The corresponding parameters flt.zero_snr and flt.zero_snrlo do exist, but do not really work since the SNR values are only available once a map has been made. By then it is to late to do the FLT masking - the FLT model has already been applied and the ringing is therefore already present in the residuals.

But a new parameter called ast.skip can get round this. This new parameter defaults to zero, but if set to a positive integer it gives the number of initial iterations for which the AST model (i.e. astronomical signal) is to be skipped. For example, if ast.skip is set to 5 then the first five iterations will not include an AST model. The map will still be created at the end of each iteration, but it willnot be used to find the expected astronomical signal, and the residuals will be left unchanged. This means that all these first five iterations will start from essentially the same time series data - the initial cleaned raw data. However, what this means is that we can now create an FLT mask automatically from the SNR values in the map, and refine that mask five times. So when we get to the sixth iteration, we have a usable FLT mask and we have not  introduced any ringing into the residuals.

It's a bit like doing a separate run of five iterations to determine the FLT mask before starting again to do the main run.

In summary, to use automatic FLT masking, add the following to your config:

ast.skip=5
flt.zero_snr=5
flt.zero_snrlo=3

If you use a fixed number of iterations, you may want to increase your numiter value by the ast.skip value, so that you are doing the same number of real iterations. As usual, you can play around with the precise values of these parameters.

FLT masking speeds up convergence as shown in the following plot of normalised change in the map against iteration number - red uses the new scheme described above, green uses a fixed external FLT mask, and blue is without FLT masking.

Changes to way makemap handles control-C interupts

There have been some changes to the way that makemap behaves when control-C is pressed (or a SIGINT signal is received by the process by any other means). When the first control-C is detected, makemap will continue to complete the current iteration (but a second interupt will cause makemap to terminate immediately). It will then pause and ask the user what to do next. The options are:

1. abort immediately with an error status
2. close the application returning the current output map
3. do one more iteration to finialise the map and then close

Option 3 is important if you are using AST masking, since one final iteration should always be done without masking to ensure correct fluxes in the background regions (see the AST.ZERO_NOTLAST config parameter). If you are not using AST masking, or if you will be re-starting makemap to perform further iterations, then you should select option 2.

At some point I will add a fourth option to ignore the interupt and continue iterating to convergence.

Note if you are running makemap from within a script, then the handling of control-C interupts will probably be quite different, because the shell (or perl, python or whatever) will catch the interupt before it gets to makemap. You may get some sort of useful behaviour by finding the process ID for the makemap process itself, and sending a SIGINT to that process explicitly:

% ps aux | grep makemap

dsb      25407  0.0  0.0  43716  1952 pts/5    S    Jul03   0:00 makemap

% kill -s INT 25407

Monitoring itermaps whilst makemap is running

The main output NDF created by makemap is not closed until the last iteration has been completed. Since itermaps are usually placed in the SMURF extension of the main output NDF, this means that itermaps cannot be examined  until makemap has completed. How annoying...

But now you can get round this by using the new ITERMAPS parameter (an ADAM parameter, not a config parameter) to specify that the itermaps should be placed somewhere else. If you do this, each itermap NDF will be closed as soon as it has been written, allowing you to examine it immediately.

So in one window, you start a big makemap job running (with "itermaps=1" in the config):

% makemap in=^infiles out=out config=^conf itermaps=fred

and then in another window you can examine the itermaps as soon as they are created:

% display fred.ch00i003

If you want a list of all the available itermaps, do

% ndfecho fred

(this is a kappa command).

Re-starting makemap after an interupt or crash

makemap can take a very long time to run, and it is very annoying if a power failure, or some system glitch causes it to crash after several hours. The good news is that it is now possible to re-start it from (more or less) where it left off, so long as you follow a couple of requirements when running makemap:

1. You must include "itermap=1" or "itermap=2" in yourt config.
2. You must assign a value to the "ITERMAPS" ADAM parameter when running makemap. This new parameter specifies the file in which the itermaps are to be placed. Previously, there was no choice about where the itermaps went - they went into the SMURF extension of the main output NDF. Now, you can specify an alternative place using ITERMAPS - note, this is an ADAM parameter and so goes on the makemap command line, NOT a config parameter. The benefit of specifying an alternative  location for itermaps is that, unlike the main output NDF, the specified file will be closed properly after writing each itermap. This means the itermaps will be usable even if makemap crashes, so long as you are not really unlucky and the crash occurs whilst it is actually writing an itermap.  

If you follow the above requirements when running makemap, you will be able to re-start makemap in the event of a crash by running makemap again in exactly the same way as you did before, but with the following changes:

1. Add "initialsky=xyz" to your config, replacing "xyz" by one of the ADAM parameter names - "REF", "MASK1" or "MASK2".
2. On the command line, set the chosen ADAM parameter to the path of the last itermap created by the previous run of makemap.  
3. In the config, add "noi.calcfirst=1"

For instance:

% cat conf

...

itermap=1

...

% makemap in=^infiles out=out config=^conf itermaps=fred

..

..

oops - the computer died... OK, not to worry, I've got 67 itermaps in fred.sdf so I can re-start at iteration 68...

% makemap in=^infiles out=out config="'^conf,initialsky=ref,noi.calcfirst=1'" \

                       ref=fred.ch00i067

Note, the re-started makemap will still need to go through the labourious task of cleaning the input time series data again, before it starts iterating.