Checking the noise properties of SCUBA-2 science data

Ever wondered if there was an easy way to assess the noise properties of the data going in to your SCUBA-2 maps? Well now there is. A new recipe called ASSESS_DATA_NOISE has been added to the SCUBA-2 pipeline and it performs standard quality-assurance analysis (as done by the quick-look pipeline at the telescope) to give you an idea of the noise properties of the sciences data.

Running ASSESS_DATA_NOISE

To run it, first put all the files you wish to process into a text file (called myfiles.lis in the example below). Create a recipe parameter file if necessary (myparams.ini below). Then initialize the pipeline as usual (not necessary if done previously):

% oracdr_scuba2_XXX -cwd

where XXX is the wavelength of choice. Finally, run the pipeline, adding the name of the recipe at the end of the command line:

% oracdr -loop file -files myfiles.lis -recpars myparams.ini \
  -log sf ASSESS_DATA_NOISE

The recipe displays its results in a KAPPA window, showing the focal-plane layout with the noise for each bolometer in the top left corner, a histogram of the noise in the lower left, and the focal-plane image of the noise equivalent power (NEP) in the lower right. (If you want to turn off this display, add -nodisplay to the ORAC-DR command line above.)

By default the results are calculated for each subscan (i.e., each 30-second file) separately. As described below, options exist to just use the first subscan or to calculate the noise properties for the entire time stream.

The focal-plane mosaics for each subscan are stacked to create a 3-d cube of noise as a function of time (the third axis is MJD). The corresponding NEP images are written into this file under the .more.smurf.nep extension. This file has the suffix _scinoistack.

The results are written out to a log file called log.scinoiseXXX (where XXX is the wavelength as above), which can be loaded into TOPCAT to display the number of bolometers, noise or weighted NEP as a function of time or subscan number for each subarray.

The recipe also writes out a file called log.bolonoise, which contains more results (with the results for each subarray on a separate line), plus index.noise and index.nep. From these files it will be possible to determine which files failed the QA tests, and thus can be excluded from the input list to the map maker.

Of course, the down side to removing data is that it will not be possible to achieve the intended noise level, and may reduce sensitivity to extended structure (as the time series may be broken into smaller chunks). However, it should be easy to determine whether or not using some fraction of the best data gets as close as could be expected.

Customizing

Like most recipes, a number of recipe parameters are available:

• NOISE_CONFIG - the config file you wish to use when calculating the noise. It is recommended that you use the same as the one used when making maps. If not given, the standard noise config file is used.
• NOISE_CALC - may be "quick", "each" or "full" which will use the first thirty-seconds of data (for a quick check), or for every thirty second data file, or use the entire time stream to calculate the overall noise properties. The default is "each".
• FREQRANGE - frequency range over which to calculate the noise. Default is 2-10 Hz.
• FREQLO - lower frequency at which to calculate the background. Default is 0.5 Hz.

See the SMURF documentation for the FREQRANGE and FREQLO before adjusting them (SUN/258).

This is just the first release and its usefulness and output data relies on user testing and input. Please try it out and let me know.

"Scuff" removal using NOI.BOX_SIZE

Occasionally "scuffs" are apparent in reduced SCUBA-2 images. These scuffs are a result of one or more sub-arrays entering into a higher noise state for a period of time. 

It is possible to down-weight data that have higher noise by setting NOI.BOX_SIZE in your configuration file. The length of time over which the noise is analyzed can either be specified in time samples (positive values) or seconds (negative values). 

It has been found setting NOI.BOX_SIZE=-15 (i.e. half a sub-scan) is a good value to use. As this value tends to -1 (one second) we find some of the source signal being down weighted. Higher than this -15 value and the map-maker becomes less sensitive to the higher noise states.

Left: reduction without NOI.BOX_SIZE specified. Right: reduction with NOI.BOX_SIZE=-15 specified in the config file.

The difference map: we see these scuffs are removed from the final image.

Changes to dimmconfig files

The $STARLINK_DIR/share/smurf directory contains a set of dimmconfig...lis files that define "pre-canned" sets of configuration parameters for use with MAKEMAP. Some changes have been made to these files recently, partly to tidy them up and make them more user-friendly, but also to incorporate some new parameter values that have been found to give better results in the majority of cases.

The most important changes that you need to be aware of are:

1. The dimmconfig.lis file now uses MAPTOL to specify the convergence criterion rather than CHITOL. 
2. The value of 850.FLT.FILT_EDGE_LARGESCALE provided by dimmconfig_bright_extended.lis has changed from 300 to 600 arc-seconds. The 450 value remains unchanged at 600 arc-seconds.
3. The dimmconfig_bright_extended.lis file now inherits values from  dimmconfig.lis instead of dimmconfig_bright.lis. This means that the modified values previously provided by dimmconfig_bright.lis for COM.CORR_TOL, COM.GAIN_ABSTOL, COM.GAIN_TOL, DCTHRESH and NOISECLIPHIGH are no longer inherited by dimmconfig_bright_extended.lis. Consequently the new dimmconfig_bright_extended.lis will perform more aggressive rejection of bad values than the old version.

Other less important changes are:

1. The descriptions of individual parameters that previously were in dimmconfig.lis have been moved into $SMURF_DIR/smurf_makemap.def. This is the file that defines the complete list of all legal parameters for makemap and their default values. These description are also included in an appendix in SUN/258.
2. Parameters that are mainly intended for experimental use by makemap developers have been removed entirely from dimmconfig.lis but are still defined in the smurf_makemap.def file.
3. The prologues of the main dimmconfig files have been standardised.
4. Experimental dimmconfig files have been moved into a subdirectory called "experimental" within $STARLINK_DIR/share/smurf.

Multiple masks revisited

I have today made another change to the way that multiple masks are handled, and I have modified the previous post "Using Multiple Masks" to reflect it.

Previously, multiple masks were always combined by taking the union of the source regions. Now, the intersection of the source regions may instead be used by setting the xxx.ZERO_UNION parameter to zero (i.e. false), where "xxx" is AST, FLT or COM. Note, the default is still to use to use the union.

This can be useful, for instance, if a basic SNR mask allows bright structures to develop near the edges - intersecting the SNR mask with a ZERO_LOWHITS mask will reduce the SNR mask to exclude the edge regions.

Automated removal of bad-baseline spectra from ACSIS/HARP time series

I presented the following poster at November's ADASS Conference, describing the methods I added to the ORAC-DR REDUCE_SCIENCE_NARROWLINE and REDUCE_SCIENCE_GRADIENT recipes to detect and remove `wobbly' and noisy spectra. It shows some examples of the then identified forms of interference and example results.  Since then an additional form---ringing---came to light from looking at more data, and is shown in the latest JCMT Newsletter 34. I'll discuss that in a further blog.

Go here for the full-sized PDF.

SCUBA-2 papers on arxiv

Three papers describing the performance of SCUBA-2 have been accepted for publication. They can be found on arxiv at the following links:

SCUBA-2: The 10000 pixel bolometer camera on the James Clerk Maxwell Telescope
Holland et al. 2013 MNRAS
http://arxiv.org/abs/1301.3650

SCUBA-2: iterative map-making with the Sub-Millimetre User Reduction Facility
Chapin et al 2013 MNRAS
http://arxiv.org/abs/1301.3652

SCUBA-2: on-sky calibration using submillimetre standard sources
Dempsey et al 2013 MNRAS
http://arxiv.org/abs/1301.3773