How to loose fewer samples with NOI.BOX_SIZE

At the end of the first iteration, when the common-mode signal (COM model), low frequency drift (FLT model), and the astronomical signal (AST model) have all been removed from the data, the remaining residuals are used to estimate the variance of the high frequency noise in each bolometer (the NOI model). The reciprocal of these variances are used as weights on the second and subsequent iterations, and are applied to the time-series values that are binned to form the map.

By default, the NOI model contains a single variance value for each bolometer , and so all samples from a single bolometer get the same weight in the map. However, the NOI.BOX_SIZE parameter can be used to force each bolometer to have multiple variance estimates, with one value for each box of samples. See this post. 

Whilst this is often in an improvement over the default situation in that it allows for varying noise levels, it has two potential issues:

1. All samples in a box are given the same variance value, thus introducing the potential for steps in the weights and corresponding artifacts in the map.
2. The noise in each box is found by taking the FFT of the data and finding the average power in the 2 to 10 Hz band. Using an FFT is problematic on short time streams, as any missing values (such as caused by steps, jumps, common-mode flagging, etc) have a proportionally larger effect, causing the FFT to be less representative of the data. A requirement that the time stream should have a majority of unflagged values in order to be usable means that a lot of data is often rejected because a reliable FFT cannot be obtained.

An alternative scheme that avoids these two issues has been added to makemap, which is enabled by setting "NOI.BOX_TYPE =1" in your configuration. This option causes the noise at a sample to be set equal to the variance of the neighbouring sample values - those that are less than half a box size (i.e. NOI_BOX_SIZE / 2) away from the sample. This scheme does not involve taking an FFT and produces a continuously varying estimate of the variance and so avoids steps in the weights used to make the map. It also is minimally affected by flagged samples and so results in fewer data samples being flagged as unusable (as shown by the "NOISE" entry in the samples statistics at the end of each iteration).

Weighting different observations in makemake and skyloop

Both makemap and skyloop can be used to produce maps from multiple overlapping observations. Up to now, it has not been possible to control the weights used for each observation when forming the total co-added map. Instead, fixed weights equal to the reciprocal of each pixel variance were used. The new CHUNKWEIGHT configuration parameter can now be used to specify a weight for each observation. The weights determined from the pixel variances are multiplied by these user-specified weights, which default to unity.

The weights can be specified in two ways:

1. The CHUNKWEIGHT configuration parameter can be set to an arbitrary algebraic expression in which the variable names are the names of FITS keywords. The keywords must exist in the input data and have numerical values. The expression should be enclosed in double quotes. As an example, you could include this line in your configuration file:
   
      chunkweight = "AMSTART * WVMTAUST"

The values of the FITS keywords are read from the observation header and substituted into the expression to get the weight to use for the observation.


2. A comma-separated list of explicit numerical weights can be supplied, enclosed in parentheses. If insufficient values are given in the list, the last value is replicated as often as is necessary.
   
      chunkweight = (1.0,2.0,3.0)

Holes in heterodyne maps

HARP has only twelve operational receptors since May 20.  If one or two also suffer from interference and fail quality assurance, then there can be locations in the reduced spectral cubes where no valid data are present.  Such missing spectra (shown in blue) will often be arranged in a grid pattern as in the example below.

This can create issues if you wish to integrate flux or simply improve the cosmetic appearance.

One approach is to smooth the data but this degrades all the spectra.  Instead use KAPPA::FILLBAD. This replaces just the bad values with a smooth function derived from neighbouring values, derived by iterating to a solution of Laplace's equation.

Just using FILLBAD's default values will duly fill in the holes.  You might prefer not to smooth in the spectral axis by setting Parameter SIZE to [s,s,0] where s is the initial scale length in pixels.  The zero means do not smooth along the third axis. The scale length should have a value about half the size of the largest region of bad data to be replaced.  Since the largest bad regions apart from the cube peripheries are two pixels across, a command like this

fillbad in=holey out=filled  size="[1,1,0]"

is appropriate.  The graphic below shows the same region as before, but with the holes filled in.

FILLBAD does what it says on the tin.

There will be some smoothing, but it's not obvious.  Below is a KAPPA::CLINPLOT graphic for a part of the filled cube.  Each spectrum is a pixel in the image above. Can you identify the three spectra which were previously bad?

Inclusion of CSO Tau Fits for Determining FCFs

During the first few months of 2013 the WVM at JCMT had several periods of instability where it was unreliable for determining the value of tau. Because of this, members of the science team collaborated to produce a collection of smooth polynomial fits to the tau values from the 225-GHz tau meter at the CSO for the affected nights, which can be used to perform extinction correction in place of the WVM tau values.

In the latest version of starlink (which you can get by rsyncing from /stardev at the JAC) makemap will now first check to see if the date of the observation is one of the (somewhat sporadic) dates when the WVM was unstable. This occurs as long as the ext.tausrc parameter is set to "auto," which it is by default. If the date is one of the affected dates, makemap will look for an available fit to the CSO data in the file specified by the parameter ext.csofit, which is set up by default to refer to the collection of CSO fits produced at the JAC. If makemap cannot find a fit for an observation in the specified path it will print a warning that no fit or WVM data is available and refuse to reduce the observation, though this shouldn't ever happen in normal operation.

If you have observations taken between January 19 and May 14 of this year, using the latest version of starlink rsynced from /stardev at the JAC will help ensure that you get the best extinction data available.

The graph below shows a comparison between the FCFs derived from WVM data and those derived using the new CSO tau fits over the period of January-May 2013. The blue diamonds represent FCFs from the WVM, and the tan circles are FCFs from the CSO tau fits.

Removing linear striations in maps

You've probably seen several maps that show the faint linear striations visible in the following 850 map.

Well, in at least some cases, these striations seem to be due to bad bolometers that have a very different common-mode signal to the others. The map-maker has an algorithm that looks for and rejects such bolometers, but by default the rejection criteria are fairly  weak, so few samples are rejected. Changing the  com.corr_abstol value from its default of 0.2 to 0.8 produces the following map:

The striations are much less noticeable (both these images use the same grey scale). The down-side to this is that since more bolometer samples are rejected, there are fewer samples to form the map. For the first map, only 1.6 % of the available samples were rejected due to common-mode mis-match, leaving a total of 65.4 % available for the map. For the second map, 11.1 % were rejected due to common-mode mis-match, leaving 55.9 % available for the map. 

Going even further, the following map was created with com.corr_abstol set to 0.97 and  com.gain_fgood set to 0.5:

Here, 38.5% of samples were rejected due to common-mode mismatch, leaving only 28.6 % available for the map. You can see that the striations have been almost completely removed, but at the expense of higher noise.

Automatic FLT masking

The estimation and subtraction of low frequency noise in each bolometer (the FLT model) very often causes ringing around bright sources. This ringing can be very bad for the first few iterations but often improves if sufficient iterations are performed. Masking the FLT model consists of excluding known bright sources from the estimation of the low frequency noise, and thus reducing the ringing. The ability to do this using the "flt.xxx" set of configuration parameters has been around for while, but sadly the ability to do automatic   FLT masking has not been available - an external mask image needs to be supplied. By comparison, the AST model can be masked automatically using the ast.zero_snr and ast.zero_snrlo parameters. The corresponding parameters flt.zero_snr and flt.zero_snrlo do exist, but do not really work since the SNR values are only available once a map has been made. By then it is to late to do the FLT masking - the FLT model has already been applied and the ringing is therefore already present in the residuals.

But a new parameter called ast.skip can get round this. This new parameter defaults to zero, but if set to a positive integer it gives the number of initial iterations for which the AST model (i.e. astronomical signal) is to be skipped. For example, if ast.skip is set to 5 then the first five iterations will not include an AST model. The map will still be created at the end of each iteration, but it willnot be used to find the expected astronomical signal, and the residuals will be left unchanged. This means that all these first five iterations will start from essentially the same time series data - the initial cleaned raw data. However, what this means is that we can now create an FLT mask automatically from the SNR values in the map, and refine that mask five times. So when we get to the sixth iteration, we have a usable FLT mask and we have not  introduced any ringing into the residuals.

It's a bit like doing a separate run of five iterations to determine the FLT mask before starting again to do the main run.

In summary, to use automatic FLT masking, add the following to your config:

ast.skip=5
flt.zero_snr=5
flt.zero_snrlo=3

If you use a fixed number of iterations, you may want to increase your numiter value by the ast.skip value, so that you are doing the same number of real iterations. As usual, you can play around with the precise values of these parameters.

FLT masking speeds up convergence as shown in the following plot of normalised change in the map against iteration number - red uses the new scheme described above, green uses a fixed external FLT mask, and blue is without FLT masking.

Changes to way makemap handles control-C interupts

There have been some changes to the way that makemap behaves when control-C is pressed (or a SIGINT signal is received by the process by any other means). When the first control-C is detected, makemap will continue to complete the current iteration (but a second interupt will cause makemap to terminate immediately). It will then pause and ask the user what to do next. The options are:

1. abort immediately with an error status
2. close the application returning the current output map
3. do one more iteration to finialise the map and then close

Option 3 is important if you are using AST masking, since one final iteration should always be done without masking to ensure correct fluxes in the background regions (see the AST.ZERO_NOTLAST config parameter). If you are not using AST masking, or if you will be re-starting makemap to perform further iterations, then you should select option 2.

At some point I will add a fourth option to ignore the interupt and continue iterating to convergence.

Note if you are running makemap from within a script, then the handling of control-C interupts will probably be quite different, because the shell (or perl, python or whatever) will catch the interupt before it gets to makemap. You may get some sort of useful behaviour by finding the process ID for the makemap process itself, and sending a SIGINT to that process explicitly:

% ps aux | grep makemap

dsb      25407  0.0  0.0  43716  1952 pts/5    S    Jul03   0:00 makemap

% kill -s INT 25407

Monitoring itermaps whilst makemap is running

The main output NDF created by makemap is not closed until the last iteration has been completed. Since itermaps are usually placed in the SMURF extension of the main output NDF, this means that itermaps cannot be examined  until makemap has completed. How annoying...

But now you can get round this by using the new ITERMAPS parameter (an ADAM parameter, not a config parameter) to specify that the itermaps should be placed somewhere else. If you do this, each itermap NDF will be closed as soon as it has been written, allowing you to examine it immediately.

So in one window, you start a big makemap job running (with "itermaps=1" in the config):

% makemap in=^infiles out=out config=^conf itermaps=fred

and then in another window you can examine the itermaps as soon as they are created:

% display fred.ch00i003

If you want a list of all the available itermaps, do

% ndfecho fred

(this is a kappa command).

Re-starting makemap after an interupt or crash

makemap can take a very long time to run, and it is very annoying if a power failure, or some system glitch causes it to crash after several hours. The good news is that it is now possible to re-start it from (more or less) where it left off, so long as you follow a couple of requirements when running makemap:

1. You must include "itermap=1" or "itermap=2" in yourt config.
2. You must assign a value to the "ITERMAPS" ADAM parameter when running makemap. This new parameter specifies the file in which the itermaps are to be placed. Previously, there was no choice about where the itermaps went - they went into the SMURF extension of the main output NDF. Now, you can specify an alternative place using ITERMAPS - note, this is an ADAM parameter and so goes on the makemap command line, NOT a config parameter. The benefit of specifying an alternative  location for itermaps is that, unlike the main output NDF, the specified file will be closed properly after writing each itermap. This means the itermaps will be usable even if makemap crashes, so long as you are not really unlucky and the crash occurs whilst it is actually writing an itermap.  

If you follow the above requirements when running makemap, you will be able to re-start makemap in the event of a crash by running makemap again in exactly the same way as you did before, but with the following changes:

1. Add "initialsky=xyz" to your config, replacing "xyz" by one of the ADAM parameter names - "REF", "MASK1" or "MASK2".
2. On the command line, set the chosen ADAM parameter to the path of the last itermap created by the previous run of makemap.  
3. In the config, add "noi.calcfirst=1"

For instance:

% cat conf

...

itermap=1

...

% makemap in=^infiles out=out config=^conf itermaps=fred

..

..

oops - the computer died... OK, not to worry, I've got 67 itermaps in fred.sdf so I can re-start at iteration 68...

% makemap in=^infiles out=out config="'^conf,initialsky=ref,noi.calcfirst=1'" \

                       ref=fred.ch00i067

Note, the re-started makemap will still need to go through the labourious task of cleaning the input time series data again, before it starts iterating. 

Great results from Skyloop

This is just a quick post to show the clear benefits of using skyloop when reducing your own data (see our earlier blog or read: http://www.starlink.ac.uk/docs/sc21.htx/node51.html). Below is an image composed of several observations reduced individually using the iterative map maker. "Blooms" (bright false emission) are clearly evident at the edges of maps where high noise has made its way into the ast model.

 

Reducing the same data using skyloop helps to identify what is noise and what is true emission-particularly in the overlap regions. The benefit in this situation is clearly seen. The cost to the user is in computing power and time but the result is clearly worth it.

Makemap now gives slightly different numbers

From today, people using a cutting edge starlink rsynced from /stardev at JAC may notice small changes in the maps created by MAKEMAP compared to previous versions.This is caused by a change to the way in which the (RA,Dec) of each bolometer sample is calculated. The change was needed to support FTS-2.

The new scheme is mathematically equivalent to the old scheme, but groups the required transformations a little differently, resulting in different (not worse) numerical rounding errors. This has no effect on the majority of samples, but some samples that are very close to the edge of a pixel will be moved just that little tiny amount needed to push them over into the adjacent pixel. This results in noticeably different (not worse) noise in the final map.

The black curve in the following plot is a histogram of the differences between the two maps produced by the two versions of makemap, for a small Uranus observation (S8A only). The red curve is a histogram of Error component (i.e. sqrt(Variance) ) for the old map. Note the logarithmic Y axis scale. 

Numerically, the RMS difference between the data values in the two maps is 8.57E-5, the square root of the mean Variance value in the old image 53.0E-5, and the  square root of the mean Variance value in the new image is 52.9E-5.

So these changes are much smaller than the noise level, and so should not be a concern.

Displaying images with the same scaling

If like me you often want to produce a plot containing several images all scaled to the same limits, you may be interested in a new feature in KAPPA:DISPLAY that simplifies this. Setting the MODE parameter to "Current" will now re-use the scaling limits that were used on the previous invocation of display.  No need to copy and paste the limits any more... As an example:

% gdclear
% picdef mode=array xpic=3 ypic=1 prefix=pic
% display map1 mode=per percentiles=\[2,98\]
% picsel pic2
% display map2 mode=cur

% picsel pic3
% display map3 mode=cur

will display map1.sdf on the left with 2% of pixels set to pure black and 2% set to pure white. It will then display map2.sdf in the center and map3.sdf on the right, using the same scaling that was used with map1.sdf.

Getting hard-copy graphics from Starlink apps

Most of you will already know that the Starlink KAPPA package provides many tools for producing useful graphical output (linplot, display, contour, etc.), and for dividing up the plotting space into "pictures", each of which can hold a separate plot (picdef, picsel, etc.). Multiple plots can be stacked on top of each other, so for instance it is possible to overlay a contour plot on an image, or to overlay multiple line-plots. However, how easy it is to do this depends on the graphics device you are using.  Using "xwindows" is straight-forward, but relies on taking screen-shots for hardcopy, which for some purposes is not good enough. Using PostScript devices produces high quality hard-copy, but is awkward because each KAPPA command produces a separate PostScript file. The user then needs to combine all the individual PostScript files into a single PostScript file.

The current development Starlink system introduces some new features that make using PostScript much easier. If you use kappa:gdnames to list all available graphics devices, you will see some new entries:

% kappa % gdnames The following graphics devices are available. The first column holds the GNS names, and the second the equivalent PGPLOT names (in parentheses). Either form can be used... png (/PNG) Portable Network Graphics file ... ... aps_p (/AVPS) Accumulating EPS, monochrome, portrait aps_l (/APS) Accumulating EPS, monochrome, landscape apscol_p (/AVCPS) Accumulating EPS, color, portrait apscol_l (/ACPS) Accumulating EPS, color, landscape ... ...

These new "accumulating" devices produce a single Encapsulated PostScript file (pgplot.ps by default) by accumulating the PostScript output automatically from subsequent Starlink graphics commands - no need for you to merge them all together yourself into one file. For example:

% gdset /acps % okular pgplot.ps & % gdclear % picdef mode=array xpic=2 ypic=1 prefix=pic nooutline % display $KAPPA_DIR/m31 \           style="'colour=black,width=2,colour(ticks,border)=green'" \           mode=perc percentiles=\[2,98\] % contour $KAPPA_DIR/m31 style="'colour=red'" noclear mode=free \           heights=\[9000,11000,13000\] % picsel pic2 % linplot $KAPPA_DIR/m31'(,169)' \           style="'colour=black,width=2,colour(curve)=red'" % linplot $KAPPA_DIR/m31'(,180)' style="'+colour=blue'" noclear

produces a single PostScript file (pgplot.ps) containing the following two pictures:

But what is that "okular pgplot.ps &" command in there? Okular (see http://okular.kde.org/) is a PDF and PostScript viewer supplied with the KDE desktop in Fedora, etc. It automatically re-draw the display if the contents of the displayed file changes on disk. Combined this with these accumulating Postscript devices provides a scheme that is similar to the use of a traditional persistent X-window device, in that the display is automatically updated after each graphics operation completes. If you do not use KDE, your OS may provide an alternative PostScript viewer.

To use these new features, you will need to rsync /stardev from Hilo.

Fixing instabilities in the WVM

The JCMT's Water Vapour Monitor (WVM) was taken off the telescope between the 20130326 and 20130408 to improve instabilities with its performance. The WVM is now back on the telescope as of 20130409 and initial results are looking good.

Tau estimates during this time have been taken from the CSO's WVM. The CSO values are used by the map maker automatically when reducing SCUBA-2 data. If you have any concerns or questions regarding data collected during this time please contact your Friend of Project. Your Friend of Project can be identified using the following link:

http://www.jach.hawaii.edu/JCMT/allocations/

New OMP features for projects.

The OMP has recently been updated to display several additional pieces of information on the project pages.

Tau graph

The first thing you will notice is a new graph of tau over the course of the night. Previously, the pages had such a plot that was generated each time the page was loaded. This took a while, and the plot had automatically generated limits on the y-axis which made comparing graphs between two night effectively impossible.

The new graph is made with consistent x- and y-limits that allows for easy comparison between nights. The various weather grades are also delineated, and the value of tau from the CSO WVM is plotted as well.

The new plot of tau vs. time. Click an any picture for a larger view.

The thicker gray horizontal lines running across the graph mark the weather grade boundaries, and the shaded area shows the night hours in Hawaii. The time in UTC is plotted along the bottom axis, with the time in HST along the top.

Pie chart

The pie chart of time spent in each grade.

There is also a pie chart that sums up the amount of time spent in each weather grade throughout the night. The grades are color-coded from green to red to give a quick visual assessment of how good a given night was (green good; red bad). Usually a given night will fall predominately within just one or two grades.


ACSIS standards table

For nights on which data was taken using ACSIS, there will be an HTML table in text form like the example below with some information relating to the ACSIS standards observed.

Obs #	Time	Integ. Int.	Peak Int.
16	19:36:32	251.33	8.82
19	19:50:31	4775.56	6.88
27	20:59:30	4157.76	5.96
36	22:22:36	3520.79	5.08

SCUBA-2 calibrations table

Similarly, for nights where SCUBA-2 was used there will be an HTML table like the example below listing the FCFs for each observation of a calibration object.

20120822			FCFasec				FCFpeak
Obs #	Time (UT)	Source	850µm	err	450µm	err	850µm	err	450µm	err
8	05:41:27	CRL2688	2.41	0.01	4.64	0.03	572.2	1.5	542.9	5.7
37	10:24:23	CRL2688	2.30	0.01	4.66	0.02	513.5	1.2	466.0	3.7
68	17:26:48	CRL618	2.23	0.01	4.28	0.04	487.8	1.4	436.8	4.8

Along with the table on SCUBA-2 nights there will three additional graphs, which will be detailed below. Each of these graphs has two sub-plots, the top one being the 450 micron data and the bottom one being the 850 micron.

NEPs vs observation number graph

This graph shows the min, max, and mean for each of the eight subarrays that make up SCUBA-2, plotted by observation number. The mean is the colored line, and the shaded areas are the filled-in areas between the min and max. By following the lines you can tell which arrays changed significantly, and by watching the shaded areas you can get a feel for the spread in the NEPs. The vertical scale is fixed, and is the same as the scale for the following plot.

NEPs vs time graph

This graph, like the previous one, shows the NEPs, but plots them against time rather than observation number. Each of the eight subarrays is present once again (with the same colors as in the preceding plot). The time (in UTC) is marked along the bottom of the plot, and the observations are marked by number along the top, with vertical lines that descend to help mark when each particular observation began.

NEFDs vs time graph

The final plot is a plot of the NEFDs vs. time. The number of points depends on the night, this particular night only had a few. Like the two previous plots, the vertical axis is fixed to make comparisons between nights easier.

Hopefully these new features will prove useful to users of the OMP, and additional updates or improvements may be forthcoming in the future. Feedback on the new features is welcome.

A new feature in CONFIGMELD

The CONFIGMELD tool in SMURF can be used to compare two different sets of MAKEMAP configuration parameters. But sometimes you are only interested in a single parameter value, and wading through the entire configuration is bit off-putting. So now you can specify a single parameter name when running CONFIGMELD, and it will just display the value of that one parameter as plain text in your terminal window (that is, no visual comparison tool is used). Also, if you are only interested in the parameter value in a single config, then you can just leave CONFIG2 unspecified.


% configmeld ^/star/share/smurf/dimmconfig_bright_extended.lis  param=flagslow
   WAVEBAND - The waveband to display - 450 or 850 /850/ > 
   Showing 850 um values

   flagslow = 30

% configmeld map1 map2 param=flt.filt_edge_largescale

  Showing 450 um values

  flt.filt_edge_largescale = 600 (config1)

  flt.filt_edge_largescale = 400 (config2)

As always, to use this feature you will need to update your starlink to the current development version. Linux users can simple rsync Starlink from JAC following the instructions at http://starlink.jach.hawaii.edu/starlink/rsyncStarlink

Straight lines in ZERO_SNRLO masks

Today I've fixed a bug that affected masks created using the "xxx.ZERO_SNRLO" configuration parameter ("xxx" = AST, FLT or COM). The effect of this bug was to reduce the size of the source area in the mask by "slicing" sections off the mask. Very often (but not always), this slicing would leave an unnaturally straight edge in the final mask.  As an example, here are two masked maps of G34 - the first map is affected by the bug, and the second is not. To get the bug fix, you will need to rsync Starlink from JACH.

Note, my initial fix for this bug was itself buggy. So if you rsynced prior to 11:20 GMT on  15th March, you will have only a partial fix for this problem, and you should rsync again.

In addition, the amount of smoothing used to create ZERO_SNRLO masks has been reduced slightly, so expect masks that are little more "crinkly" round the edges than before.

Getting Pipelines and Archives info by email

You should now be able to see a "Subscribe via email" link at the top right of the "Pipelines and Archives" blog page. Click on this and follow the instructions if you want to get email alerts of any new blog posts - you'll only get at most one email per day (and no emails at all on days when no posts are made).

Checking SCUBA-2 calibrator data

A new PICARD recipe has been added called SCUBA2_CHECK_CAL which can be used to assess the quality of the calibration by calculating flux conversion factors (FCFs) and estimating the beam size to compare with standard values.

It can be run on any processed observations of standard sources (except, obviously, focus data). The input files must be uncalibrated, either the output from running makemap by hand or from the pipeline and then processed with the PICARD recipe UNCALIBRATE_SCUBA2_DATA.

Run it as any other PICARD recipe:

% picard -log sf -recpars myparams.ini SCUBA2_CHECK_CAL calfiles*.sdf

The recipe trims the image, optionally removing a background before it fits the source to estimate the beam.

The next step is calculating the FCFs which are reported to the screen. The map is then calibrated using either a standard FCF or one of those just calculated (controlled by a recipe parameter). The noise in the map is calculated from this calibrated image.

The recipe writes a log file called log.checkcal which contains various parameters. The values of interest include the total flux and uncertainty derived from the uncalibrated (input) map, the noise in the calibrated version of that map, the FCFs and associated uncertainties and the beam FWHM. In the future there will also be an estimate of the error beam. The log file can be read into Topcat so values can be plotted as a function of time, elevation or any other parameter (if there are enough of them).

The total flux is measured using aperture photometry with an aperture radius of 30 arcsec. The aperture size may be adjusted using a recipe parameter, but be aware that comparison with the standard ARCSEC FCF will require correcting for the different aperture. See the SCUBA-2 calibration paper for further details.

By default the recipe can only estimate FCFs from known calibrators. However, it also be used to estimate the FCF from non-standard sources if the flux is known to the user. Recipe parameters exist to allow this to be specified (on a per-source basis if desired). Note that the sources should still be unresolved, point sources.

The FCF calculations can be compared with the standard values (given in the calibration paper) to determine whether or not there may be particular issues with the calibration on a given night. The BEAM FCF is more sensitive to focus than the ARCSEC value, so if the former is out of spec, while the latter is not, it could indicate that the telescope was not as well focussed as it could have been. If both are far out of spec (indicated by red text in the recipe output) then it may suggest that a non-standard FCF is in order. It is also important to look at the trend in the FCFs over the night, and not just a single value.

The main recipe parameters of interest are:

• APERTURE_RADIUS - radius of photometry aperture in arcsec. Default is 30.
• USEFCF - a flag to denote whether the derived FCF should be used to calibrate the data. Default is 0 (use the standard FCF).
• FLUX_850 - total flux of a point source at 850 um. The corresponding 450 um parameter is FLUX_450. Fluxes may be specified for multiple sources by adding the source name (upper case with spaces removed) to the parameter - e.g., FLUX_850.MYSOURCE1, FLUX_850.MYSOURCE2 etc.
• REMOVE_BACKGROUND - a flag to denote whether or not a background should be removed. Default is 0 (do not remove a background). If true, then there are other parameters available to control the background fitting and removal process.

The full list of recipe parameters is described in SUN/265 and will be available in the upcoming Starlink release.

Comparing MAKEMAP configurations

You have two SCUBA-2 maps, created by makemap from the same observation at different times, and they look different. Not unnaturally you want to know why. The first thing you may suspect is that they were made with different configuration parameter values. You can do this by using the hislist command in kappa to look at the history information in each map, but it's a little painful. The good news is that the new configmeld command in smurf now makes this a lot easier. It will read the configuration parameter values from the History information of the two maps and use a visual comparison tool such as "meld" to display them side-by-side, sorted alphabetically, and with any differences highlighted. For instance

% configmeld map1.sdf map2.sdf

may produce a display similar to the one below...

Instead of reading the configurations to be displayed from the history of an NDF, either (or both) configuration may instead be specified "as normal" - that is, as you would do when running makemap, so if you have a set of configuration parameter values in text file myconfig.lis, you could do something like:


% configmeld ^myconfig.lis map2.sdf

to compare the configuration specified by myconfig.lis with the configuration used to create the map in map2.sdf. As for all Starlink commands, you can use the findme command to see the full list of options provided by configmeld. Type:

% findme configmeld

which should open a web page, and click on the link for configmeld.


Notes:

1. If your system does not have the meld command, the first available command in the following list will be used instead: opendiff, diffmerge, kdiff3, tkdiff, diffuse.
2. Configuration values that have "no value" by default (usually shown as "") are not included in the history information stored in the map created by makemap, and so will not appear in the list shown by configmeld. The sharp ones among you may therefore notice a small lie above - the screenshot shown earlier was actually made by comparing a "normal" config file with the configuration from an NDF, not from two NDFs as suggested by the example command. Sorry! That's why one side (the normal config file) has "" values and the other side (read from an NDF) does not. 
3. configmeld is in fact a Python script designed to run with python 2.7 or later. If you have problems using it, it may be because your Python installation is too old.
4. The configmeld script just uses the configecho command from KAPPA to list the two sets of configuration parameters to two separate temporary text file and then uses meld (or your selected tool) to display the two files side-by-side.

Checking the noise properties of SCUBA-2 science data

Ever wondered if there was an easy way to assess the noise properties of the data going in to your SCUBA-2 maps? Well now there is. A new recipe called ASSESS_DATA_NOISE has been added to the SCUBA-2 pipeline and it performs standard quality-assurance analysis (as done by the quick-look pipeline at the telescope) to give you an idea of the noise properties of the sciences data.

Running ASSESS_DATA_NOISE

To run it, first put all the files you wish to process into a text file (called myfiles.lis in the example below). Create a recipe parameter file if necessary (myparams.ini below). Then initialize the pipeline as usual (not necessary if done previously):

% oracdr_scuba2_XXX -cwd

where XXX is the wavelength of choice. Finally, run the pipeline, adding the name of the recipe at the end of the command line:

% oracdr -loop file -files myfiles.lis -recpars myparams.ini \
  -log sf ASSESS_DATA_NOISE

The recipe displays its results in a KAPPA window, showing the focal-plane layout with the noise for each bolometer in the top left corner, a histogram of the noise in the lower left, and the focal-plane image of the noise equivalent power (NEP) in the lower right. (If you want to turn off this display, add -nodisplay to the ORAC-DR command line above.)

By default the results are calculated for each subscan (i.e., each 30-second file) separately. As described below, options exist to just use the first subscan or to calculate the noise properties for the entire time stream.

The focal-plane mosaics for each subscan are stacked to create a 3-d cube of noise as a function of time (the third axis is MJD). The corresponding NEP images are written into this file under the .more.smurf.nep extension. This file has the suffix _scinoistack.

The results are written out to a log file called log.scinoiseXXX (where XXX is the wavelength as above), which can be loaded into TOPCAT to display the number of bolometers, noise or weighted NEP as a function of time or subscan number for each subarray.

The recipe also writes out a file called log.bolonoise, which contains more results (with the results for each subarray on a separate line), plus index.noise and index.nep. From these files it will be possible to determine which files failed the QA tests, and thus can be excluded from the input list to the map maker.

Of course, the down side to removing data is that it will not be possible to achieve the intended noise level, and may reduce sensitivity to extended structure (as the time series may be broken into smaller chunks). However, it should be easy to determine whether or not using some fraction of the best data gets as close as could be expected.

Customizing

Like most recipes, a number of recipe parameters are available:

• NOISE_CONFIG - the config file you wish to use when calculating the noise. It is recommended that you use the same as the one used when making maps. If not given, the standard noise config file is used.
• NOISE_CALC - may be "quick", "each" or "full" which will use the first thirty-seconds of data (for a quick check), or for every thirty second data file, or use the entire time stream to calculate the overall noise properties. The default is "each".
• FREQRANGE - frequency range over which to calculate the noise. Default is 2-10 Hz.
• FREQLO - lower frequency at which to calculate the background. Default is 0.5 Hz.

See the SMURF documentation for the FREQRANGE and FREQLO before adjusting them (SUN/258).

This is just the first release and its usefulness and output data relies on user testing and input. Please try it out and let me know.

"Scuff" removal using NOI.BOX_SIZE

Occasionally "scuffs" are apparent in reduced SCUBA-2 images. These scuffs are a result of one or more sub-arrays entering into a higher noise state for a period of time. 

It is possible to down-weight data that have higher noise by setting NOI.BOX_SIZE in your configuration file. The length of time over which the noise is analyzed can either be specified in time samples (positive values) or seconds (negative values). 

It has been found setting NOI.BOX_SIZE=-15 (i.e. half a sub-scan) is a good value to use. As this value tends to -1 (one second) we find some of the source signal being down weighted. Higher than this -15 value and the map-maker becomes less sensitive to the higher noise states.

Left: reduction without NOI.BOX_SIZE specified. Right: reduction with NOI.BOX_SIZE=-15 specified in the config file.

The difference map: we see these scuffs are removed from the final image.

Changes to dimmconfig files

The $STARLINK_DIR/share/smurf directory contains a set of dimmconfig...lis files that define "pre-canned" sets of configuration parameters for use with MAKEMAP. Some changes have been made to these files recently, partly to tidy them up and make them more user-friendly, but also to incorporate some new parameter values that have been found to give better results in the majority of cases.

The most important changes that you need to be aware of are:

1. The dimmconfig.lis file now uses MAPTOL to specify the convergence criterion rather than CHITOL. 
2. The value of 850.FLT.FILT_EDGE_LARGESCALE provided by dimmconfig_bright_extended.lis has changed from 300 to 600 arc-seconds. The 450 value remains unchanged at 600 arc-seconds.
3. The dimmconfig_bright_extended.lis file now inherits values from  dimmconfig.lis instead of dimmconfig_bright.lis. This means that the modified values previously provided by dimmconfig_bright.lis for COM.CORR_TOL, COM.GAIN_ABSTOL, COM.GAIN_TOL, DCTHRESH and NOISECLIPHIGH are no longer inherited by dimmconfig_bright_extended.lis. Consequently the new dimmconfig_bright_extended.lis will perform more aggressive rejection of bad values than the old version.

Other less important changes are:

1. The descriptions of individual parameters that previously were in dimmconfig.lis have been moved into $SMURF_DIR/smurf_makemap.def. This is the file that defines the complete list of all legal parameters for makemap and their default values. These description are also included in an appendix in SUN/258.
2. Parameters that are mainly intended for experimental use by makemap developers have been removed entirely from dimmconfig.lis but are still defined in the smurf_makemap.def file.
3. The prologues of the main dimmconfig files have been standardised.
4. Experimental dimmconfig files have been moved into a subdirectory called "experimental" within $STARLINK_DIR/share/smurf.

Multiple masks revisited

I have today made another change to the way that multiple masks are handled, and I have modified the previous post "Using Multiple Masks" to reflect it.

Previously, multiple masks were always combined by taking the union of the source regions. Now, the intersection of the source regions may instead be used by setting the xxx.ZERO_UNION parameter to zero (i.e. false), where "xxx" is AST, FLT or COM. Note, the default is still to use to use the union.

This can be useful, for instance, if a basic SNR mask allows bright structures to develop near the edges - intersecting the SNR mask with a ZERO_LOWHITS mask will reduce the SNR mask to exclude the edge regions.

Automated removal of bad-baseline spectra from ACSIS/HARP time series

I presented the following poster at November's ADASS Conference, describing the methods I added to the ORAC-DR REDUCE_SCIENCE_NARROWLINE and REDUCE_SCIENCE_GRADIENT recipes to detect and remove `wobbly' and noisy spectra. It shows some examples of the then identified forms of interference and example results.  Since then an additional form---ringing---came to light from looking at more data, and is shown in the latest JCMT Newsletter 34. I'll discuss that in a further blog.

Go here for the full-sized PDF.

SCUBA-2 papers on arxiv

Three papers describing the performance of SCUBA-2 have been accepted for publication. They can be found on arxiv at the following links:

SCUBA-2: The 10000 pixel bolometer camera on the James Clerk Maxwell Telescope
Holland et al. 2013 MNRAS
http://arxiv.org/abs/1301.3650

SCUBA-2: iterative map-making with the Sub-Millimetre User Reduction Facility
Chapin et al 2013 MNRAS
http://arxiv.org/abs/1301.3652

SCUBA-2: on-sky calibration using submillimetre standard sources
Dempsey et al 2013 MNRAS
http://arxiv.org/abs/1301.3773

MAKEMAP default method

When running MAKEMAP, you indicate the method it is to use by setting the METHOD parameter on the command line. In the past, if you failed to do this you would be prompted for a value for METHOD, with the suggested default of "REBIN". I've changed this so that METHOD will now default to "ITERATE" and you will no longer be prompted if it is not specified on the command line.

So note, if for any reason you want to use the "REBIN" method, you will now need to indicate this by adding "METHOD=REBIN" to the command line.