Calibrating SCUBA-2 data in surface brightness units

In most cases the default calibration for SCUBA-2 data processed by the ORAC-DR pipeline is mJy beam^-1. The exception is the recipe for extended sources, REDUCE_SCAN_EXTENDED_SOURCES, which calibrates data in mJy arcsec^-2.

Unfortunately there was an error in an earlier version of this recipe which meant that the FCF was applied incorrectly. The corrected method is available now with an update of ORAC-DR (either from github or via rsync from JAC). If you have data processed with this recipe (either by running it yourself, or downloading processed products from CADC) then re-calibrating the data is easy: simply divide by the pixel area using KAPPA cdiv.

There is a new PICARD recipe for easy calibration of maps produced by running makemap by hand. CALIBRATE_SCUBA2_DATA allows data to be calibrated in in per-beam and surface brightness units. With no parameters, this recipe will calibrate data in mJy beam^-1. For surface brightness calibration, set the recipe parameter USEFCF to 1 and FCF_CALTYPE to ARCSEC, and the recipe will then use the default ARCSEC FCF for the wavelength of the given data.

The recipe can also convert the calibration from one type to another. If your data are already calibrated in mJy beam^-1, they can be given to CALIBRATE_SCUBA2_DATA with the FCF_CALTYPE recipe parameter above, and the recipe will create a new file (with suffix _cal) with units of mJy arcsec^-2. The value and units of the FCF are written into the FITS header of the calibrated file.

The companion recipe, UNCALIBRATE_SCUBA2_DATA, will undo the current calibration, reverting the units to pW in the output file (which has a suffix of _uncal).

Using ORAC-DR or PICARD to perform the (un)calibration is preferred to simply multiplying your data by the FCF as they also set the units correctly for the output files(s), and write the value of the FCF used into the FITS header of the file.

However, there is one note to highlight: the recommended way to calibrate data (either from raw or when changing from per beam to per square-arcsec) is to calibrate the individual observations first, and then coadd those (re)calibrated files. Calibrating or re-calibrating coadds will fail because the coadding step was recently updated to remove FITS header entries that differ between the input files. These usually include the UTDATE which is used by the ORAC-DR calibration system. A future upgrade will provide a workaround though the recommendation to calibrate individual observations stands.

CHECK_RMS: comparing SCUBA-2 map noise with the ITC

There's a new SCUBA-2 analysis tool in town, and its aim is to help observers get a feel for whether their maps are reaching the intended sensitivity levels. It comes in two forms:

• PICARD recipe: SCUBA2_CHECK_RMS
• Offline pipeline recipe REDUCE_SCAN_CHECKRMS

At its core, the "CHECK_RMS" analysis estimates the RMS and NEFD for a given observation to compare with values from the SCUBA-2 integration time calculator (ITC). To obtain access to these new recipes, you must be using /stardev in Hilo or update your own version using rsync (see http://starlink.jach.hawaii.edu/starlink/rsyncStarlink).

PICARD recipe: SCUBA2_CHECK_RMS

As usual, the PICARD recipe is designed to be run on processed data. To run it, type:

% picard -log sf -nodisplay SCUBA2_CHECK_RMS myfiles*.sdf

in the directory containing the files to be analyzed. You may notice a warning (cyan text) about missing NEP data: that doesn't affect the main analysis and can safely be ignored. However, if run in Hilo, the recipe can query the archive of log files generated by the QL pipelines for NEP data. These NEP data are used to determine an effective noise and NEFD from the timeseries noise calculations, which can then be compared with the other values.

The recipe can be given calibrated or uncalibrated maps: the default FCF will be applied to uncalibrated data. Note that these maps must be made from a single complete observation, either by hand with makemap or the offline pipeline. Single observations must be used because the recipe relies on FITS header information that will be incorrect for coadded files. Note that maps made by the SUMMIT pipeline should not be given to this recipe; the SUMMIT pipeline will soon start recording the same CHECK_RMS data itself.

The input images are cropped to a circle of radius 90 arcsec from which the median noise and NEFD (the values RMS_MAP and NEFD_MAP in the log file) are calculated. The mean exposure time is estimated over the same map area (TEXP). It's important to note that the RMS_MAP is derived from the variance component in the map, and not from the map itself. This means that the results should be largely independent of the chosen makemap config parameters, and are more likely to reflect changes in the number of samples that go into the map rather than any differences in spatial structure introduced by applying different high-pass filters in the map-maker.

The map size and scan type are read from the FITS header and used to define the observation type for the ITC. The RMS corresponding to this observation type is calculated from the SCUBA-2 ITC (RMS_ITC); the integration time is used to convert that to a corresponding NEFD (NEFD_ITC). However, it is important to note that if you have chosen a non-standard map size for a PONG then this recipe will be unable to calculate values from the ITC, rendering it somewhat less useful.

If the NEP data are available (in Hilo only, and for dates after 20120515), the average NEP for a given observation is calculated from the archived QL pipeline log file (log.nep) corresponding to the date and wavelength of the input map. The FCF (either from the file or the standard value) and mean transmission over the observation is used to convert this to a zenith NEFD (NEFD_NEP). The ITC scaled elapsed time is used to convert that to an RMS noise (RMS_NEP).

The results are written to a log file called log.checkrms, the format of which is described below.

The following recipe parameters are supported:

• KEEPFILES - if set to 1, the _crop files created during processing will be retained on disk, otherwise all intermediate files will be deleted.
• MAP_RADIUS: radius (in arcsec) of the region used for calculating the map noise and NEFD. Default is 90 arcsec. Note that poor results can be derived if the map radius is too small and it may be more useful to use a larger radius for larger maps.
• STATS_ESTIMATOR: Statistical estimator for the NEFD and RMS calculated from the input map. May be mean or median - default is median.

Offline pipeline: REDUCE_SCAN_CHECKRMS

An alternative to the PICARD recipe, this recipe performs all the necessary steps on the raw data to do a complete self-consistent analysis for each observation, from calculating the timeseries noise to processing the data and comparing results with the ITC. The steps performed are:

• Calculate timeseries noise from the entire observation;
• Determine various values from the FITS headers (elapsed time, average airmass, elevation, optical depth, transmission, date etc);
• Make a map using the given config file (default if not specified);
• Calculate the noise (from the variance component), NEFD and exposure time within a 3-arcmin diameter circle at the map centre;
• Use the mapping parameters and elapsed time to derive the expected RMS and NEFD from the ITC;
• Write results to log file.

Since a custom makemap config file can be given (see below), it is possible to use this recipe as your main data reduction recipe (if the standard processing is sufficient), to avoid re-reducing data. Note that the recipe does not yet do any coadding, so currently output maps will have to be combined with MOSAIC_JCMT_IMAGES.

To use it, run it like any other pipeline recipe. Initialize the pipeline in the usual way:

% oracdr_scuba2_XXX -cwd

Add the names of the relevant files to a text file (say, myfiles.lis), and run the pipeline with:

% oracdr -log sf -nodisplay -loop file -files myfiles.lis REDUCE_SCAN_CHECKRMS

All of the recipe parameters supported by the default REDUCE_SCAN recipe can be given to this recipe, the most important of which is likely to be MAKEMAP_CONFIG for specifying an alternative config type for makemap. If not given, the default config file (dimmconfig.lis) is used. Note that the same config file will be used for all data so be sure to use an appropriate config.

Log file format

The log file, called log.checkrms, produced by each method contain the following entries for each map (observation):

• UT - UT date including day fraction
• Source - object name, upper case with spaces removed
• Obs - observation number
• FILTER - filter (wavelength)
• telapsed - elapsed time of observation (sec)
• texp - mean exposure time, derived from EXP_TIME NDF component (sec)
• trans - mean line-of-sight transmission
• nep_av - mean NEP for current observation (W/sqrt(Hz))
• nep_av_err - uncertainty in nep_av (W/sqrt(Hz))
• rms_nep - RMS derived from nefd_nep (mJy/beam)
• nefd_nep - NEFD derived from nep_av and scaled elapsed time (mJy sqrt(sec))
• rms_map - RMS noise in map, obtained from median of error component (mJy/beam)
• nefd_map - NEFD derived from combination of variance and exposure time images (mJy sqrt(sec))
• rms_itc - RMS noise estimated by SCUBA-2 integration time calculator (ITC) (mJy/beam)
• nefd_itc - NEFD derived from rms_itc and scaled elapsed time (mJy sqrt(sec))
• itc_obstype - observation type used by the ITC to derived RMS
• rms_ratio - ratio of rms_map to rms_itc
• El - mean elevation in degrees
• CSO - mean zenith optical depth at 225 GHz, derived from WVM
• Tau - mean zenith optical depth at the current wavelength (given by FILTER above)
• Radius - radius in arcsec of image used in analysis
• pixscale - pixel scale in arcsec
• f - ITC f parameter
• project - project ID

Note that ITC-derived values will be undefined if the PONG map size is non-standard.

So what should I look for?

The log file contains all the results of the analysis. It's worth comparing RMS_ITC with the number you determined at the time your proposal was written. It's important to be aware that the PICARD and ORAC-DR recipes estimate values from the ITC using the actual observed airmass/opacity, rather than an average obtained from the source declination. This will undoubtedly lead to differences in the ITC-derived parameters.

Perhaps the most important number is the rms_ratio which is the ratio of the RMS_MAP to the RMS_ITC. Ideally this value should be 1, though in practice that's rarely the case. Sometimes it's down to the region used for calculating the RMS being too small (in this case, increaseIf the results are very far from the expected values then contact your Friend of the Project to see if it's possible to work out why.

SUMMIT pipeline

While not yet released, the SUMMIT pipeline will also perform its own CHECK_RMS analysis and will write a corresponding log.checkrms with the same format as for the other methods.

A few things to keep in mind when this is released:

• The NEP and derived values will be undefined (NaN) in the log file, as the SUMMIT pipeline does not have access to the QL log file data.
• An entry is written each time a new image is created, which may yield multiple entries for a given observation. However, each entry should be self-consistent.

This blog entry will be updated with the relevant documentation when the SUMMIT pipeline starts to produce these data.

Inclusion of CSO Tau Fits for Determining FCFs

During the first few months of 2013 the WVM at JCMT had several periods of instability where it was unreliable for determining the value of tau. Because of this, members of the science team collaborated to produce a collection of smooth polynomial fits to the tau values from the 225-GHz tau meter at the CSO for the affected nights, which can be used to perform extinction correction in place of the WVM tau values.

In the latest version of starlink (which you can get by rsyncing from /stardev at the JAC) makemap will now first check to see if the date of the observation is one of the (somewhat sporadic) dates when the WVM was unstable. This occurs as long as the ext.tausrc parameter is set to "auto," which it is by default. If the date is one of the affected dates, makemap will look for an available fit to the CSO data in the file specified by the parameter ext.csofit, which is set up by default to refer to the collection of CSO fits produced at the JAC. If makemap cannot find a fit for an observation in the specified path it will print a warning that no fit or WVM data is available and refuse to reduce the observation, though this shouldn't ever happen in normal operation.

If you have observations taken between January 19 and May 14 of this year, using the latest version of starlink rsynced from /stardev at the JAC will help ensure that you get the best extinction data available.

The graph below shows a comparison between the FCFs derived from WVM data and those derived using the new CSO tau fits over the period of January-May 2013. The blue diamonds represent FCFs from the WVM, and the tan circles are FCFs from the CSO tau fits.

Checking the noise properties of SCUBA-2 science data

Ever wondered if there was an easy way to assess the noise properties of the data going in to your SCUBA-2 maps? Well now there is. A new recipe called ASSESS_DATA_NOISE has been added to the SCUBA-2 pipeline and it performs standard quality-assurance analysis (as done by the quick-look pipeline at the telescope) to give you an idea of the noise properties of the sciences data.

Running ASSESS_DATA_NOISE

To run it, first put all the files you wish to process into a text file (called myfiles.lis in the example below). Create a recipe parameter file if necessary (myparams.ini below). Then initialize the pipeline as usual (not necessary if done previously):

% oracdr_scuba2_XXX -cwd

where XXX is the wavelength of choice. Finally, run the pipeline, adding the name of the recipe at the end of the command line:

% oracdr -loop file -files myfiles.lis -recpars myparams.ini \
  -log sf ASSESS_DATA_NOISE

The recipe displays its results in a KAPPA window, showing the focal-plane layout with the noise for each bolometer in the top left corner, a histogram of the noise in the lower left, and the focal-plane image of the noise equivalent power (NEP) in the lower right. (If you want to turn off this display, add -nodisplay to the ORAC-DR command line above.)

By default the results are calculated for each subscan (i.e., each 30-second file) separately. As described below, options exist to just use the first subscan or to calculate the noise properties for the entire time stream.

The focal-plane mosaics for each subscan are stacked to create a 3-d cube of noise as a function of time (the third axis is MJD). The corresponding NEP images are written into this file under the .more.smurf.nep extension. This file has the suffix _scinoistack.

The results are written out to a log file called log.scinoiseXXX (where XXX is the wavelength as above), which can be loaded into TOPCAT to display the number of bolometers, noise or weighted NEP as a function of time or subscan number for each subarray.

The recipe also writes out a file called log.bolonoise, which contains more results (with the results for each subarray on a separate line), plus index.noise and index.nep. From these files it will be possible to determine which files failed the QA tests, and thus can be excluded from the input list to the map maker.

Of course, the down side to removing data is that it will not be possible to achieve the intended noise level, and may reduce sensitivity to extended structure (as the time series may be broken into smaller chunks). However, it should be easy to determine whether or not using some fraction of the best data gets as close as could be expected.

Customizing

Like most recipes, a number of recipe parameters are available:

• NOISE_CONFIG - the config file you wish to use when calculating the noise. It is recommended that you use the same as the one used when making maps. If not given, the standard noise config file is used.
• NOISE_CALC - may be "quick", "each" or "full" which will use the first thirty-seconds of data (for a quick check), or for every thirty second data file, or use the entire time stream to calculate the overall noise properties. The default is "each".
• FREQRANGE - frequency range over which to calculate the noise. Default is 2-10 Hz.
• FREQLO - lower frequency at which to calculate the background. Default is 0.5 Hz.

See the SMURF documentation for the FREQRANGE and FREQLO before adjusting them (SUN/258).

This is just the first release and its usefulness and output data relies on user testing and input. Please try it out and let me know.

Automated removal of bad-baseline spectra from ACSIS/HARP time series

I presented the following poster at November's ADASS Conference, describing the methods I added to the ORAC-DR REDUCE_SCIENCE_NARROWLINE and REDUCE_SCIENCE_GRADIENT recipes to detect and remove `wobbly' and noisy spectra. It shows some examples of the then identified forms of interference and example results.  Since then an additional form---ringing---came to light from looking at more data, and is shown in the latest JCMT Newsletter 34. I'll discuss that in a further blog.

Go here for the full-sized PDF.

Processing CLS data with ORAC-DR

A dedicated recipe has been written to process data from the Cosmology Legacy Survey using a "jack-knife" method of estimating and removing residual large-scale noise. The recipe is based on a script written by Ed for S2SRO data (and used to produce the figures in the SMURF paper). In this article I will describe how to use the recipe, what steps it performs and what parameters may be used to control the processing. In the future, this technique may be made more widely available for applying to other data sets (not just blank field ones).

How do I use it?

To use this recipe, initialize the pipeline, create a text file with the names of all the files to process and a file with recipe parameters if necessary. Then add the recipe name REDUCE_CLS to the usual command line:

% oracdr -loop file -files filenames.lis -recpars params.ini 
  -log sf -nodisplay REDUCE_CLS

By default the recipe uses the "blank field" makemap config file (though this may be overridden with the MAKEMAP_CONFIG recipe parameter; see below). The recipe creates a slew of output files with the following suffices:

1. _fmos - signal map (1 for each observation)
2. _mappsf - map-filtered PSF (1 for each observation). This is the same as the above signal map but with an artifical gaussian added to the time-series (and located at the map centre).
3. _wmos - coadded signal map
4. _wpsf - coadded map-filtered PSF
5. _jkmap - jack-knife map
6. _whiten - whitened signal map
7. _whitepsf - whitened, map-filtered PSF map
8. _cal - calibrated, whitened signal map
9. _mf - above map processed with matched-filter (using the _whitepsf map as the PSF)
10. _snr - signal-to-noise map created from above map (_mf)

Note that the per-observation files begin with s, and the coadds (and subsequent products) begin with gs. The ones you are probably most interested in are the _mf and _snr files.

What does it do?

The recipe works as follows.

1. Each observation is processed separately to produce a signal map (_fmos). In addition, each observation is re-processed with an artificial gaussian source added at the map centre (this will be used to create the "map-filtered PSF" image, _mappsf).
2. The signal maps are combined using inverse-variance weighting to create a coadded signal map (_wmos). The images with the artificial gaussians added in are also coadded to produce the "map-filtered PSF" (_wpsf).
3. The data are split into two groups made from alternating observations. Each group is coadded, and the two coadds are subtracted to produce the jack-knife map (_jkmap).
4. The SMURF command sc2filtermap is used to estimate the radial angular power spectrum (circular symmetry is assumed) within a region defined by twice the minimum noise in the coadded signal map. (The size of this region is written to the FITS header of the output map under the keyword WHITEBOX.)
5. The inverse of this angular power spectrum is applied to the coadded signal and PSF images (_whiten and _whitepsf) to remove residual low spatial-frequency noise.
6. The amplitude of the fake source in _whitepsf is compared with the input value to derive a corrected FCF. This new FCF is used to calibrate the whitened signal map to create the _cal map.
7. A matched filter is applied to _cal using _whitepsf as the PSF to create _mf.
8. Finally, a signal-to-noise ratio image is created (_snr).

What parameters are available?

The following recipe parameters can be used to control the processing:

• FAKEMAP_SCALE - Amplitude of the fake source (in Jy) added to the timeseries to assess the map-making response to a point source.
• MAKEMAP_CONFIG - Name of a config file for use with the SMURF makemap task. The file must exist in the current working directory, $MAKEMAP_CONFIG_DIR, $ORAC_DATA_OUT, $ORAC_DATA_CAL or $STARLINK_DIR/share/smurf.
• MAKEMAP_PIXSIZE - Pixel size in arcsec for the output map. Default is wavelength dependent (4 arcsec at 850 um, 2 arcsec at 450 um).
• WHITEN_BOX - Size of the region used to calculate the angular power spectrum for removing residual low-frequency noise in the data. Default is a square region bounded by the noise being less than twice the minimum value.

What if I want to run it again?

In addition to the full pipeline recipe, there is a PICARD recipe called SCUBA2_JACKKNIFE which performs all of the post-map-making steps. This can be used to examine the influence of including or omitting individual observations (say ones with visible artefacts that the pipeline is not able to trap), or investigate the effect of varying the size of the whitening region, or trim the images to a specific size before re-running the jack-knife steps.

Note that the pipeline must have been run once to produce all the necessary files which go in to this recipe. And note that the full pipeline recipe must be run again if a different config file or input gaussian amplitude is to be used.

The majority of the flexibility in controlling the processing occurs after the individual signal maps have been created. As a refresher, running this recipe would mean typing:

% picard -log sf -recpars params.ini SCUBA2_JACKKNIFE myfiles*.sdf

All of the control is through the recipe parameters in the file params.ini. The most important item to note is that you must provide a map-filtered PSF (otherwise a default will be used). However, you already have one of these: the file ending _mappsf.sdf listed above.

• PSF_MATCHFILTER - the name of the map-filtered PSF file
• WHITEN_BOX - Size of the region used to calculate the angular power spectrum for removing residual low-frequency noise in the data. Default is a square region bounded by the noise being less than twice the minimum value.

This is only the first version of the recipe, and its limitations have yet to be determined. However, one case I have found where the recipe fails is in a large, multi-field map with highly-variable noise. In this case it might be better to process each field separately before creating the mosaic (though this has the disadvantage that different filtering may be applied to each field). Try it out and see what you get. All feedback welcome.

Automatically adding fake sources to real data, part 2

In a previous entry, I described how to go use the new ORAC-DR recipe REDUCE_SCAN_FAKEMAP to automatically add fake sources to the raw time series in order to get a handle on the transfer function of the map-maker. That's all very well, but how about starting with something simpler, such as a Gaussian? Now you can add a Gaussian of given amplitude and FWHM without first creating an image to feed in to the pipeline. As before, the Gaussian properties are defined using recipe parameters:

[REDUCE_SCAN_FAKEMAP]
FAKEMAP_FWHM = 15
FAKEMAP_SCALE = 0.5
FAKEMAP_OFFSET = dx,dy

The FWHM is in arcsec and is the only mandatory parameter (the SCALE and OFFSET are optional). In this case, the FAKEMAP_SCALE parameter is the amplitude of the Gaussian in Jy/beam. The Gaussian can be positioned anywhere in the map by giving an offset (RA, Dec in arcsec).

The pipeline can help you out further by using a couple of useful defaults. If the world "beam" is given as the FWHM, then the pipeline will use the default beam size appropriate for the current wavelength (14" at 850 um, 8" at 450 um). If the amplitude is not specified then (somewhat arbitrary) defaults of 1 and 4 Jy/beam are assumed (at 850 and 450 um respectively).

Note that the FWHM must be greater than half the pixel scale (the pipeline will set it to that value if the given value is less), but in principle there is no upper limit. In practice the upper limit is defined by the array footprint (about 2.5 arcmin for S2SRO data): anything much larger than that will be removed by makemap as part of fitting the common mode.

As usual, update your Starlink installation (with rsync) to try it out.

Recipe Parameters and the Science Archive

When processing jobs are submitted to the science archive ORAC-DR is configured to load a specialist recipe parameter file associated with the project. This can be used by PIs or surveys to tune processing to their liking. The idea is that PIs or survey teams send us their recipe parameter files and we then make sure that the processing jobs use them.

This week I've updated the recipe parameter system to allow tuning based on the object name as well as the recipe name. In some projects the objects are completely different so a single parameter file was not useful. Hopefully this change will encourage more people to send us parameter files.

To use the object based scheme simply append the object name to the recipe name in the file along with a colon. No spaces in the object name.

  [REDUCE_SCAN:M82]
  PAR1 = A
  PAR2 = B

If there is also a recipe-based entry those parameters will be merged in

  [REDUCE_SCAN]
  PAR1 = Aprime
  PAR3 = C

So in this example if the object name is "M82" PAR1, PAR2 and PAR3 will be set and PAR1 will have a value "A". If the object is no "M82" only PAR1 and PAR3 will be set and PAR1 will have value "Aprime".

This change is currently on the stardev rsync server, or if you have git installed you can update your own ORAC-DR distribution.

Automatically adding fake sources to real data

Further to Ed's post on adding fake sources to SCUBA-2 maps, I've added some new functionality to the pipeline which allows this process to be automated.

There is a new recipe called REDUCE_SCAN_FAKEMAP which should be specified on the command-line when running the pipeline. (Normally the recipe is chosen automatically.) This recipe behaves in exactly the same way as the standard recipe, and just passes the given `fake' map (adjusted and scaled as necessary) to the map-maker.

The recipe has a small number of parameters which can be given in addition to those already accepted by the pipeline (e.g. an alternative makemap configuration file). These parameters are all prefixed with "FAKEMAP_":

[REDUCE_SCAN_FAKEMAP]
FAKEMAP_MAP = mymap.sdf
FAKEMAP_SCALE = 0.5
FAKEMAP_REGRID = 1/0
FAKEMAP_OFFSET = dx,dy

The name of the base input map is mandatory (otherwise the pipeline exits immediately). The remaining parameters are optional. The SCALE parameter is the same as the "fakescale" parameter used by the map-maker and defaults to 1 if not given. Two further parameters control how the input map is adjusted before adding to the raw timeseries data.

The OFFSET parameter is a pair of RA,Dec offsets in arcsec which allow the input map to be shifted so that it doesn't coincide with a source in the SCUBA-2 data. The default is not to shift the input map. The offsets are applied to the source: i.e. a shift of 60,60 puts a source originally at (02:30:00.0, 00:00:00.0) at (02:30:04.0, +00:01:00.0) in the output map.

The REGRID parameter is a flag which, if true, regrids the input map to match the pixel coordinates of the output map (i.e. same pixel bounds and pixel size). The default is false, unless an OFFSET is given: in this case the input map must be aligned (regridded) to match the output map so that the pixel bounds are correct. Note that the original input file is not modified: the pipeline makes a copy and operates on that.

Here's an example of calling this recipe on the command line (processing a list of files contained in mydata.lis):

% oracdr -loop file -files mydata.lis -recpars myrecpars.ini REDUCE_SCAN_FAKEMAP

This new capability is available now with an update of your Starlink installation. Please try it out and let me know how well it worked (or didn't!), or send a message to the scuba2dr mailing list.

Automated advanced processing at CADC

As previously mentioned, the ORAC-DR data reduction pipeline could be run at CADC to generate basic nightly products. This processing used to be run at JCMT on a nightly basis, with the products transferred to CADC.

This processing is now being run at CADC on their processing system. Processing requests are made at 0900 HST every day, and nightly products will be available sometime after that (depending on the amount of processing needed -- scans take longer to reduce than jiggles).

In the near future, effort will be undertaken to process the backlog of ACSIS data.

ORAC-DR and Starlink on Twitter!

Much to Tim and Frossie's chagrin, I've created two Twitter accounts for ORAC-DR and Starlink. I haven't completely sorted out what will be done with them at this time, but for now I'll probably use them to disseminate short tips and tricks for using ORAC-DR and Starlink software.

Follow them at http://twitter.com/oracdr and http://twitter.com/starlinksoft!

Initial results of "better" ORAC-DR reduction

As previously mentioned, ORAC-DR is improving how ACSIS data are reduced. To show the progress between "summit" and "better":

• integrated intensity map, group coadd: summit better


• integrated intensity map, single observation: summit better


• intensity-weighted velocity map, group coadd: summit better

A few notes:

• By "summit" pipeline I mean the pipeline currently running at the summit. This pipeline will be replaced by an "improved" pipeline pending JCMT support scientist approval. The "improved" pipeline will not be run at CADC, they will run the "better" pipeline that created the "better" images linked above.


• The "group" summit integrated intensity map is not generated by the pipeline, it's created by manually running wcsmosaic to mosaic together the individual baselined cubes (the _reduced CADC products), then collapsing over the entire frequency range. This is how the summit pipeline would create those files, though.


• Ditto for the group summit velocity map, except the pipeline wouldn't even create those in the first place, as it doesn't know which velocity ranges to collapse over to get a proper velocity map. This example is just done by naively collapsing over the entire velocity range. The "better" pipeline automatically finds these regions and creates velocity maps -- not only for the coadded group cube, but also for individual observation cubes.


• The difference between the "better" pipeline (which is what will be running at CADC) and the "improved" pipeline (which is what will be running at the summit) is very small for this given dataset.

ACSIS DR and hybrid observations

In the version of the ACSIS pipeline that will ship in the upcoming Starlink "humu" release subsystems from ACSIS are processed independently of each other. This is fine for true multi-subsystem observations such as simultaneous C18O and 13CO modes but is not the correct thing to do when a true hybrid mode is observed. This was not an issue when we relied on the "real-time" DR to do the sub-band merging but since the decision was made to configure 500MHz and 2GHz observations as pseudo multi subsystem observations (albeit with shared reference pixel so that the channels are aligned) it has become obvious that ORAC-DR needed to be modified.

This week I modified the internals of ORAC-DR to allow it to recognize hybrid observations and treat them as a single "frame" (in ORAC-DR speak). It was a little more involved than expected because there was no merging of header information in the pipeline outside of bespoke implementations in UKIRT spectroscopy and SCUBA-2 classes. I moved some header parsing code into a base class and removed the special code from UKIRT (SCUBA-2 is still special). This required some minor changes to all the header translation code but was worth it since all instruments can make use of the header merging.

Next step is to actually use this information to merge the sub-bands. One minor caveat in all this is that ORAC-DR will still not try to merge multi-subsystem observations simply because subsystems overlap. If that is required (for example for the SLS)  then we need to be told the requirements.