New OMP features for projects.

The OMP has recently been updated to display several additional pieces of information on the project pages.

Tau graph

The first thing you will notice is a new graph of tau over the course of the night. Previously, the pages had such a plot that was generated each time the page was loaded. This took a while, and the plot had automatically generated limits on the y-axis which made comparing graphs between two night effectively impossible.

The new graph is made with consistent x- and y-limits that allows for easy comparison between nights. The various weather grades are also delineated, and the value of tau from the CSO WVM is plotted as well.

The new plot of tau vs. time. Click an any picture for a larger view.

The thicker gray horizontal lines running across the graph mark the weather grade boundaries, and the shaded area shows the night hours in Hawaii. The time in UTC is plotted along the bottom axis, with the time in HST along the top.

Pie chart

The pie chart of time spent in each grade.

There is also a pie chart that sums up the amount of time spent in each weather grade throughout the night. The grades are color-coded from green to red to give a quick visual assessment of how good a given night was (green good; red bad). Usually a given night will fall predominately within just one or two grades.


ACSIS standards table

For nights on which data was taken using ACSIS, there will be an HTML table in text form like the example below with some information relating to the ACSIS standards observed.

Obs #	Time	Integ. Int.	Peak Int.
16	19:36:32	251.33	8.82
19	19:50:31	4775.56	6.88
27	20:59:30	4157.76	5.96
36	22:22:36	3520.79	5.08

SCUBA-2 calibrations table

Similarly, for nights where SCUBA-2 was used there will be an HTML table like the example below listing the FCFs for each observation of a calibration object.

20120822			FCFasec				FCFpeak
Obs #	Time (UT)	Source	850µm	err	450µm	err	850µm	err	450µm	err
8	05:41:27	CRL2688	2.41	0.01	4.64	0.03	572.2	1.5	542.9	5.7
37	10:24:23	CRL2688	2.30	0.01	4.66	0.02	513.5	1.2	466.0	3.7
68	17:26:48	CRL618	2.23	0.01	4.28	0.04	487.8	1.4	436.8	4.8

Along with the table on SCUBA-2 nights there will three additional graphs, which will be detailed below. Each of these graphs has two sub-plots, the top one being the 450 micron data and the bottom one being the 850 micron.

NEPs vs observation number graph

This graph shows the min, max, and mean for each of the eight subarrays that make up SCUBA-2, plotted by observation number. The mean is the colored line, and the shaded areas are the filled-in areas between the min and max. By following the lines you can tell which arrays changed significantly, and by watching the shaded areas you can get a feel for the spread in the NEPs. The vertical scale is fixed, and is the same as the scale for the following plot.

NEPs vs time graph

This graph, like the previous one, shows the NEPs, but plots them against time rather than observation number. Each of the eight subarrays is present once again (with the same colors as in the preceding plot). The time (in UTC) is marked along the bottom of the plot, and the observations are marked by number along the top, with vertical lines that descend to help mark when each particular observation began.

NEFDs vs time graph

The final plot is a plot of the NEFDs vs. time. The number of points depends on the night, this particular night only had a few. Like the two previous plots, the vertical axis is fixed to make comparisons between nights easier.

Hopefully these new features will prove useful to users of the OMP, and additional updates or improvements may be forthcoming in the future. Feedback on the new features is welcome.

Automated removal of bad-baseline spectra from ACSIS/HARP time series

I presented the following poster at November's ADASS Conference, describing the methods I added to the ORAC-DR REDUCE_SCIENCE_NARROWLINE and REDUCE_SCIENCE_GRADIENT recipes to detect and remove `wobbly' and noisy spectra. It shows some examples of the then identified forms of interference and example results.  Since then an additional form---ringing---came to light from looking at more data, and is shown in the latest JCMT Newsletter 34. I'll discuss that in a further blog.

Go here for the full-sized PDF.

FIT1D: A new SMURF command for ACSIS data

More time ago than I am willing to admit, I started coding a Starlink routine to fit spectral lines in ACSIS cubes. Got a long way until SCUBA-2 commissioning and calibration put a halt to it, but I have finally managed to finish the program, technically a beta-version, as part of the upcoming Kapuahi release of Starlink.

Usage:

% fit1d  in  out  rms  [config]  [userval]  [pardir]  [parndf]  [parcomp]

What distinguishes FIT1D from other profile fitting routines is that it specifically attempts to deal with two issues: non-gaussian profile shapes and the fact that ACSIS data-cubes have many, potentially very different, profiles to fit. Regarding the latter, there are many fitting routines that produce fitted profiles for data-cubes, but FIT1D also produces cubes with the fitted parameters themselves and can use such files as input to give control over the fit of, in principle, each individual profile in the cube. Thus it is e.g. possible to fit broad lines on the nucleus of a galaxy and narrow lines everywhere else. More about that below.

A section has been added to the SMURF documentation in SUN/258 about FIT1D, which I will try to summarize here. FIT1D is generic in that it can fit profiles along any axis of an up to 7-dim hyper-cube, but will be discussed here in the context of a default RA-Dec-Vel ACSIS cube. Note that the routine assumes that data have been baseline-subtracted, using e.g. MFITTREND, i.e. that the profiles have a zero-level at 0. 

Gauss-Hermite shapes as a function of the 3rd-order
      skewness coefficient 'h3' and the 4th-order the kurtosis (peakiness)
      coefficient 'h4'. The red box indicates the limits on acceptable
      values for h3 and h4 as defined in the defaults configuration file. Note
      that the fitted profile by default is restricted to positive values
      and will omit the shown negative features.

Non-gaussian Profiles.

FIT1D essentially re-implements the fitting code for non-gaussian profiles from the GIPSY package (Kapteyn Institute, Groningen, The Netherlands). Function types that can be fitted are Gaussian, Gauss-Hermite, and Voigt profiles. In particular, Gauss-Hermite functions are a powerful extention when fitting profiles that are skewed, peaky, or only approximately gaussian. The figure above shows Gauss-Hermite profiles as a function of the skewness coefficient h3 and the kurtosis (peakiness) coefficient h4. See  for SUN/258 further details, but note that the default setting in the configuration file is for FIT1D to suppress the negative features in the fitted profiles and to leave only the positive part of Gauss-Hermites.

Because of their ability to fit distorted shapes, Gauss-Hermites are particularly well suited to "capture" the maximum amount of emission from a cube. The fits can be remarkably accurate as is shown in the the figure below showing a 3-component fit (i.e. up to 3 spectral lines) using gausshermite2 functions (i.e. fitting both h3 and h4). Collapsing the resulting cube with fitted profiles can thus result in an accurate and almost noise-free white-light or total-emission map.

Fit1d - Black: original profiles; Red: results of a
    3-component Gauss-Hermite2 fit (fitting both h3 and h4)

FIT1D derives its ability to fit a complex line-shape both from the Gauss-Hermite function but also from that it can fit multiple (sub) components to get the best match possible. However, that can make the interpretation of the fits in terms of the physical characteristics and quantities difficult, hence for those you may also want to make a fit of the line-shape using a single standard Gaussian function. 

Component Parameter files

Besides a data-cube with the fitted profiles FIT1D also outputs so-called Component parameter files as NDF extensions in the header of the output file. These can also be copied out as independent data-cubes. There is a file for each component (i.e. line) that was fitted along the profile up to the number of components requested by the user. Each plane of a Component parameter file has an image of the value of a fitted parameter across the field-of-view. For instance, the one resulting from a gaussian fit has images respectively showing the fitted Amplitude, Position (velocity), and FWHM as well as a plane with an id-number of the function used.

Much of the (anticipated) use of FIT1D derives from the fact that Component parameter files can be used as input as well: either to provide initial estimates or fixed values to the fitting routine.  The difference between values specified in the Component parameter files
and ones declared in a User parameter values file is that the former can vary across the field-of-view whereas the latter will result in the same value being used for all profiles. E.g. for use with spectral-line surveys the User parameter values file can be used to provide initial estimates of the frequencies or velocities at which lines are expected or to fix fits at those frequencies.

By manipulating Component parameter files e.g. resulting from an initial fit, the user can customize or correct subsequent fits. In extrema, a Component parameter file could be made from scratch based on a model and be used to create a spectral-line data-cube with that model (config option: model_only=1) or be used as initial estimates for a fit. Of more practical use, Component parameter files can be used to correct problems associated with a fit since the art of fitting is not in the fitting algorithm, but in providing accurate initial estimates. For instance, the left image below shows a section of an Amplitude plane of a fit where there are problems in a few locations. Setting these location to bad values and using FILLBAD to interpolate over them, the corrected Component parameter file was used as initial estimate for a subsequent fit, resulting in the image on the right

Fit1d - Left: Section of a parameter file showing
      originally fitted amplitudes; Right: Amplitudes after using a
      corrected parameter file from the original fit as initial estimates
      for a subsequent fit.

More creative options are possible: after an initial fit with a gaussian, the function id can be changed to a gausshermite1 in part of the field and the resulting file used as initial estimates for a subsequent fit to account for skewed profiles there. Similarly, the initial guess of the FWHM can be made wide on e.g. nucleus of a galaxy while leaving it more narrow outside. As another example, the fit of multiple components can be limited to only part of the field by setting the parameter file for the second and higher components to bad values outside the relevant region (multiple component parameter files can be used as input: one for each component to be fitted).

In conclusion: please remember that this is a beta-release and that you may run into unanticipated issues. Also chosen limits in the configuration file may need tweaking. If an initial fit looks poor, try adjusting minamp (in units of rms!) or, in particular, minfwhm (in units of pixels!) in the configuration file (see: $SMURF_DIR/smurf_fit1d.def). Also use range to limit the fit to a relevant region of the spectrum.

The released implementation of FIT1D can fit up to 7 components per profile per run, but the output of multiple runs each covering a range in velocities or frequencies can be combined. The fit itself is fully multi-threaded and will be much faster on a modern multi-core computer: a 3-component gausshermite2 fit of 1.1 million spectra (a 2 Gb input file) took 15 minutes on a dual-core, 16 Gb memory machine versus 4 minutes on one with 12 cores and 75 Gb of memory.

Happy fitting!

Remo

QA-enabled pipeline released in Hilo

ORAC-DR has been updated in Hilo to include quality-assurance testing. Based on a number of QA tests, observations are given a pass/questionable/fail status. QA is automatically done on all science observations, and survey-specific QA parameters can be given.

This version will eventually be released to the summit, where it will give telescope operators feedback on which surveys are suitable to do, along with enhancing the JCMT Science Archive pipeline.

Creating artifical time series from a sky cube

I've just added a new command to smurf called UNMAKECUBE. It takes a (ra,dec,spec) sky cube and generates artifical time series by resampling the sky cube at the detector positions in a set of existing reference time series NDFs. It's a sort of inverse to MAKECUBE. It should be useful for investigating baselines, and for iterative map making. I'm currently playing around with it, using data from Christine Wilson.

ACSIS DR and hybrid observations

In the version of the ACSIS pipeline that will ship in the upcoming Starlink "humu" release subsystems from ACSIS are processed independently of each other. This is fine for true multi-subsystem observations such as simultaneous C18O and 13CO modes but is not the correct thing to do when a true hybrid mode is observed. This was not an issue when we relied on the "real-time" DR to do the sub-band merging but since the decision was made to configure 500MHz and 2GHz observations as pseudo multi subsystem observations (albeit with shared reference pixel so that the channels are aligned) it has become obvious that ORAC-DR needed to be modified.

This week I modified the internals of ORAC-DR to allow it to recognize hybrid observations and treat them as a single "frame" (in ORAC-DR speak). It was a little more involved than expected because there was no merging of header information in the pipeline outside of bespoke implementations in UKIRT spectroscopy and SCUBA-2 classes. I moved some header parsing code into a base class and removed the special code from UKIRT (SCUBA-2 is still special). This required some minor changes to all the header translation code but was worth it since all instruments can make use of the header merging.

Next step is to actually use this information to merge the sub-bands. One minor caveat in all this is that ORAC-DR will still not try to merge multi-subsystem observations simply because subsystems overlap. If that is required (for example for the SLS)  then we need to be told the requirements.