Data processing at CADC

Right now we are having trouble reducing SCUBA-2 data for the JCMT Science Archive (JSA).Briefly the problems seem to be:

• The JSA wrapper is reporting success even in some cases of failure
• Some maps take so long to make that MAKEMAP is timing out during long maps (this will also affect processing on private computers)
• There is a bizarre NFS problem on the processing nodes that causes required perl modules to "disappear" when the pipeline is looking for them.

We're working through fixing these, so until then availability for products is a bit patchy. The good news is that processing throughput is great, so catching up with the backlog does not seem to  be a problem.

Monitoring changes to the DR software

Yesterday I gave instructions on how to retrieve the newest version of the Starlink software but it may not always be clear to people what changes are being made to the software to decide whether you want to get an update. The source code repositories for the Starlink software and ORAC-DR have RSS feeds that you can monitor in your standard news feed reader (e.g. Google Reader). Click on the RSS icon in the URL bar for the Starlink repository and the ORAC-DR repository.

SCUBA-2 DR status report

We have been making some good progress with the data reduction software. Since hawaiki there have been updates to the flatfielding and an improvement in the code that detects working bolometers by comparing the common-mode signal. The down side of all this work is that for people getting their data from shared-risks observing they need to be using the cutting-edge version of SMURF and not the hawaiki version. The switch of flatfielding technique actually means that hawaiki will not generate a reasonable map at all using hawaiki SMURF.

To get the latest version of SMURF there are a number of options:

• Use rsync to get a version for Centos5.4 (aka Scientific Linux 5.4 or RedHat Enterprise Linux 5.4) for either 32- or 64-bit linux. We attempt to keep these builds up to date. To see just how new you can run "$STARLINK_DIR/Perl/bin/starversion" and look at the date.
• I can periodically make available 64-bit OSX Snow Leopard starlink releases. Check this directory periodically.
• Download the source code and build it yourself (instructions on the Starlink home page)

As of 20100223 we have modified data acquisition to include a fast flatfield "ramp" at the start and end of every observation. These provide a direct calibration and should be more accurate than doing standalone flatfield observations as we have done previously (and continue to do). SMURF can not support these ramps at the moment but I am working on the issue. When the new code is working we will re-process the data from 20100223 at CADC.

Starlink release: Hawaiki (Deneb)

Hawaiki has shipped!

http://starlink.jach.hawaii.edu/starlink/Hawaiki

Checkout the first (it sure won't be the last) version of the SCUBA-2 data reduction cookbook:

http://starlink.jach.hawaii.edu/docs/sc19.htx/sc19.html

Watching the data reduce

Thanks to the folks at CADC, Dustin Jenkins in particular, JAC now has a really nice interface that allows us to monitor the jobs submitted to their Grid Engine, look for faults and browse the thumbnails of the products in order to spot problems - or just sit back and admire the results :-)

JLS/ACSIS DR telecon

Attendance: ACC, BEC, TJ, FE, JVB, RP, CW, JdiF
Date: 15 Oct, 2009

Minutes:
1. Review of actions from previous meeting:

1. i. JAC will provide information on how to rsync the starlink releases to get latest patches/fixes. Information will also include for which operating system these patches/fixes are available. – DONE
2. JAC to make a more compact and readable QA report format and make this log available to observers/co-Is following nightly reduction via the OMP (as a downloadable file). – ONGOING
3. For SLS to provide JAC (ie Brad) with list of statistics and requirements for their QA, and also what they want for their reduction recipes to do. – ONGOING. Material has been received from SLS (also available on SLS wiki) and Brad is working through it.
4. For JLS teams to provide JAC with images/data/log of spikes when they come across them in their data. – ONGOING
5. ACC to organise the production of pipeline documentation. – STARTED. ONGOING.
6. ACC to poll for a date and time for next telecon and make these meeting notes available – DONE

2. News from JAC

• Nightly reductions are now being carried out at CADC.
• There have been tweaks to raster maps production such that the pipeline trims off the tassled edges of maps. Note that this is only for the maps and not the cubes.
• We can now regrid to specific wcs coords (e.g galactic).
• The infrastructure for controlling the pipeline via a parameter system and a config file is in place. Now need to decide which parameters are to be accessible before implementing the system. JAC will come up with an initial list and then invite feedback from users.
• There will be two relevant newsletter articles in Autumn edition of JCMT Newsletter. One on JSA and another on the ORAC-DR pipeline.

ACTION : JLS teams to provide feedback on what should go in parameter file (once JAC have made initial list).

3. Updates from survey teams

• GBS : working towards getting a consistent reduction on the data set. Need QA format finalised first.
• NGS : data taking almost complete. Data is currently run through manual QA - flagging of bad data and receptors is critical and it’s subtlety requires human interaction. As the results of this is documented and the recorded state of the receptors isn’t going to change, we should be able to transfer that knowledge to an automated system in the future when we have a triggered re-reduction. Reduced products generated by NGS can be uploaded to the JSA. Improvements in moments maps: based on utilising SNR and noise maps. Good for data taken in different weather conditions.

ACTION : CW to pass on NGS QA and moment map making scripts to JAC.

• iii. SLS : biggest issue is manpower. GAF has sent information to us to kick off the SLS pipeline QA process. Meeting @ JAC with Paul Ruffle tomorrow to discuss SLS DR issues.

4. Designated DR contacts
JAC would like to have email contacts with individuals in survey teams who are regularly looking at data and have the time and inclination to correspond and work with JAC between meetings. It is important that these individuals do have the time available (e.g. students, post-docs) to run tests and feedback ideas and improvements. Although these telecons are proving useful, progress is slow if we wait for monthly meetings to get feedback on more minor (yet potentially critical) issues. Everybody agreed that this is a good idea.

ACTION : ACC to email coords asking them to provide email contacts of people who are reducing ACSIS data and are willing to act as ACSIS DR contacts.

ACTION : JLS coords to provide email contacts of people who are reducing ACSIS data and are willing to act as survey DR contacts (send to FE).

Automated advanced processing at CADC

As previously mentioned, the ORAC-DR data reduction pipeline could be run at CADC to generate basic nightly products. This processing used to be run at JCMT on a nightly basis, with the products transferred to CADC.

This processing is now being run at CADC on their processing system. Processing requests are made at 0900 HST every day, and nightly products will be available sometime after that (depending on the amount of processing needed -- scans take longer to reduce than jiggles).

In the near future, effort will be undertaken to process the backlog of ACSIS data.