JLS/ACSIS DR telecon – 20100304

Attendance: GAF, JH, CW, PR, JDF, TJ, FE, HST, ACC

Actions from last meeting:

• JAC to make a more compact and readable QA report format and make this log available to observers/co-Is following nightly reduction via the OMP (as a downloadable file).
  • update: there had been correspondence between JennyH and BradC and although there had been some work done, this item had not been completed before BradC left – ONGOING
    
• JLS teams to provide feedback on what should go in parameter file (once JAC have made initial list). – ONGOING
• BradC to work through SLS pipeline and QA requirements as provided to him (and available on SLS wiki)
  • update: following BradC's departure, this will have to be picked up by his replacement (hopefully by next month) – ONGOING
• For JLS teams to provide JAC with images/data/log of spikes when they come across them in their data.
  
  • update: JAC had received some spike examples from SLS. These have been forwarded to DaveB but given his current priorities we may not see any movement on this for a few months yet (summer maybe?) – ONGOING
• ChristineW to pass on NGS QA and moment map making scripts to JAC. – DONE
  
  • ACTION: TimJ to look into whether BradC did code this into the pipeline?
• AntonioC to organise the production of pipeline documentation
  
  • update: some minor organisation of documentation but this has fallen down the list of priorities since the start of SCUBA-2 commissioning – ONGOING/STALLED
• AntonioC to email coords asking them to provide email contacts of people who are reducing ACSIS data and are willing to act as ACSIS DR contacts. – DONE
  
• JLS coords to provide email contacts of people who are reducing ACSIS data and are willing to act as survey DR contacts (send to FrossieE). – DONE
  

News from JAC

• The software group took a big hit to its available effort when BradC left for new employment in February 2010. We are working on buying in some effort from an experienced ex-Starlink programmer to pick up on many of the tasks that Brad left. This person's effort should start becoming available to us from April 2010.
• since the commissioning of SCUBA-2 started and the subsequent onset of SCUBA-2 Shared Risks Observing, JAC has been inundated in SCUBA-2 data
• we can now re-reduce data at the JSA across different UT nights to create so-called "project products". This is a very important step and these products are essentially the Advanced Data Products (ADPs) that the JSA will serve.
• FrossieE continues to submit data reduction jobs to the JSA but is experiencing problems as makecube has been timing out (in project mode) when cubes are very, very large

Report from GBS

• a meeting of the GBS was help in Leiden in February 2010; discussions were held on the QA at that meeting
• there are still missing QA steps in the pipeline reduction. For instance, bad scans are propagating through to final product, (bad scans = one or combination of high rms, many bad pixels, high variation across spectra)
  • ACTION JAC: this is a high priority for the ACSIS pipeline at the JSA and JAC will look into this when effort materialises

Report from NGS

• no issues reported
• observing on the ACSIS portion of the survey has completed.
• the team is reducing the data themselves via student effort; they will then want to upload the final products to the archive

Report from SLS

• the team is focussing on getting all data reduced to then attack QA issue consistently
  • ACTION: GaryF to direct FrossieE to SLS wiki where QA requirements are published
• PaulR and HollyT have enough data now to compare pipeline reduced data with hand reduced data
  • ACTION : PaulR to forward DR documentation to JAC – DONE

Next meeting May 6th.

JLS/ACSIS DR telecon

Attendance: ACC, BEC, TJ, FE, JVB, RP, CW, JdiF
Date: 15 Oct, 2009

Minutes:
1. Review of actions from previous meeting:

1. i. JAC will provide information on how to rsync the starlink releases to get latest patches/fixes. Information will also include for which operating system these patches/fixes are available. – DONE
2. JAC to make a more compact and readable QA report format and make this log available to observers/co-Is following nightly reduction via the OMP (as a downloadable file). – ONGOING
3. For SLS to provide JAC (ie Brad) with list of statistics and requirements for their QA, and also what they want for their reduction recipes to do. – ONGOING. Material has been received from SLS (also available on SLS wiki) and Brad is working through it.
4. For JLS teams to provide JAC with images/data/log of spikes when they come across them in their data. – ONGOING
5. ACC to organise the production of pipeline documentation. – STARTED. ONGOING.
6. ACC to poll for a date and time for next telecon and make these meeting notes available – DONE

2. News from JAC

• Nightly reductions are now being carried out at CADC.
• There have been tweaks to raster maps production such that the pipeline trims off the tassled edges of maps. Note that this is only for the maps and not the cubes.
• We can now regrid to specific wcs coords (e.g galactic).
• The infrastructure for controlling the pipeline via a parameter system and a config file is in place. Now need to decide which parameters are to be accessible before implementing the system. JAC will come up with an initial list and then invite feedback from users.
• There will be two relevant newsletter articles in Autumn edition of JCMT Newsletter. One on JSA and another on the ORAC-DR pipeline.

ACTION : JLS teams to provide feedback on what should go in parameter file (once JAC have made initial list).

3. Updates from survey teams

• GBS : working towards getting a consistent reduction on the data set. Need QA format finalised first.
• NGS : data taking almost complete. Data is currently run through manual QA - flagging of bad data and receptors is critical and it’s subtlety requires human interaction. As the results of this is documented and the recorded state of the receptors isn’t going to change, we should be able to transfer that knowledge to an automated system in the future when we have a triggered re-reduction. Reduced products generated by NGS can be uploaded to the JSA. Improvements in moments maps: based on utilising SNR and noise maps. Good for data taken in different weather conditions.

ACTION : CW to pass on NGS QA and moment map making scripts to JAC.

• iii. SLS : biggest issue is manpower. GAF has sent information to us to kick off the SLS pipeline QA process. Meeting @ JAC with Paul Ruffle tomorrow to discuss SLS DR issues.

4. Designated DR contacts
JAC would like to have email contacts with individuals in survey teams who are regularly looking at data and have the time and inclination to correspond and work with JAC between meetings. It is important that these individuals do have the time available (e.g. students, post-docs) to run tests and feedback ideas and improvements. Although these telecons are proving useful, progress is slow if we wait for monthly meetings to get feedback on more minor (yet potentially critical) issues. Everybody agreed that this is a good idea.

ACTION : ACC to email coords asking them to provide email contacts of people who are reducing ACSIS data and are willing to act as ACSIS DR contacts.

ACTION : JLS coords to provide email contacts of people who are reducing ACSIS data and are willing to act as survey DR contacts (send to FE).

JLS DR telecon - 1st meeting

Attendance: A. Chrysostomou, R. Tilanus, T. Jenness, R. Plume, M. van der Wiel, J. Di Francesco, G. Fuller, B. Cavanagh, H. Thomas, D. Johnstone, H. Roberts, D. Nutter, J. Hatchell, F. Economou

- initial discussion on whether we will have a SCUBA-2 pipeline ready. There will be something in place for shared risks but basic. More development will have to wait until we have all arrays in place as it is not worth sinking any effort into this at this time.

- some people are having issues getting the pipeline installed and the fact that there is a lack of documentation. If people/institutes are having issues installing (any) Starlink software, then please inform the JAC (stardev@jach.hawaii.edu) providing the relevant details.

ACTION 1: JAC will provide information on how to rsync the starlink releases to get latest patches/fixes. Information will also include for which operating system these patches/fixes are available.

DONE(!): Instructions are available on the starlink web site (http://starlink.jach.hawaii.edu/).
To download the most recent release go to: http://starlink.jach.hawaii.edu/starlink/Releases
To keep up to date with the latest fixes and patches go to:
http://starlink.jach.hawaii.edu/starlink/rsyncStarlink


- GAF requested for more statistics to be made available from the QA. GAF will follow up with specific request to Brad (see Action 3 below)

- it was clarified that the summit pipeline (during normal night-time observing) only runs basic QA on calibrations. After the end of observing, all data taken that night is re-reduced by a “nightly pipeline” which executes the full QA and advanced processing. The reduced data products which result from this are shipped to CADC and can be downloaded with (or without) the raw data in the normal way.

ACTION 2a: JAC to make QA log available to observers/co-Is following nightly reduction via the OMP (as a downloadable file).
ACTION 2b: JAC to make a more compact and readable QA report format.

ACTION 3: For SLS to provide JAC (ie Brad) with list of statistics and requirements for their QA, and also what they want for their reduction recipes to do.

- JH raised some existing issues from the GBS: flatfielding (striping) of early HARP data; some bad baselines are not being picked up by QA; although not as prevalent as in older data, spikes are not trapped by the QA; an investigation is needed on how the gridding should best be done

+++ the flatfielding problem is on Brad’s worklist

+++ we need more feedback from the teams on which bad baselines are not been filtered out

+++ de-spiking data is not a problem that JAC has been able to tackle as yet. Part of the issue is that these do not seem to be as prevalent in data any more and observers (PI as well as JLS) are not reporting the issue any longer. GAF reported that spikes are still present but at a small level, which is an issue for SLS who are looking for weak, narrow lines.

ACTION 4: For JLS teams to provide JAC with images/data/log of spikes when they come across them in their data.

- RPT raised a few issues from the NGLS:

+++ need ability to baseline fit both wide and narrow lines in same data set

+++ need ability to restrict e.g. moments analysis to known velocity range.

+++ QA generally fails for (at least) early NGLS data. Will need to investigate this more but need an easy means to switch off in recipes. This is easy in the main recipe, but less so in the advanced interative part.

- there is a blog available for data reduction and pipeline activities (you’re problably looking at it right now!): http://pipelinesandarchives.blogspot.com/

- the issue of making the pipeline more controllable through a config file to set parameters was discussed. TJ announced that he is developing infrastructure so that the pipeline can be parameterised

- ACC received several emails prior to the meeting. A common theme was the lack of documentation explaining what the pipeline does to data, and how to use the pipeline. JH repeated this concern at the meeting.

ACTION 5: ACC took an action following the close of the meeting to organise the production of pipeline documentation. These will probably take the form of a detailed account of what the pipeline does, and a separate cookbook which explains how to run the pipeline with the different options available.

ACTION 6: ACC to poll for a date and time for next telecon and make these meeting notes available.

Time accounting for JLS

Another issue for JLS observing is with time accounting. How to account in the OMP the time for data that has been deemed unacceptable. These data do not get charged to the surveys so if the data were initially QUESTIONABLE retroactive action will need to be taken to correctly account for that time (notwithstanding the issue of shared calibrations...see later). So that we can track how much time has been REJECTed by each survey, there shall be a special project code (eg. MJLSG00) for each which we will use to charge REJECT observations to.

The idea is that when the obslog flag changes:

1. an email is triggered to ACC and PIs notifying the change

2. if the change is to REJECT then the release date is automatically changed to TODAY (or equivalent)

3. ACC runs up nightrep for the night in question and changes the time accounting accordingly.

4. changes to flags are propagated to CADC

In a situation where calibrations are provided by the observatory this system should work flawlessly (and a tool which takes care of the time accounting automatically would also be feasible). However, in the current situation where calibrations are shared amongst the projects, it is difficult to do the time accounting properly in this scheme as it is not immediately obvious how much calibration time should be taken with the REJECTed observation. It would have to be recalculated.

How to properly deal with BAD/QUESTIONABLE data within the JLS

The problem stems from what happens to questionable data which remains in some form of limbo until its status is deemed to be GOOD or BAD. Setting data to BAD is an issue in itself as such data are usually BAD because of a fault. However, the Legacy nature of JLS means that some data will be deemed to be unacceptable for the surveys and so should not be processed into Advanced Data Products. These data are not intrinsically BAD and so the plan is for them to be immediately released to the public.

We resolve this issue by having a new quality parameter in obslog - REJECT. Definitions:

GOOD: data is good and is processed by the pipeline

QUESTIONABLE: data may have problems with it - a human needs to look and make a decision on its quality

BAD: data is not good, do not process 

REJECT: data does not meet agreed standards for survey

We don’t expect to be using the REJECT flag during normal PI observations. Furthermore, it is expected that with working QA in the survey pipelines the number of REJECT and QUESTIONABLEs will be small (but there will be enough, especially at the beginning as we're bedding the system, that we need to deal with them appropriately).

The following table summarises what happens to data with these flags:

_cube _reduced    group   proprietary

GOOD      Y         Y         Y         Y  

BAD       Y         N         N         N  

QUEST.    Y         Y         N         Y  

REJECT    Y         Y         N         N 

          ADP    charged  VO/master product

GOOD       Y        Y           Y

BAD        N        N           N

QUEST.     N        Y           N 

REJECT     N        N           Y

(N.B: QUESTIONABLE data should not be combined into the public VO product as those data are in an undefined quantum state and until their wave functions have collapsed into either of GOOD, BAD or REJECT you don’t know what to do with them)