*1-2 December 2016, Meridian room, Royal Observatory of Belgium, 3 Avenue Circulaire, Brussels* ## Preliminary Agenda ### Thursday, December 1, 2016, 02:00 pm - 06:00 pm 1. Data searchability - Metadata for SVO - Our SOLARNET SVO prototype and how to manually ingest data from the SOLARNET telescopes (ROB, presentation by B. Mampaey and R. Vansintjan) - Based on D20.4, building up the final list of keywords to be compatible with the SVO requirements on data searchability (KIS, Nazaret Bello González) - To what degree of detail should the data be characterised? - How will the 'final list' evolve with instrument technology and data calibration methods? - [Telescope data base <--> data fits header] --> SVO metadata - Metadata concepts developed for DKIST (Kevin Reardon) - Application to existing data archives: GRIS (schliche), BBI (schliche), GFPI(?), CRISP(?), CHROMIS(?) 2. Defining metadata parameters for data quality from ground-based observations (KIS) - What data should be available via SVO? How to define quality? What data should be deleted? ### Friday, December 2, 2016, 09:00 am - 01:00 pm 1. Data handling at EST: Data pipelines, archives, metadata, analysis (KIS) - What data will be transferred from the mountain (telescope) to the operation centre at 'sea level'. 2. SOLARNET WP20&50 goals - Status and To Do's (Kiselman & Schlichenmaier) Upcoming deliverables: - D20.5 Report on the VO tools prototype (due 03/2017) - D50.8 Data archive prototype linked to the SVO (due 03/2017) - D20.7 Coordinated campaign (due 03/2017) - D20.10 Report on TAC tasks - D50.4 Standardised and improved MFBD and speckle codes - D50.7 Status, requirements and development of the instrument pipelines - MS6 Analysis of data compression algorithms 3. Other European projects on big data (KIS) - Report on the ASTERICS project and the SOLARNET/EST enrollment on it - Heidelberg Meeting (Summer 2016) - Report on the GEDE project and enrollment of EST on it 4. Miscellaneous - Making use of the Cloud for data storage/sharing - The use of Python in the Solar Physics community - [SOLARNET SVO Prototype](./uploads/MeetingFAS20161201/SOLARNET_FAS_meeting_VSO.pdf) - [wp20wp50deliverables](./uploads/MeetingFAS20161201/wp20wp50deliverables_slides.pdf) - [Getting data ready for dissemination through the SVO](./uploads/MeetingFAS20161201/metadata.pdf) - [ASTERICS Astronomy ESFRI and ResearchInfrastructure Cluster](./uploads/MeetingFAS20161201/asterics_gede.pdf) - [The IBIS data Archive](./uploads/MeetingFAS20161201/ibisa_ie_01122016.pdf) - [Inputs from HELIO](./uploads/MeetingFAS20161201/HELIO-SNet-Bentley.ppt) - [Multiwavelength study of penumbral decay using GREGOR, VTT, DST, NST, and Hinode.](./uploads/MeetingFAS20161201/Meetu_small.pdf) - [DKIST Data Center --Metadata and Search Capabili6es](./uploads/MeetingFAS20161201/DKIST.search.SolarNet.FAS.Dec2016.pdf) Minutes of the meeting - [Minutes of this FAS meeting](./uploads/MeetingFAS20161201/WP20_4th_FAS_minutes.pdf) ### Links shared in chat during the meeting During the discussion of searching on FITS headers: a poster on documenting FITS headers, from 2014 AAS/SPD meeting : https://vso1.nascom.nasa.gov/print/print_2014SPD_FITS.pdf When there was mention of searching for active regions by time, and then by spatial extent: spatial searching, from 2013 SPD : https://vso1.nascom.nasa.gov/print/2013_SPD_Spatial_Search_draft3.pdf During the discussion of 'Required' FITS headers: - Maybe it'd be worth having a 'Recommended' between 'Required' and 'Optional' ? And the mention of NAIF: the tool is called NAIF : https://naif.jpl.nasa.gov/naif/toolkit.html - wait, no ... NAIF i[s] the group distributing SPICE And UCD was from the nighttime VO community : http://www.ivoa.net/documents/latest/UCDlist.html And mention of different observing modes for instruments: Hinode/SOT has 'SP4D' (4D spectro-polarimeter) vs. the normal 'FG' (filtergram) And then when we got back to 'required': and related to the required FITS headers -- a poster from the 2012 AAS meeting, talking about \*types\* of things that need to be in the FITS headers, without arguing for specific keywords : http://sdac.virtualsolar.org/docs/SPD2012/2012_SPD_FITS_headers.pdf - .. which later evolved into this checklist : http://docs.virtualsolar.org/wiki/MinimumInformation ... During the first presentation, there was an example of searching for events, then using those times to search for observations: I also should have mentioned -- the VSO looked into searching by events .... but then it went on hold (due to SDO) and was never finished. rather than just using the times from the events directly, we allowed the research to specify a 'window' ... stuff like '4 hrs before the event' to '1 hr after the event' ... I like the idea of specifying keywords that don't agree with solarnet [definitions] -- but most archives aren't willing to go back and modify headers, from my experience. We might need a separate file to refer to, instead. ... Kevin's talk, he mentioned drilling down type searches: - aggregation -- see the bottom of the 'Describing Catalogs' box on http://sdac.virtualsolar.org/catalogs/jhourcle_SPD_2009.pdf Just thought I'd share a link to the IDL VM solution: http://www.harrisgeospatial.com/docs/VirtualMachineApplicationsIntro.html It works very well, the only limitation is you can't compile anything in that mode, so you need a license for development. But "some people in SolarSoft" (won't name names) have a fondness for the execute("...") command, which constitutes "compilation" and is therefore not permitted in the VM mode. We never failed to find a way to rewrite those routines (locally), but gave up since there were too many new routines/changes that needed to be followed up. And we had enough licenses and multi-core machines (1 license per user/machine/display combination regardless of number of cores). We could've made a push to discourage people from using the execute function, but felt it wasn't really worth the hassle.