gpscope-DT UPS 855

Partenaires

menu

• Parc mobile
  • Les logiciels de communication et les notices d’utilisation
  • Les caractéristiques des antennes
  • Rapports d’activités et comptes rendus
  • Contrôle des antennes
• Chantiers pemanents hors de France
• Campagnes
• Documentation technique
  • Monumentation
  • foudre
  • Récepteurs
  • Capteurs météo
  • Accessoires
  • télécommunications
    • GPRS
  • Présentations techniques
  • traitement des données

Rechercher

Unlike campaign data, which can be treated at any opportune moment, the flow of data from permanent stations requires regular treatment. If data are not treated regularly, a backlog will accrue which can rapidly become several weeks-worth of uninterrupted calculation time. The solution described here makes it possible to obtain the graphs for time series on a daily basis as the data arrive.

The method and tools described below are based on the gamit/globk/glorg suite from MIT.

A basic rule which I have set myself in my developments is to never modify MIT programmes, even when it might appear to simplify things. The advantage is that MIT updates can be used without needing to re-insert (and adapt) any modifications.

When it is about automated treatment, whatever the procedures, it is wise to create a dedicated user for this application (here gpscope). This prevents any mistake from a “real life†users which could compromise the correct functioning of automated tasks.

The whole process relies on a structure in four main folders :
 the folder of data to be treated
 the calculation folder
 the data archiving folder
 the folder for the Web page devoted to the site

Data transfer

This step is specific to each data source. “Generic†data treatment can only start after this step, for which the procedures are necessarily specific. The only shared constraint for all these procedures is to bring the data, in a gamit-readable format (RINEX, Hatanaka, compressed or not), to the entry point for the next, generic, step.

This entry point is a “deposit†folder where all the new data for a site converge. In our case, we have the gpscope/deposit/cori folder for the Corinth network, gpscope/deposit/chil for Chile, etc.

I have developed several data transfer methods, which are reviewed rapidly here :

In the first case, the data arrive directly from the GNSS receiver. We then have a raw data file in the manufacturer’s format. We are therefore an “operations centre†and it is our duty to backup these raw files before translating them into rinex and using them for any calculations.

In the second case, we have direct access to RINEX data.

I first worked with shell scripts using ftp. All these data recovery scripts had to maintain a list of data which had already been downloaded. This list was compared every day to the list of data present on the remote site [1], so as to download only new files. The critical element with this method is the list of files already downloaded : any mistake can cause you to have to re-start right from the beginning.

Now, I prefer to compare the remote files to the contents of the local archive AND to that of the repository for data awaiting calculations. This is somewhat convoluted in shell script, but is easy to do in Perl thanks to the Net::FTP and POSIX modules.

The best case is that where we have ssh access to the remote machine. I then write a routine on the remote machine whichdayly creates a text file for each year archived. This file lists the RINEX for the year with their full access address (the direct output of ls ${annee}/*/rinex/*d.Z). This list can be retrieved by scp for transfer to the local machine (using the ~/.netrc file further simplifies script writing) and I copy the missing files, also by scp : no more ftp!

These procedures are reinforced with each new problem encountered. Interruption of the connection during downloading remains a problem to which there is no definitive solution. Even comparing file sizes may not be enough, as it depends on the types of remote and local file systems. An ASCII-mode ftp on a Windows machine can also add /r to each line...

To monitor retrieval, the sh_mail_new script, activated once daily, explores all the logs for all the retrieval scripts (they all start with sh_ftp_data) to look for lines such as “sitejjj0.aad.Z is a new file†. The script then sends an e-mail to a distribution list with a list of these files, for each script, indicating the date on which this information was gathered.

I have dealt with the case of raw data arriving “on foot†at the institute by creating a “bulk†folder to which anyone can write. All users can deposit data here, they then inform me of their deposit by e-mail. I take it upon myself to verify the format and include the data in the automated workflow. Even though I am not infallible, this procedure limits many of the risks of introducing “sand†into the mechanics.

Identification of the days to calculate

The first step in identifying the days to calculate consists in making an inventory of the calculation days corresponding to the data present in the deposit folder. The sh_newdata script returns a line containing all the (year, day) pairs found and copies all these data to a ./rinex sub-folder in the experiment folder.

As all the data are calculated as they arrive, some of them have been calculated with predicted or rapid orbits. sh_process_master explores the ./rinex sub-folder to look for data from two days ago (time needed to obtain rapid orbits) and from 23 days ago (worst-case scenario of time required to determine final orbits). If such data are present, these days are added to the list of calculations to be launched.

Finally, various infrastructure failures (electrical failures, network disk failures, server name failures, etc.) can hinder daily launch of these calculations. For this reason, sh_process_master maintains an orbit.log file in the experiment folder like this one :

This file lists (year, day) pairs which have not been treated with the final orbits. At this stage, every day beyond the 23 days indicated above is added to the list of calculations to be launched.

If the $yrdoy matrix, which contains all the (year, day) pairs to be calculated does not contain any elements, the script stops.

The longer a process executes for, the more chances there are of incidents interrupting it (electrical failure, disappearance of remote disks, machine restart, etc.). Over-frequent interruptions cause the $yrdoymatrix to exceed the maximum size allowed by the operating system (the classic error message is “Argument list too long†). I have ended up setting a hard-wired limit on the number of days treated in a single run. For the moment, I have set this limit to 24, and each network is treated once per day. To be sure that two successive launches never overlap, I stagger the launch of the procedure as a function of the duration of a daily calculation (once per day for Reunion Island, once every four days for Chile, etc.). To catch up on major delays without losing whole half-days, I have even ended up writing a procedure to test every 15 minutes whether the main procedure is already running in memory. If not, it launches it !

actual calculationf

After displaying the list of (year, day) pairs for which a calculation will be launched on-screen, sh_process_master launches sh_gamit for each of them (the corresponding day directory, if it already exists, is deleted before each calculation).

The screen output from sh_gamit is redirected to the experiment folder and the sh_gamit.log file (which thus corresponds only to the current calculation or the last calculation performed).

To give a minimum of flexibility to this generic procedure, sh_process_master looks in the experiment folder for a gamit.OPT file. This file can contain non-standard options to be added to the sh_gamit command line. For example, this is where I add the -rinex_ftpsites ign kreiz -ftp_prog wget options, which are specific for a given network, and the localisation of the machine performing the calculation inside the IPGP gateway [2] (NB, an IPGP-specific ftp_info file is also required).

Once the calculation has been performed, sh_process_master updates the orbit.log file and displays on-screen any lines of sh_gamit.log containing the words ERROR or FATAL along with the list of rinex files treated.

validity of calculations

The validity of any calculation is a huge question. I have chosen, at this stage, to stick modestly to a criterion that I consider purely computer-based : the existence of a file “o†indexed in “a†in the day directory.

If the file exists,
 the data are deleted from the deposit repository (only deleting them from this repository at this point is a safeguard against failures which could interrupt the process : these data remain in the “waiting room†until we are sure that they have been treated at least once).
 the various “h†files present (because of previous calculations) in ./glbf and ./gsoln and the globk files (.prt,.log,.gdl,.org) for the day are deleted. Thus, if the procedure is interrupted, the time series will not display points from previous calculations.
 the corresponding nav file is deleted from ./brdc
 If the calculation was performed with final orbits, the external RINEX files are deleted from ./rinex (identified from the ftprnx contained in the sites.defaults file).

If the file does not exist,
 the sh_gamit.log file is renamed by adding the Julian day as an extension (making it possible to check what happened later).
 the ./igs ./gfiles ./brdc sub-forlders and the day directory are deleted along with the ut1 and pole files from ./tables [3].

Cleaning up

At the end of all the daily gamit calculations, sh_clean_expe deletes all the temporary files which have accumulated in the various sub-folders. Going back over several years of calculations in one go can create a larger number of temporary files than the operating system can handle, triggering an "Argument list too long." message. But this should never occur with routine calculations ; human intervention is required to cause this type of overloading.

calculation of the time series

At every pass of the daily gamit calculation, sh_process_master updates the values for the oldest (year, day) pair taken into consideration. It is therefore capable of launching sh_glred over the period spanning from the oldest day calculated to the most recent present in the experiment folder.

As with sh_gamit, a minimum of flexibility is introduced by the glred.OPT file, which may be present in the experiment folder [4]. In its absence, the G E H options are taken as default. But the sh_plot_... scripts which follow make the E option in sh_glred unnecessary.

When a gamit calculation is interrupted (e.g. due to an electrical failure) a truncated h file may be generated. As a result, htoglb never finishes working on this file, which blocks both sh_glred and sh_process_master. To overcome this problem, before launching sh_glred I first verify whether a htoglb process is already running. In the affirmative, I kill it. This causes the sh_glred process linked to this htoglb to terminate and the previous instance of sh_process_master can continue (in this case, I have two sh_process_master running on the machine). After executing sh_glred I check that the last htoglb mentioned in the sh_glred.log effectively treated the last day indicated in the sh_glred command. If not, I delete the corresponding daily repository (to avoid the next calculation blocking on this h file again) and I transfer the rinex files for this day to the “deposit†repository. The h file will thus be regenerated by the next gamit calculation.

creation of “short-term†graphs

After destruction of all the gif or post-script files which may be present in ./gsoln, sh_plot_shortterm launches the tools for the gamit/globk suite to visualise the time series for the last 90 days’ of data (sh_plot_shortterm takes as arguments the two extreme (year, day) pairs for the range of days that we wish to view). The files generated are gif files.

creation of “long-term†graphs

After destruction of all the gif or post-script files which may be present in ./gsoln, sh_plot_longterm launches the tools for the gamit/globk suite on all the org files present in ./gsoln to obtain the complete time series currently available. The files generated are gif files.

I will not go into specificities of the gamit/globk tools here, but creation of a graph for a time series covering several years can exceed the system’s capacities depending on how the tools offered by MIT are used.

In both cases (short- and long-term), I have allowed the possibility to transfer options to the gamit tools (sh_plotcrd) through the plot.OPT file in ./gsoln [5]. Networks on volcanoes tend to generate larger error bars than other sites : I take this into account when choosing the various values for the maxsign and maxsigu parameters, which change the level of noise for the points allowed on the visualisation of time series.

archiving, publication of time series and making data available

sh_process_master first deletes all the images contained in the repository of the Web site dedicated to it, and then copies the new images of the time series to the repository. sh_process_master then launches sh_web_data which explores the ./rinex sub-folder and copies the data which are not already present to the team’s archive. This programme also creates various symbolic links to the Web site to make it possible to find these data for each site or date without increasing the number of files unnecessarily.

specific procedures

For some sites I was asked to supply additional graphs (e.g. the baseline for a site). To allow this flexibility I added the option of executing an sh_specific_name_experiment1 script just after sh_plot_shortterm, and an sh_specific_name_experiment2 script just after sh_plot_longterm. If this script is present in the experiment folder it will be executed by sh_process_master [6].

tools

Some tasks should be executed independently from the main calculation :
 sh_updt_tables : updates the gamit tables from the sopac and gamit sites. It works on all the gamit versions present [7] and finishes by re-writing all the station.info files for all sites (by combining ./tables.templates/station.info.EXPE with the latest station.info.sopac.newfmt. This allows me to effortlessly track the evolution of IGS stations).
 sh_clean_old : eletes all the daily repositories and rinex data older than a given number of days (currently 60) from the experiment folder (after verifying that they are present in the team’sarchive). This helps prevent disk saturation.
 sh_arch_gamres : tidies the o, h and q files present in the experiment folder into the data archive.

Automated management

The key to all this is the crontab for the machine performing calculations. This is an example of a machine which not only performs the calculations for three different networks, but also retrieves all the data for all the machines. Some calculation results are backed up (locally to save access time and avoid saturating the local network) every week to the network disk which is then backed up by the IT department.

For each script, the output is redirected towards a specific log file which can be used to retrace the events in case of problems. To avoid generating too many unreadable logs, all outputs except those of sh_process_master overwrite the previous output. I delete this file from time to time, e.g. when I have checked that no errors have occurred in recent weeks.

Evolution of procedures as part of a service offered

I wrote the first version of this presentation when I worked at IPGP which is what the IGS calls an “operational centre†, i.e., a place where we look after data retrieval from the field (whether after passing through an observatory or directly from “headquarters†). These data were only made available on the Web after calculations and this was the ultimate step in transferring these data.

Today, as a support service for the national GNSS community, I have to distinguish between two types of service that I can offer to operational centres :
 The availability on the Web and centralised archiving of INSU data in a broad sense.
 The automated calculation of data for some sites upon request from the researchers involved.

To distinguish between these two services, I have created a transit folder, with sub-folders for each site. Every day I compare what is contained in the archive for these operational centres with what I have already and I download any new files to my transit folder.
On the files present in this folder, I perform the following sequence :
 a copy to my archive
 creation of links from the Web site to the archive to make data available
 if necessary, copy of these files to the deposit repository, for calculations.
 removal of everything from the transit folder.

The transit folder is therefore empty most of the time. Archiving and making data available on the Web are no longer part of the standard calculation procedure !