The DIR_CONFIG file

The AMG expects a configuration by default in the file $AFD_HOME_DIR/etc/DIR_CONFIG. This default can be changed in $AFD_HOME_DIR/etc/AFD_CONFIG with the parameter DIR_CONFIG_NAME. By inserting multiple DIR_CONFIG_NAME lines in AFD_CONFIG, you split up the content of the configuration over several files. The content should conform to a certain set of rules. All entries are grouped in sections, a section begins with the line [section name] and continues until the next section starts. Supported sections names and the corresponding entries are listed below and must be set in this order:

Comments start with # and may be placed anywhere except directly behind the files and destination identifiers.

The directory entry [directory]

The directory in which the AMG has to search for files which have to be send. Each directory must begin with / or a ~ and this directory must exists or the AMG must have read and write access so it can create the directory recursively. Otherwise it ignores the complete entry for this directory. The number of directory entries in the DIR_CONFIG is unlimited. However duplicate directory entries will be ignored. If you wish to give the directory an alias name you can do so by putting the name directly behind the directory entry: [directory] <dir alias>. AFD is also able to fetch files from a remote host, the directory must then be specified in URL format as follows: <scheme>://[[<user>][;fingerprint=<SSH fingerprint>][:<password>]@]<host>[:<port>][/<url-path>][;type=<i|a|d|f|n>][;protocol=<protocol number>]

The items are separated by special characters :, /, @ and ;. If these characters appear within one of the items, they must be masked with a \.

See recipient for a more detailed description on how to use the fingerprint parameter and others.

There are currently five schemes for fetching files supported:

Scheme Description Example
ftp Normal FTP transfers. ftp://donald:secret@ducktown/directory
ftp[sS] FTP transfers over secure sockets (TLS/SSL). When ftps is used only the control connection is encrypted. With ftpS both control and data connection are encrypted. ftps://donald:secret@ducktown/directory
sftp SFTP, file transfer via SSH. sftp://donald:secret@ducktown/directory
http Normal HTTP transfers. http://donald@ducktown/directory
https HTTP transfers over secure sockets (TLS/SSL). https://donald@ducktown/directory
exec Execute some external command to retrieve data. exec://ext_fetch/fetch_command

For remote FTP, SFTP and HTTP directories it is possible to insert a date or current hostname with the following modifiers:

Feature Description Example [directory] entry Resulting directory name
%tx Insert the actual time in any form. The character x determines the time format which can be one of the following:
   a - short day "Tue",           A - long day "Tuesday",
   b - short month "Jan",         B - long month "January",
   i - day of month [1,31],       J - month [1,12],
   d - day of month [01,31],      m - month [01,12],
   j - day of the year [001,366], y - year [01,99],
   Y - year 1997,                 R - Sunday week number [00,53],
   w - weekday [0=Sunday,6],      W - Monday week number [00,53],
   o - hour [0,23],               H - hour [00,23],
   M - minute [00,59],            S - second [00,60],
   U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC
       
ftp://daisy@ducktown/diary/%tY/%tm/%tH
 ftp://daisy@ducktown/diary/2006/04/01
%T[+|-|*|/|%]xS|M|H|d Time modifier for the above %tx option. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier as shown in the example. 
ftp://goofy@hollywood/%td_%T+1d%td_%T0%td
 ftp://goofy@hollywood/28_29_28
%h Insert the hostname of the current node. Depending on how the system is configured, this might return full network node hostname. 
ftp://donald@daisy/archive/%h/todo
 ftp://donald@daisy/archive/ducktown/todo

NOTE: The option [directory] must always start at the beginning of a line. There may not be any tabs or spaces in front.

The directory options [dir options]

With the options in the table below, it is possible to specify what the AFD should for example do with files that are in the directory but are not being distributed, how files are to be retrieved from a remote hosts, etc. [dir options] may only be specified once for each [directory] option. A full list of options for this optional section and the defaults is shown in the table below:

Option Description Local Dir Remote Dir Default
inotify <flag> This option uses a feature of the operating system that notifies the application when some change occurs in the monitored directory. It is much faster than scanning the directory for new files. For the flag option it possible to specify 1 for rename event only, 2 for close event only and 4 for create event (link) only. One can also combine the events. For example 3 for both rename and close event. Or 5 for rename and create event. Do not use this option together with: time, force reread, wait for, accumulate and accumulate size X    
delete unknown files [time in hours] Delete all files that are older then the number of hours specified with the <old file time> option and are not to be distributed. If the optional time in hours is specified it will override the value given in "old file time". If you specify a value of 0, it will be deleted after 120 (DEFAULT_TRANSFER_TIMEOUT) seconds. This can be overwritten by setting -1. Then the file will be deleted immediatly. Only use the -1 if you really know what you are doing. X X  
do not delete unknown files The opposite of the above option. X X X
delete queued files [time in hours] Delete all files queued on input that are older then the number of hours specified with the <old file time> option and are to be distributed. If the optional time in hours is specified it will override the value given in "old file time". X    
delete old locked files <time in hours> Delete all old locked files on input that are older then the number of hours specified. X    
delete unreadable files [time in hours] Delete all unreadable files that are older then the number of hours specified with the <old file time> option. If the optional time in hours is specified it will override the value given in "old file time". If you specify a value of 0, it will be deleted immediatly.   X  
report unknown files Any files found in this directory that are not being distributed and are older then the number of hours specified in the <old file time> option will be reported in the system log. X   X
do not report unknown files The opposite of the above option. X    
old file time <value in hours> The age of a file in hours before a file is being reported in the system log or being deleted. X   24
end character <decimal number> Sometimes some application may not be able to send files with a leading dot and then rename these files after they have send all data. With this option AFD will wait until the character specified with <decimal number> will appear a the last character in the file before it will grab the file. X    
create source dir When set and the directory does not exist, it will be created. This will override option CREATE_SOURCE_DIR in AFD_CONFIG. X X  
do not create source dir When set the source directory will not be created if it does not exist. This will override option CREATE_SOURCE_DIR in AFD_CONFIG. X    
important dir This marks the directory as important. This is useful when you have lots of directories that are to be monitored by the AMG and/or you have only a slow disk. The process dir_check will now check all important directories every DIR_CHECK_TIMEOUT (default: 30) second. This ensures that the important directories will at least be checked every DIR_CHECK_TIMEOUT second and not until all directories have been checked, which could take very long. X    
time <min hour dom month dow> With this option it is possible at what interval the directory is to be scanned. See time under options for a more detailed description of what each field means. There may be up to 12 (MAX_FRA_TIME_ENTRIES) time entries. X X  
time external Setting this will allow one to trigger the scan event externally via the command 'afdcmd -R'. X X  
do not get dir list Setting this option will disable the directory scanning for this directory. This can be used when directory listing are not possible or disabled because of security reasons. Currently this only works for HTTP. This also means that under [files] the full filenames without wildcards must be listed, since the protocol does not resolve any wildcards.   X  
do not remove By default when AMG will remove all files that it finds in the directory being scanned. With this option set it will NOT remove these files. X X  
store retrieve list [once] When the above option <do not remove> is set, it can happen that always the same files are being collected by the AMG. To avoid this situation this option can be set. Then AMG will keep a list of files and their modification time and there size of all files found during the last scan. So only when a file has changed will it be retrieved by the AMG. When the optional parameter 'once' is appended, files will be fetched only once, regardless if the size or modification time changes. This is can be useful for directories with lots of files and protocols where we need to poll each individual file for size and date. For SFTP the once parameter is not needed since all needed values are retrieved during the directory scan and no additional polling is required. However when set for SFTP it will only fetch the file once. X X  
priority <value> With this you can give the directory a priority. This option is only useful for remote directories, ie. from where the files have to be retrieved. 0 is highest priority and 9 is the lowest. X X 9
force reread Before AFD scans the directory for files it checks if the time has changed since the last scan. With this options set it will always read the directory. This can be useful for some broken filesystems that do not update this information correctly. Needless to say that this will create a higher system load. X   not set
force copy Normally AFD tries to move files from the source directory into AFD's local pool directory. Sometimes this needs to be set on a global filesystem that does not handle atomic operations. In this case, one can set this option which forces AFD to copy the files into the pool directory. Normally it is not necessary to set this option, since copying has a much higher overhead. X   not set
max process <value> Here you may limit the number of process that the AMG may fork to handle files for this directory. AMG will only fork a process when using the exec option, or some other option that can use a lot of time to process. X X 10
max errors <value> The number of errors that may occur before this source/directory will become red in dir_ctrl dialog or if an action is configured this will activate this action. X X 10
do not parallelize Normally when AFD fetches files from a remote site it tries to parallelize this. However, if you have many directories on a remote host that are all fetched via the same alias, it can take very long before AFD fetches files from the other directories. If you set this option it does not parallelize the fetching of one directory/source.   X not set
ignore size <[=|>|<] value> If the files in the directory have the given size they will be ignored. A greater or less then sign may also be used to ignore files which have a size greater or less then the given value. The unit entered is in bytes. X X  
ignore file time <[=|>|<] value> If the files in the directory have the given difference of the current time and the files modification time, then it will be ignored. The unit is in seconds. A greater or less then sign may also be used. For example specifying 
             ignore file time < 60
means it will ignore any file whose modification time is less then 60 second. This option is best used together with 'force reread', because the directory is only scanned when something happens (rename, delete or added a new file) in the directory.		 X X  
max files <value> The maximum number of files that are copied before they are send to the FD. This value is set so that the AFD starts sending files as soon as possible even when there are thousands of files waiting in the source directory to be send. Otherwise it might take to long to process all the files at once and time is lost that could be used to transmit data. X X 100
max size <value> The maximum size in Kilobytes that are copied before they are send to the FD. X X 100 MB
accept dot files Setting this option will enable AFD to retrieve files starting with a dot from a remote host. When downloaded the dot will be automatically removed. Use the trans_rename option if you want to keep the dot.   X  
wait for <file pattern> With this option it waits for the given file before it sends all the other files from this directory. X    
warn time <time in seconds> When no data has arrived in the directory for the given time it will warn in the RECEIVE_LOG. X X 0 (disabled)
accumulate <number of files> Waits for the given number of files before it accepts files from this directory. X    
accumulate size <number of bytes> Waits for the given number of bytes before it accepts files from this directory. X    
dupcheck[ <timeout in secs>[ <check type>[ <action>[ <CRC type>]]]] Performs a duplicate on files that come into this directory. The options have the following meaning:
Option Description Default
timeout Time in seconds when this CRC value is to be discarded from the database. 3600
check type What type of check is to be performed, the following values are possible:
1 On filename only.
2 File content.
3 Filename and file content.
4 On filename without last suffix.
5 On filename and size.
1
action What action is to be taken when we find a duplicate. The following values are possible:
24 Delete the file.
25 Store the duplicate file in the following directory: $AFD_WORK_DIR/files/store/<id>. Where <id> is the directory id.
26 Only warn in SYSTEM_LOG.
33 Delete and warn.
34 Store and warn.
24
CRC type What type of CRC check is to be performed. The following values are possible:
16 CRC32
17 CRC32C. This CRC algorithm computes better checksums and if your CPU supports SSE4.2 calculation is done in hardware and is faster.
16
X X  
keep connected <time in seconds> The number of seconds that the connection should be kept open after retrieving all files. Note this can also be set via Keep connected in edit_hc dialog.   X 0 (disabled)

The files entry [files]

In this section it possible which files are to be distributed from the directory specified in the directory section. Each file name may contain the wild cards '*' and/or '?' in any order. One may also list characters enclosed in []. A pair of characters separated by a hyphen denotes a range expression; any character that sorts between those two characters is matched. If the first character following the [ is a ! then any character not enclosed is matched.

If the file entry starts with a '!' it means NOT this file. It is important to put these entries first otherwise the '!' option is meaningless. Here follow some more additional options that one can insert into the file filter:

Feature Description Example [files] entry Resulting filter
%tx Insert the actual time in any form. The character x determines the time format which can be one of the following:
   a - short day "Tue",           A - long day "Tuesday",
   b - short month "Jan",         B - long month "January",
   i - day of month [1,31],       J - month [1,12],
   d - day of month [01,31],      m - month [01,12],
   j - day of the year [001,366], y - year [01,99],
   Y - year 1997,                 R - Sunday week number [00,53],
   w - weekday [0=Sunday,6],      W - Monday week number [00,53],
   o - hour [0,23],               H - hour [00,23],
   M - minute [00,59],            S - second [00,60],
   U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC
       
cloud_%tH-*-%tY.png
cloud_19-*-2006.png
%T[+|-|*|/|%]xS|M|H|d Time modifier for the above %tx option. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier as shown in the example. 
*.%td_%T+1d%td_%T0%td
*.28_29_28
%h Insert the hostname of the current node. Depending on how the system is configured, this might return full network node hostname. 
*.%h
*.ducktown

The number file mask entries is unlimited, however the maximum length of a single entry should not exceed 256 (FILE_MASK_STEP_SIZE) characters. There is also no limit to the numner of file groups.

The destination entry [destination]

This marks the beginning of the destination section which contains one recipient and one options section. There is no limit in the number of destination entries per files section.

The recipient entry [recipient]

In the recipient section you specify the user name, password, remote host and remote directory in the following URL format:

<scheme>://[[<user>][;fingerprint=<SSH fingerprint>][;auth=login|plain;user=<user name>][:<password>]@]<host>[:<port>][/<url-path>][;type=<i|a|d|f|n>][;server=<server name>][;protocol=<protocol number>]

The items are separated by special characters :, /, @ and ;. If these characters appear within one of the items, they must be masked with a \.

If you do not want to write the password in clear text into the DIR_CONFIG you can set it via the program set_pw. If you have an external tool that generates the DIR_CONFIG automatically and you want to write the encoded passwords, ie. not in clear text, it can be done specifiying the encoding with $<encode type>$ and then the encoded password (eg. $2$XRZ5WwxA). For the exact details please read the code or contact the author. Currently only 3 types are known: 0, 1 and 2. 0 is in clear plain text. 1 is the AFD internal version. And 2 is a simple XOR encryption with a key that you need to specify in $AFD_WORK_DIR/etc/.xor.key or by compiling it in with the configure option --with-xor-key=KEY.

The fingerprint= is only valid for scheme scp and sftp. To get the fingerprint from a host with OpenSSH enter the following command: 

                ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub

This will generate the fingerprint as shown in the following example: 

     1024 11:ff:22:ee:33:dd:44:cc:55:bb:66:aa:77:99:88:88 /etc/ssh/ssh_host_rsa_key.pub

This fingerprint is then added as follows in the recipient line: 

     sftp://donald;fingerprint=11-ff-22-ee-33-dd-44-cc-55-bb-66-aa-77-99-88-88:secret@hollywood//home/user

Another valid way of writing this is as follows: 

     sftp://donald;fingerprint=ssh-rsa-11-ff-22-ee-33-dd-44-cc-55-bb-66-aa-77-99-88-88:secret@hollywood//home/user

Note that the : are replaced by a minus -.

It is not necessary to specify the real hostname or IP number for host. In fact it might be much more user friendly to use an alias name here, since this will be the name displayed in the afd_ctrl dialog. This alias name is then resolved via the HOST_CONFIG file. However, the host must always be supplied.

When scheme is sftp or scp one can use the option protocol= to specify the SSH protocol version.

The server= option is only useful for the mailto scheme. With this option it is possible to set the mail server address. If it is not set the default is localhost (SMTP_HOST_NAME in protocols/smtpdefs.h). If it is set this will be the hostname that AFD will take as alias/real hostname.

The type= option has the following meaning:

character scheme Description
i ftp[s|S] Does no modification to the data and sends as data representation TYPE IMAGE.
  wmo In WMO header a BI is inserted to indicate binary message.
a ftp[s|S] Inserts a linefeed (decimal 13) to all newline characters (decimal 10), if the linefeed does not exist. As data representation TYPE ASCII is send.
  wmo In WMO header a AN is inserted to indicate ASCII message. Note AFD does not check if the following data does not have any binary characters.
d ftp[s|S] Inserts a linefeed (decimal 13) to all newline characters (decimal 10), if the linefeed does not exist. As data representation TYPE IMAGE is send.
f wmo In WMO header a FX is inserted to indicate FAX message. Note AFD does not check if the following data is a fax.
n ftp[s|S] Does no modification to the data and sends no data representation TYPE.

Another useful feature for the mailto scheme is the possibility to group several mail addresses into one group. Under recipient it is then only necessary to specify one group name and not each recipient each on a separate line. This can be done as follows: 

         [recipient]
	 mailto://$hotels@HOTELS

You then need to create a file in $AFD_WORK_DIR/etc/group.list with a list of mail addresses as shown in the following example: 

         [hotels]
	 manager1@hotel1.com
	 manager2@hotel2.com
	 manager3@hotel3.com
	 manager4@hotel4.com

In this case it is also possible to just write mailto://$hotels. In the afd_ctrl dialog it will then appear as hotels. The group name may not be longer then 40 characters.

The directory name may contain modifiers for which AFD will insert the current or a modified time, the current host name where AFD is running and/or the output of some program executed. The modifiers are as follows:

Feature Description Example [recipient] entry Resulting directory name
%tx Insert the actual time in any form. The character x determines the time format which can be one of the following:
   a - short day "Tue",           A - long day "Tuesday",
   b - short month "Jan",         B - long month "January",
   i - day of month [1,31],       J - month [1,12],
   d - day of month [01,31],      m - month [01,12],
   j - day of the year [001,366], y - year [01,99],
   Y - year 1997,                 R - Sunday week number [00,53],
   w - weekday [0=Sunday,6],      W - Monday week number [00,53],
   o - hour [0,23],               H - hour [00,23],
   M - minute [00,59],            S - second [00,60],
   U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC
       
ftp://daisy@ducktown/diary/%tY/%tm/%tH
 ftp://daisy@ducktown/diary/2006/04/01
%T[+|-|*|/|%]xS|M|H|d Time modifier for the above %tx option. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier as shown in the example. 
ftp://goofy@hollywood/%td_%T+1d%td_%T0%td
 ftp://goofy@hollywood/28_29_28
%h Insert the hostname of the current node. Depending on how the system is configured, this might return full network node hostname. 
ftp://donald@daisy/archive/%h/todo
 ftp://donald@daisy/archive/ducktown/todo
%e`<command to execute>` Insert the output of the executed command. 
ftp://donald@daisy/archive/%e`date "+%Y-%m-%d"`/todo
 ftp://donald@daisy/archive/2013-11-24/todo

There are currently seven schemes supported:

Scheme Description Example
ftp Normal FTP transfers. ftp://donald:secret@ducktown/directory
ftp[sS] FTP transfers over secure sockets (TLS/SSL). When ftps is used only the control connection is encrypted. With ftpS both control and data connection are encrypted. ftps://donald:secret@ducktown/directory
mailto Sending files via SMTP. mailto://daisy@ducktown
file Copies files to another directory. file://local/directory
exec Calls an external program to transmit the file to another destination. exec://printer/lpr -Pmy_printer %s
wmo Sends files via TCP according to WMO regulations at port 1111. wmo://@wmo-host:1111
sftp Sends files via SSH using the SFTP protocol. sftp://donald:secret@ducktown/directory
scp Sends files via SSH using the SCP protocol. scp://donald:secret@ducktown/directory

Each recipient section may have an unlimited number of recipients and each may not be longer then 256 (MAX_RECIPIENT_LENGTH) characters.

The options entry [options]

For each destination section you may specify one options section and may contain up to 15 (MAX_NO_OPTIONS) options each 256 (MAX_OPTION_LENGTH) characters long. All options specified here count for all recipients that are specified in the recipient section. Valid options can be taken from the following table:

Option name Description AMG ftp mailto file wmo sftp scp exec
priority x The priority of this job. Valid entries for x are 0-9, 0 is the highest priority and 9 is the lowest. (default 9) X X X X X X X X
delete Deletes all files. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
time <min hour dom month dow>

This option allows to enter cron like time jobs in the AFD. The five time and date fields in the time option specify when files are to be distributed, that is when the minute, hour, and month of year fields match the current time, and when at least one of the two day fields (day of month, or day of week) match the current time. The time and date fields are as follows:

time and date fields
minute
hour
day of month
month
day of week
allowed values
0-59
0-23
1-31
1-12
1-7

In the <day of week> field 1 is Monday and 7 is Sunday.

A field may be an asterisk, which always stands for all values in this field.

Ranges of numbers are allowed. The ranges consist of two numbers and a hyphen. There may not be any blanks! The specified range is inclusive. Example: "19-22"

Its also possible to list a set of numbers or even ranges. Examples: "1,2,4,15", "4-8,18-20"

Step sizes can be used after an asterisk. So to specify every five minutes, just use "*/5". They can also be used in combination with a range. For example, "0-30/5" would equal "0,5,10,15,20,25,30".

There may be more then one time option. The number of time option is not limited, its however limited to the total number of options 15 (MAX_NO_OPTIONS) that may be set per job.

X 
 
 
 
 
 
 
 
 
 
 
 
 
 
time no collect <min hour dom month dow> This option is the same as the above time option only that no files will be collected (these will be deleted) outside the given time frames. The number of time no collect option is not limited. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
basename[ overwrite] Sends only the basename of the file. For example if you have the following file name xxxx.yyyy.zzzz, it will rename the file on the remote site to xxxx. If the optional parameter overwrite is not set and the rename would overwrite another file, it will add a minus (-) and a number to the end of the filename so that no files will be overwritten as follows: xxxx-0. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
extension[ overwrite] Removes only the extension from the file name. Taking the example file name from above, the file name on the remote site would be xxxx.yyyy. If the optional parameter overwrite is not set and the rename would overwrite another file, it will add a minus (-) and a number to the end of the filename so that no files will be overwritten as follows: xxxx.yyyy-0. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
prefix add xx Adds a prefix to the file name. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
prefix del xx Remove a prefix from the file name. If the AMG does not find this prefix, this option is ignored. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
Option name Description AMG ftp mailto file wmo sftp scp exec
toupper Convert all letters of the file name to upper case. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
tolower Convert all letters of the file name to lower case. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
lchmod mode Changes the permission of each file before being distributed. Only a numeric mode from one to four octal digits (0-7) is accepted. The first digit selects the set user ID (4), set group ID (2) or the save text image (1) attributes. The second digit selects permissions for the user who owns the file: read (4), write (2) and/or execute (1). The third selects permissions for other users in the files group. The fourth digit is for all other users (world). X 
 
 
 
 
 
 
 
 
 
 
 
 
 
mirror source Deletes all files from destination not found in source. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
rename <rule> [overwrite] Rename files as defined in the 'rule'-section of the etc/rename.rule file. If the overwrite option is NOT set, multiple files with the same name will not be overwritten. This is done by appending a ; and a number to the file name that is renamed. The rule name itself may not be longer then 80 (MAX_RULE_HEADER_LENGTH) characters long. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
trans_rename <rule> [primary_only|secondary_only|dupcheck[see dupcheck options]] Rename files on the remote site after the files have been transmitted, as defined in the 'rule'-section of the etc/rename.rule file. This renaming may include directory names, however the directories must exist on the remote site. If the additional option primary_only is specified and the system has a secondary host specified, the renaming will only occur on the primary system. When secondary_only is set, the renaming will only happen on the secondary node. The rule name itself may not be longer then 80 (MAX_RULE_HEADER_LENGTH) characters long. If dupcheck is specified it will perform a duplicate check on the renamed file. 
 
 X X X 
 
 X 
 
 
 
force copy Force files to be copied and not linked nor memory mapped (Good for copying files to /dev/null). 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
mirror source Will delete files from target directory if they are not found in the source directory. This will only work if the [dir options] store retrieve list is set! Also note currently the implementation is incomplete in that it only deletes from destination when it finds files to fetch from source. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
chmod mode Changes the permission of each file distributed. Only a numeric mode from one to four octal digits (0-7) is accepted. The first digit selects the set user ID (4), set group ID (2) or the save text image (1) attributes. The second digit selects permissions for the user who owns the file: read (4), write (2) and/or execute (1). The third selects permissions for other users in the files group. The fourth digit is for all other users (world). 
 
 X 
 
 X 
 
 X X X
chown [user][:.][group] Changes the user and/or group ownership of files. If only a user name (or numeric user ID) is given, that user is made the owner of each given file, and the files group is not changed. If the user name is followed by a colon or dot and a group name (or numeric group ID), with no spaces between them, the group ownership of the files is changed as well. 
 
 
 
 
 
 X 
 
 
 
 
 
 
 
create target dir[ mode] Creates the target directory recursively if it does not exist. This will not work when the option Fast cd is set, since the directory will be created during the cd command. The optional mode can be used to set the permission of created directory. Only a numeric mode from one to four octal digits (0-7) is accepted. The first digit selects the set user ID (4), set group ID (2) or the save text image (1) attributes. The second digit selects permissions for the user who owns the file: read (4), write (2) and/or execute (1). The third selects permissions for other users in the files group. The fourth digit is for all other users (world). If the option CREATE_TARGET_DIR is set to YES in AFD_CONFIG, then this option does not need to be set since then it is the default. 
 
 X 
 
 X 
 
 X 
 
 
 
do not create target dir If the option CREATE_TARGET_DIR is set to YES in AFD_CONFIG, then it can be overridden with this option. 
 
 X 
 
 X 
 
 X 
 
 
 
Option name Description AMG ftp mailto file wmo sftp scp exec
tiff2gts Converts T4-coded TIFF files to T4-files having a GTS header and end. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
gts2tiff Converts T4-GTS files to TIFF-files. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
fax2gts[ <fax ID>]

Converts FAX to a GTS T4 file. The GTS header is taken from the filename, so this must be a typical "TTAAii CCCC YYGGgg" WMO header. Optionally one can specify the following fax ID's:

1 DFAX1062 1728 Pixels 100 - 2352 lines
2 DFAX1064 1728 Pixels 100 - 2352 lines
3 DFAX1074 3456 Pixels 100 - 2352 lines
4 DFAX1084 2432 Pixels 100 - 3120 lines
5 DFAX1099 4752 Pixels 100 - 3040 lines

If no fax ID is specified it will take by default 2.

 X 
 
 
 
 
 
 
 
 
 
 
 
 
 
grib2wmo[ CCCC] Converts GRIB to WMO files with "TTAAii CCCC YYGGgg" headers, SOH and ETX at start and end and 10 Byte ASCII length and type indicator. You can optionally supply the CCCC if you do wish to overrule the one from the GRIB. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
wmo2ascii wmo2ascii will remove SOH, ETX and the two carriage returns from the given WMO file. It will convert the following text from: 
          <SOH><CR><CR><LF>nnn<CR><CR><LF>
	  WMO header<CR><CR><LF>WMO message<CR><CR><LF><ETX>
to 
          WMO header<LF>WMO message<LF>
in addition <CR><CR><LF> will be changed to <LF>. The file name will not be changed.		 X 
 
 
 
 
 
 
 
 
 
 
 
 
 
afw2wmo afw2wmo will convert the old AFW format used at DWD to WMO bulletin format. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
assemble <type>[ <file name>] This options assembles all files to a single file. The original files will be deleted. To help extraction of the files a length indicator may be inserted. The following types can be assembled:
VAX Two byte length indicator.
LBF Four byte length indicator with low byte first.
HBF Four byte length indicator with high byte first.
DWD Four byte length indicator DWD standard, that is like HBF but with four NULL bytes at end.
MSS Four byte length indicator MSS standard.
WMO Eight byte ASCII length indicator plus two byte type indicator (00 - with SOH and ETX, 01 - no SOH and ETX) . This is the current WMO standard.
WMO+DUMMY Same as WMO above but with a dummy length indicator at end of file.
ASCII No length indicator. The data of the next file is always appended, no matter if it contains binary or ascii data.
The <file name> may contain a %n to insert a unique number four characters long and/or with the help of %tx insert the actual time, where character x may be one of the following: 
         a - short day "Tue",           A - long day "Tuesday",
         b - short month "Jan",         B - long month "January",
         d - day of month [01,31],      m - month [01,12],
         j - day of the year [001,366], y - year [01,99],
         Y - year 1997,                 R - Sunday week number [00,53],
         w - weekday [0=Sunday,6],      W - Monday week number [00,53],
         H - hour [00,23],              M - minute [00,59],
         S - second [00,60]
         U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC

With the help of the %T[+|-|*|/|%]xS|M|H|d time modifier it is possible to modify the current time. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier.

If no file name is specified the default will be no_file_name. 		X 
 
 
 
 
 
 
 
 
 
 
 
 
 
Option name Description AMG ftp mailto file wmo sftp scp exec
convert <type> This options converts all files from one format into another. The original file will be overwritten with the new one. The following types can be converted:
sohetx[+nnn] Adds <SOH><CR><CR><LF> to the beginning of the file and <CR><CR><LF><ETX> to the end of the file. Optionaly one can add with the plus sign a sequence number. The maximum of digits is specified by the number of n's. So for example +nnn would add a three digit sequence number.
wmo[+nnn] Just add WMO 8 byte ascii length and 2 bytes type indicator. If the message contains <SOH><CR><CR><LF> at start and <CR><CR><LF><ETX> at the end, these will be removed. Optionaly one can add with the plus sign a sequence number. The maximum of digits is specified by the number of n's. So for example +nnn would add a three digit sequence number.
sohetxwmo[+nnn] Adds WMO 8 byte ascii length and 2 bytes type indicator plus <SOH><CR><CR><LF> at start and <CR><CR><LF><ETX> to end if they are not there. Optionaly one can add with the plus sign a sequence number. The maximum of digits is specified by the number of n's. So for example +nnn would add a three digit sequence number.
sohetx2wmo0[+nnn] converts a file that contains many ascii bulletins starting with SOH and ending with ETX to the WMO standard (8 byte ascii length and 2 bytes type indicators). The SOH and ETX will be copied to the new file. Optionaly one can add with the plus sign a sequence number. The maximum of digits is specified by the number of n's. So for example +nnn would add a three digit sequence number.
sohetx2wmo1[+nnn] Same as sohetx2wmo0 only that the SOH and ETX will NOT be copied to the new file.
mrz2wmo Converts GRIB, BUFR and BLOK files to files with 8 byte ascii length and 2 bytes type indicator plus <SOH><CR><CR><LF> at start and <CR><CR><LF><ETX> to the end, for the individual fields.
iso8859_2ascii Converts ISO8859 to ASCII. Curently only the following conversions have been implemented: 
                ä (e4) -> ae    Ä (c4) -> Ae | AE | A E
                ö (f6) -> oe    Ö (d6) -> Oe | OE | O E
                ü (fc) -> ue    Ü (dc) -> Ue | UE | U E
                ß (df) -> ss
Please contact the author if more conversions are required.
unix2dos Converts text files UNIX format to DOS format.
dos2unix Converts text files in DOS format to UNIX format.
lf2crcrlf Converts a single <LF> or a <CR><LF> to a <CR><CR><LF>.
crcrlf2lf Converts <CR><CR><LF> to a single <LF>.
X 
 
 
 
 
 
 
 
 
 
 
 
 
 
extract [-b|-B|-c|-C|-d|-D|-f|-F|-H|-n|-N|-r|-R|-s|-S|-t|-T] <type> [<pattern>] This options extracts files with WMO bulletins into one file per bulletin and will delete the original file. Since it is not possible to automatically detect the file type and its length encoding, you to specify what type of file it is. To indicate the length of the bulletin there is either a two or four byte length indicator at the beginning of the following format:
VAX Two byte length indicator.
LBF Four byte length indicator with low byte first.
HBF Four byte length indicator with high byte first.
MSS Four byte length indicator MSS standard.
WMO Eight byte ASCII length indicator plus two byte type indicator. This is the current WMO standard.
WMO+CHK Eight byte ASCII length indicator plus two byte type indicator plus a check if the length is correct.
ASCII No length indicator. The start of a bulletin is marked by SOH (binary 1) and end is ETX (binary 3).
BINARY This option is only useful together with the -H option to cut away SOH (binary 1), WMO header and the end ETX (binary 3). This only works if there is only one data set.
ZCZC No length indicator. The start of a bulletin is marked by ZCZC and end is NNNN.
MRZ Special four byte length indicator used in the DWD. The resulting file name will have the following name: <data type>_<originator>_YYYYMMDDhhmmss_<hexadecimal counter>
GRIB Standard GRIB, however since there are so many different GRIB types and limited name space for WMO header, many GRIB's will overwrite each other.

The file name of the new file will always be the WMO-header, that is: TTAAii_CCCC_[<year><month>]YYGGgg[_BBB][_<station indicator>]-<CRC checksum> The year will be shown with centry and month will always be shown as a two digit number. So March will be shown as 03. The year and month will only be added when the -d parameter is supplied.

The optional options have the following meaning:
      -b     extract individual reports
      -B     do not extract individual reports [default]
      -c     adds a CRC checksum to end of filename [default]
      -C     do not add a CRC checksum to end of filename
      -d     add full date to bulletin header in filename
      -D     do not add full date to bulletin header in filename
      -f     add the full bulletin name and the original filename
             where this bulletin was extracted from as the first
             line of the bulletin/report
      -F     the opposite of the above option
      -H     remove WMO header line including SOH and ETX
      -n     adds a four digit unique number to end of filename
      -N     do not add a four digit unique number to end of filename [default]
      -r     generate file name with report type
      -R     generate file name without report type [default]
      -s     add SOH and ETX if not there [default]
      -S     remove SOH and ETX
      -t     in report add additional header showing report type
      -T     in report do not show the additional header with report type [default]

If a pattern is supplied, only those files will be extracted that match the given pattern.

X 
 
 
 
 
 
 
 
 
 
 
 
 
 
Option name Description AMG ftp mailto file wmo sftp scp exec
exec [-d|-D|-l|-L|-s|-t] <command> Executes a command specified in <command> for each file to be distributed. The file name can be supplied with %s. There may be up to 10 places in the command where you can insert the file name. The optional options have the following meaning:
          -d                  Deletes the original file after executing
                              the command regardless if the command
                              was successful or not.
          -D                  When an error occurs during execution all
                              files will be deleted. This only works if
                              no %s is found in the command string
                              otherwise it will be ignored.
          -l                  This sets a lock so that only one process
                              can be executed at one time for each file.
                              If %s is being used the lock is only set
                              for the execution duration of that file,
                              this gives other jobs a chance but only
                              one can do the execution at one time.
          -L                  Some as above only that the lock is held
                              for the duration of all files when %s is
                              being used.
          -s                  On error save the original to the directory
                              $AFD_WORK_DIR/files/store/<creation time>_
                              <unique number>_<split job counter>
          -t <time in second> If after the given timeout the execution
                              has not finished the process will be killed.
X 
 
 
 
 
 
 
 
 
 
 
 
 
 
execd <command> Same as the exec option above, but deletes the original file after executing the command. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
execD <command> Again this is similar to the exec option and differs in that when an error occurs during execution that all files will be deleted. This only works if no %s is found in the command string, otherwise it will behave just like the exec option. X 
 
 
 
 
 
 
 
 
 
 
 
 
 
pexec [-l|-L|-t] <command> Executes a command specified in <command> for each file after it was distributed. The file name can be supplied with %s. There may be up to 10 places in the command where you can insert the file name. The files stored in the archive will NOT be effected by this option. The optional options have the same meaning as those above in the exec option. 
 
 X X X X X X X
[u]lock <type>
DOT When transmitting file "xxxx.yyyy" it creates the file on the remote site as ".xxxx.yyyy". When the file has been completely transmitted it is renamed to its original name. This ensures that the remote user can read the file when it is complete.
DOT_VMS When transmitting file "xxxx.yyyy" it creates the file on the remote site as ".xxxx.yyyy". After transmission this file is renamed to "xxxx.yyyy.". This ensures that the remote user can read the file when the transfer is complete.
OFF No locking on the remote site.
LOCKFILE [file name] Before starting transmission of the actual data files, the lock file "CONNECTED______.LCK" is send or if given the file name after LOCKFILE is used. When all files have been send successful this file is removed again.
xyz. When transmitting file "xxxx.yyyy" it creates the file on the remote site as "xyz.xxxx.yyyy". This ensures that the remote user can read the file when the transfer is complete.
If this option is not set the default is lock DOT. If ulock is used instead of lock, a unique number is added to the end of the filename. If you for example set 'ulock DOT' and taking the above example name "xxxx.yyyy", it is first transmitted with the name ".xxxx.yyyy.zzz", where zzz is some unique number.		 
 
 X 
 
 X 
 
 X 
 
 
 
lockp .zz When transmitting file "xxxx.yyyy" it creates the file on the remote site as "xxxx.yyyy.zz". This ensures that the remote user can read the file when the transfer is complete. 
 
 X 
 
 X 
 
 X 
 
 
 
sequence locking When transmitting file "xxxx.yyyy" it adds the number of retries to the end of the file name "xxxx.yyyy-<retries>" during transmission. After the file was transmitted successful it is renamed to the original name "xxxx.yyyy". Note, when set appending will NOT be possible. This option is useful when the remote side has problems with still having this file open by some other process, since with every try it will open a new file with a different name. It will try to delete the previous before it opens a new one. This option can be combined with the other locking options. 
 
 X 
 
 
 
 
 
 X 
 
 
 
Option name Description AMG ftp mailto file wmo sftp scp exec
archive x[d|h|m|s] Tells the AFD to archive files to this user for x days if no unit [d|h|m|s] is specified. The meaning of the units is as follows:
d Days
h Hours
m Minutes
s Seconds
When no archive option is set, the default (0 = no archiving (DEFAULT_ARCHIVE_TIME)) is set. With afdcfg the system administrator can disable archiving completely for all jobs.		 
 
 X X X X X X X
age-limit x Files older then x seconds will not get send. They will be removed. 
 
 X X X X X X X
match size After file is transmitted check the size of the destination file with that of the local file just send. If they do not match, show an error and then resend the file. 
 
 X 
 
 
 
 
 
 X 
 
 
 
no log output When set each file name distributed will NOT be logged to OUTPUT_LOG. 
 
 X X X X X X X
file name is subject The subject of the mail is the file name. This can only be used when sending mails. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
file name is user[ rename rule] The user name of the mail address (the first part up to the @ sign) is the file name. This can only be used when sending mails. The optional rename rule allows to filter or create the user name from the file name. For a detailed description of the renaming procedure see rename rule. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
file name is target[ rename rule] The full recipient of the mail address is the file name. So the file name must be a fully qualified mail address. This can only be used when sending mails. The optional rename rule allows to filter or create the user name from the file name. For a detailed description of the renaming procedure see rename rule. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
mail header[ <full path to mail header file>|rename rule] Is used to insert some default information at the top of the main body of the mail. If the option is used without the optional mail header file, it will take the file MAIL-host_alias in the $AFD_WORK_DIR/etc directory as the header file. If the optional mail header fail does not start with a / it take this as a rename rule identifier. For a detailed description of the renaming procedure see rename rule. Note that in this case if the resulting file name does not start with a / it will search for this file in $AFD_WORK_DIR/etc/mail_header. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
attach file[ rename rule] With this option it is possible to attach a file to a mail. Each file will be attached to a separate mail (one file per mail). The encoding will be done in BASE64 according to RFC 1521. If a rename rule is specified the filename will be renamed in the attachment. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
attach all files[ rename rule] With this option it is possible to attach files to a mail. If more than one file is available they will all be attached to ONE mail. The encoding will be done in BASE64 according to RFC 1521. If a rename rule is specified the filename will be renamed in the attachment. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
show all group members When using groups the name of each member is not shown in the To: line. It will just show the group name. When this option is set it will show all group names. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
Option name Description AMG ftp mailto file wmo sftp scp exec
subject <full path to subject file> [rename rule] Adds the subject in the <subject file> to the mail. It is possible to insert the current date or filename, for more details see the next option. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
subject "xxx"[ rename rule] Adds the line xxx after the subject identifier to the mails subject. Note that only 7-bit ASCII characters are allowed in the subject. It is possible to insert the current date in any form with %tx where character x determines the time format which can be one of the following:
            a - short day "Tue",           A - long day "Tuesday",
            b - short month "Jan",         B - long month "January",
            d - day of month [01,31],      m - month [01,12],
            j - day of the year [001,366], y - year [01,99],
            Y - year 1997,                 R - Sunday week number [00,53],
            w - weekday [0=Sunday,6],      W - Monday week number [00,53],
            H - hour [00,23],              M - minute [00,59],
            S - second [00,60]
            U - Unix time, number of seconds since 00:00:00 01/01/1970 UTC

With the help of the %T[+|-|*|/|%]xS|M|H|d time modifier it is possible to modify the current time. It adds (+), subtracts (-), multiplies (*), divides (/) or computes the modulo (%) with the value x to the current unix time. The default unit for the time modifier is seconds this can be changed to minute (M), hour (H) or day (d). Once set the modifier is used for all subsequent %t until it changed by the next modifier.

Additionally it is possible to add the filename with %s. If more then one file is send it will send a comma separated list with all the filenames. The maximum length of the subject line depends on the Transfer Blocksize and/or Transfer rate limit, whichever is larger. Should the subject exceed this length, the subject is just truncated. 

 
 
 
 X 
 
 
 
 
 
 
 
 
 
from <mail address> This allows to change the From: line of a mail. This can also be set in AFD_CONFIG with DEFAULT_SMTP_FROM globally. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
reply-to <mail address> This adds Reply-To: header to a mail with another mail address to which the recipient should reply. This can also be set in AFD_CONFIG with DEFAULT_SMTP_REPLY_TO globally. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
hide all group members When using mail groups, this option can be used to hide all group members of the group (BCC). Only the first member of the group will be shown in the To: line. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
charset <character set> This adds a MIME header, sets the transfer encoding to 8bit and specifies the character set. It does NOT check if the character set specified is a valid one. Possible values could be: 
          ISO-8859-1     --> Latin 1; Western European Languages
	  ISO-8859-2     --> Latin 2; Slavic/Central European Languages
	  ISO-8859-3     --> Latin 3; Esperanto, Galician, Maltese, Turkish
	  ISO-8859-4     --> Latin 4; Estonian, Latvian, Lithuanian
	  ISO-8859-5     --> Cyrillic
	  ISO-8859-6     --> Arabic
	  ISO-8859-7     --> Modern Greek
	  ISO-8859-8     --> Hebrew
	  ISO-8859-9     --> Latin 5; Turkish
	  ISO-8859-14    --> Latin 8; Celtic
	  ISO-8859-15    --> Latin 9; Western European Languages with Euro
       
 
 
 
 X 
 
 
 
 
 
 
 
 
 
encode ansi This converts ASCII to ISO8859 when mailing the content. Note not all characters are converted, mainly only the german characters are working. Please contact developer if you need more. 
 
 
 
 X 
 
 
 
 
 
 
 
 
 
Option name Description AMG ftp mailto file wmo sftp scp exec
file name is header The file name is send as WMO header. This can only be used when sending files via TCP according to WMO regulations. The file name must then be of the following format: TTAAii_CCCC_YYGGgg[_BBB]. 
 
 X 
 
 X X X X 
 
sequence numbering Puts in a three digit sequence number into the WMO header. 
 
 
 
 
 
 
 
 X 
 
 
 
 
 
check reply According to the WMO protocol messages are send as a continuous stream without any form of handshaking. When this option is set AFD waits for an acknowledge for each message its sends. The acknowledge consists of the following 10 characters: 00000000AK for a positive acknowledge or 00000000NA for a negative acknowledge. 
 
 
 
 
 
 
 
 X 
 
 
 
 
 
mode <active | passive> There two ways to communicate with another FTP server, either active or passive. Active mode is the more common one while passive is more secure. If no mode is specified the default will be active. NOTE: It's much better to use the HOST_CONFIG to set active or passive mode. 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
login site <command> This option is used to execute the command once after login. 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
site <command> This option is used to execute the command after each file is transmitted. 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
file busy rename <character> Appends the given character when we fail to open the remote file due to some broken FTP server that will not allow to open it because a hanging child still has this file open. 
 
 X 
 
 
 
 
 
 
 
 
 
 
 
dupcheck[ <timeout in secs>[ <check type>[ <action>[ <CRC type>]]]] Performs a duplicate on all files for this job. The options have the following meaning:
Option Description Default
timeout Time in seconds when this CRC value is to be discarded from the database. 3600
check type What type of check is to be performed, the following values are possible:
1 On filename only.
2 File content.
3 Filename and file content.
4 On filename without last suffix.
5 On filename and size.
1
action What action is to be taken when we find a duplicate. The following values are possible:
24 Delete the file.
25 Store the duplicate file in the following directory: $AFD_WORK_DIR/files/store/<id>. Where <id> is the job id.
26 Only warn in SYSTEM_LOG.
33 Delete and warn.
34 Store and warn.
24
CRC type What type of CRC check is to be performed. The following values are possible:
16 CRC32
17 CRC32C. This CRC algorithm computes better checksums and if your CPU supports SSE4.2 calculation is done in hardware and is faster.
16 
 
 X X X X X X X
socket send buffer <buffer size> Specifies the socket send buffer size in bytes. 
 
 X 
 
 
 
 
 
 X 
 
 
 
socket receive buffer <buffer size> Specifies the socket receive buffer size in bytes. 
 
 X 
 
 
 
 
 
 X 
 
 
 
eumetsat <DestEnvId> This option creates the header and sub header for EUMETSAT according to "MSG Ground Segment Design Specification Volume F". You might also need to execute create_eumetsat_name to create the correct file name. Syntax of create_eumetsat_name is as follows: 
       create_eumetsat_name <file name>[ <rename rule>]
       
 
 X 
 
 
 
 
 
 X 
 
 
 
restart <file> Append file on remote site. This option is for internal use only! 
 
 X 
 
 
 
 
 
 X 
 
 
 
Option name Description AMG ftp mailto file wmo sftp scp exec
Copyright © 1997 - 2014 by H.Kiehl
Holger.Kiehl@dwd.de
Last updated: 02.12.2014
[red dot]Index [red dot]Home