5.4 The distribution files

5.4.7.1 Simplified information

This section tells you how to quickly prepare fine grained fetching of multiple distribution files and patches from different sites and subdirectories. We describe here a case of simplified MASTER_SITES:n usage. This will be sufficient for most scenarios. However, if you need further information, you will have to refer to the next section.

Some applications consist of multiple distribution files that must be downloaded from a number of different sites. For example, Ghostscript consists of the core of the program, and then a large number of driver files that are used depending on the user's printer. Some of these driver files are supplied with the core, but many others must be downloaded from a variety of different sites.

To support this, each entry in DISTFILES may be followed by a colon and a ``tag name''. Each site listed in MASTER_SITES is then followed by a colon, and the tag that indicates which distribution files should be downloaded from this site.

For example, consider an application with the source split in two parts, source1.tar.gz and source2.tar.gz, which must be downloaded from two different sites. The port's Makefile would include lines like Example 5-1.

MASTER_SITES=   ftp://ftp.example1.com/:source1 \
        ftp://ftp.example2.com/:source2
DISTFILES=      source1.tar.gz:source1 \
        source2.tar.gz:source2

Multiple distribution files can have the same tag. Continuing the previous example, suppose that there was a third distfile, source3.tar.gz, that should be downloaded from ftp.example2.com. The Makefile would then be written like Example 5-2.

MASTER_SITES=   ftp://ftp.example1.com/:source1 \
        ftp://ftp.example2.com/:source2
DISTFILES=      source1.tar.gz:source1 \
        source2.tar.gz:source2 \
        source3.tar.gz:source2

5.4.7.2 Detailed information

Okay, so the previous section example did not reflect your needs? In this section we will explain in detail how the fine grained fetching mechanism MASTER_SITES:n works and how you can modify your ports to use it.

1. Elements can be postfixed with :n where n is [^:,]+, i.e., n could conceptually be any alphanumeric string but we will limit it to [a-zA-Z_][0-9a-zA-Z_]+ for now.

   Moreover, string matching is case sensitive; i.e., n is different from N.

   However, the following words cannot be used for postfixing purposes since they yield special meaning: default, all and ALL (they are used internally in item ii). Furthermore, DEFAULT is a special purpose word (check item 3).

2. Elements postfixed with :n belong to the group n, :m belong to group m and so forth.

3. Elements without a postfix are groupless, i.e., they all belong to the special group DEFAULT. If you postfix any elements with DEFAULT, you are just being redundant unless you want to have an element belonging to both DEFAULT and other groups at the same time (check item 5).

   The following examples are equivalent but the first one is preferred:

   MASTER_SITES=   alpha
   
   MASTER_SITES=   alpha:DEFAULT
   

4. Groups are not exclusive, an element may belong to several different groups at the same time and a group can either have either several different elements or none at all. Repeated elements within the same group will be simply that, repeated elements.

5. When you want an element to belong to several groups at the same time, you can use the comma operator (,).

   Instead of repeating it several times, each time with a different postfix, we can list several groups at once in a single postfix. For instance, :m,n,o marks an element that belongs to group m, n and o.

   All the following examples are equivalent but the last one is preferred:

   MASTER_SITES=   alpha alpha:SOME_SITE
   
   MASTER_SITES=   alpha:DEFAULT alpha:SOME_SITE
   
   MASTER_SITES=   alpha:SOME_SITE,DEFAULT
   
   MASTER_SITES=   alpha:DEFAULT,SOME_SITE
   

6. All sites within a given group are sorted according to MASTER_SORT_AWK. All groups within MASTER_SITES and PATCH_SITES are sorted as well.

7. Group semantics can be used in any of the following variables MASTER_SITES, PATCH_SITES, MASTER_SITE_SUBDIR, PATCH_SITE_SUBDIR, DISTFILES, and PATCHFILES according to the following syntax:

   1. All MASTER_SITES, PATCH_SITES, MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR elements must be terminated with the forward slash / character. If any elements belong to any groups, the group postfix :n must come right after the terminator /. The MASTER_SITES:n mechanism relies on the existence of the terminator / to avoid confusing elements where a :n is a valid part of the element with occurrences where :n denotes group n. For compatibility purposes, since the / terminator was not required before in both MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR elements, if the postfix immediate preceding character is not a / then :n will be considered a valid part of the element instead of a group postfix even if an element is postfixed with :n. See both Example 5-3 and Example 5-4.

      Example 5-3. Detailed use of MASTER_SITES:n in MASTER_SITE_SUBDIR

      MASTER_SITE_SUBDIR=     old:n new/:NEW
      

      • Directories within group DEFAULT -> old:n

      • Directories within group NEW -> new

      Example 5-4. Detailed use of MASTER_SITES:n with comma operator, multiple files, multiple sites and multiple subdirectories

      MASTER_SITES=   http://site1/%SUBDIR%/ http://site2/:DEFAULT \
              http://site3/:group3 http://site4/:group4 \
              http://site5/:group5 http://site6/:group6 \
              http://site7/:DEFAULT,group6 \
              http://site8/%SUBDIR%/:group6,group7 \
              http://site9/:group8
      DISTFILES=      file1 file2:DEFAULT file3:group3 \
              file4:group4,group5,group6 file5:grouping \
              file6:group7
      MASTER_SITE_SUBDIR=     directory-trial:1 directory-n/:groupn \
                  directory-one/:group6,DEFAULT \
                  directory
      

      The previous example results in the following fine grained fetching. Sites are listed in the exact order they will be used.

      • file1 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site1/directory/

        • http://site1/directory-one/

        • http://site1/directory-trial:1/

        • http://site2/

        • http://site7/

        • MASTER_SITE_BACKUP

      • file2 will be fetched exactly as file1 since they both belong to the same group

        • MASTER_SITE_OVERRIDE

        • http://site1/directory/

        • http://site1/directory-one/

        • http://site1/directory-trial:1/

        • http://site2/

        • http://site7/

        • MASTER_SITE_BACKUP

      • file3 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site3/

        • MASTER_SITE_BACKUP

      • file4 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site4/

        • http://site5/

        • http://site6/

        • http://site7/

        • http://site8/directory-one/

        • MASTER_SITE_BACKUP

      • file5 will be fetched from

        • MASTER_SITE_OVERRIDE

        • MASTER_SITE_BACKUP

      • file6 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site8/directory-one/

        • MASTER_SITE_BACKUP

8. How do I group one of the special variables from bsd.sites.mk, e.g., MASTER_SITE_SOURCEFORGE?

   See Example 5-5.

   Example 5-5. Detailed use of MASTER_SITES:n with MASTER_SITE_SOURCEFORGE

   MASTER_SITES=   http://site1/ ${MASTER_SITE_SOURCEFORGE:S/$/:sourceforge,TEST/}
   DISTFILES=      something.tar.gz:sourceforge
   

   something.tar.gz will be fetched from all sites within MASTER_SITE_SOURCEFORGE.

9. How do I use this with PATCH* variables?

   All examples were done with MASTER* variables but they work exactly the same for PATCH* ones as can be seen in Example 5-6.

   Example 5-6. Simplified use of MASTER_SITES:n with PATCH_SITES.

   PATCH_SITES=    http://site1/ http://site2/:test
   PATCHFILES=     patch1:test
   

5.4.7.3 What does change for ports? What does not?

1. All current ports remain the same. The MASTER_SITES:n feature code is only activated if there are elements postfixed with :n like elements according to the aforementioned syntax rules, especially as shown in item 7.

2. The port targets remain the same: checksum, makesum, patch, configure, build, etc. With the obvious exceptions of do-fetch, fetch-list, master-sites and patch-sites.

   • do-fetch: deploys the new grouping postfixed DISTFILES and PATCHFILES with their matching group elements within both MASTER_SITES and PATCH_SITES which use matching group elements within both MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR. Check Example 5-4.

   • fetch-list: works like old fetch-list with the exception that it groups just like do-fetch.

   • master-sites and patch-sites: (incompatible with older versions) only return the elements of group DEFAULT; in fact, they execute targets master-sites-default and patch-sites-default respectively.

     Furthermore, using target either master-sites-all or patch-sites-all is preferred to directly checking either MASTER_SITES or PATCH_SITES. Also, directly checking is not guaranteed to work in any future versions. Check item iii.ii for more information on these new port targets.

3. New port targets

   1. There are master-sites-n and patch-sites-n targets which will list the elements of the respective group n within MASTER_SITES and PATCH_SITES respectively. For instance, both master-sites-DEFAULT and patch-sites-DEFAULT will return the elements of group DEFAULT, master-sites-test and patch-sites-test of group test, and thereon.

   2. There are new targets master-sites-all and patch-sites-all which do the work of the old master-sites and patch-sites ones. They return the elements of all groups as if they all belonged to the same group with the caveat that it lists as many MASTER_SITE_BACKUP and MASTER_SITE_OVERRIDE as there are groups defined within either DISTFILES or PATCHFILES; respectively for master-sites-all and patch-sites-all.