#+TITLE: Packages in Common Lisp, a tutorial #+AUTHOR: Francis Sergeraert #+DATE: January 2014 * Introduction. It is not possible to write a serious piece of code without the use of some kind of /identifiers/ or names. Such identifiers are convenient means for the programmer to reference, process and modify data using more or less descriptive names, which represent the data and depend in some way on the context. The standard /modularization/ technology encourages the division of large programs in /modules/. Such modules can be relatively independent of each other, but conversely, they must be able to /communicate/. Most programming languages offer methods for such an organization. For example, the set of identifiers of a module could be divided into two parts, the /private/ identifiers, known only inside this module, and the /public/ identifiers, known everywhere. In this domain as in many others, Common Lisp is by far the most powerful and flexible programming language. At the expense of considerable complexity, Common Lisp's modular technology requires a good grasp of the corresponding /definitions/ in the language standard. If the ANSI definition of the language is the only available documentation, these definitions are not easy to understand. An identifier in Lisp is a /symbol/ which has a rich internal structure and which has lifetime and visibility during a Lisp session. The modular structure of a Lisp program is defined through the /packaging/ system, which allows a programmer to divide a large program into several files, where each file is aware of all the symbols of a certain package, among them the current /default/ package, and only the /external/ (public) symbols of some other packages. A symbol which is not external in a package is /internal/ (private). Also, a package may /import/ (that is, "see") an arbitrary collection of symbols, internal or external to other packages. In the general spirit of Lisp, the whole structure of such an organization, if necessary, can be modified dynamically, even though this is not usually the best course of action. In this tutorial, a /mathematical/ definition of this system is provided. This should elucidate the process for many a Lisp programmer and enable her to use it more effectively. A basic but /inexact/ description of the packaging system is the following: - A /package/ is a Lisp object in the same way that numbers, lists, and character strings are Lisp objects. In particular, the /type/ of a package object is the system class [[http://clhs.lisp.se/Body/t_pkg.htm][=package=]]. In Common Lisp, packages are first class citizens, which means they can be the value of a symbol, can be passed as an argument to a function, can be returned by a function, etc. - A /symbol/ is a Lisp object whose /type/ is the system class [[http://clhs.lisp.se/Body/t_symbol.htm][=symbol=]]. The comments made about packages apply for symbols. In particular, a symbol can be the value of another symbol, or even of itself. - Symbols are divided into packages. - At any time during a Lisp session, some package is the /default package/ and, as a first approximation, only the symbols of this package are /accessible/. The default package can be modified at any time by the user or a program. - If necessary, there are several methods to access the symbols of packages other than the default package. Conversely, a package may make its symbols more or less easily accessible from the other packages. This first approximation of the packaging system conveys a general idea of its organization, but general ideas can be deceiving. For example, the idea that the symbols are divided into packages is essentially /false/. More precisely, this division can be made to appear essential (!), via the so-called home package of a symbol, but it plays only a /minor/ role and can be ignored by the programmer risk-free. This notion of a home package is terribly misleading for beginners when they have to design non-trivial packaging structures. In a Lisp environment, one has to deal with two realms, the space of symbols and the space of packages. It is only /inside/ of packages that something can be read, which, as an organization, looks a /little/ like a division of symbols into packages, but the symbols themselves essentially /ignore/ this division. Our subject is to clarify and explain this organization. The ANSI definition of the packaging system is a /complete/ description of this organization, and, in a sense, nothing is missing. However, experience shows that it is easily misunderstood, especially, in non-trivial situations, where it can lead to erroneous programming choices and mysterious bugs. The ANSI definition of packaging is complete and it is the perfect /Reference Manual/. However, for a topic of this complexity a reference guide is not sufficient and a /User Guide/ might be useful. And for that matter, an elementary /mathematical/ description of the Common Lisp packaging system might help programmers to understand it, and then to use it with greater ease and also to greater effect. In every domain, when precise mathematical language can be employed and is understood, the state-of-affairs improves. We begin with a review of the ANSI definition of the packaging system while adding in parallel a mathematical description of the main concepts. Finally, we give a complete mathematical definition of this system as a set of mathematical formulas. * Main sets of objects. The first part of the packaging system is made of three disjoint sets: - The set $St$ of *strings*. - The set $P$ of *packages*. - The set $Sm$ of *symbols*. In our description of the packaging system, the statuses of these sets are different. Think of the (infinite) set $St$ of /strings/ as made of /all/ the character strings that /can/ be used by a user or a program at any time. Any string can be used at any time, whatever the history of the Lisp session in which we are working may be. In the packaging system, most often, a particular string is used only for a short time, mainly during the read-eval-print loop, when the Lisp reader works. A string is just an intermediary connecting the user and the Lisp session, more precisely the current environment. In contrast, in a given environment, at any time during a Lisp session, some finite set $P$ of /specific/ packages is installed in this environment, and also a finite set $Sm$ of /specific/ symbols is available. Those packages and symbols can /then/ be used, and they are the historical consequence of the current Lisp session. Various Lisp functions allow the user to modify the sets $P$ and $Sm$, and the way they are combined and related to each other. In other words, consider the set of strings $St$ as constant, vast and independent of the environment, while, on the contrary, the sets $P$ and $Sm$ are restricted, they change during a Lisp session, and they are an essential component of the /current/ environment. * Packages There is a simple, almost bijective (one-to-one), correspondence between strings and packages, and a convenient place to start our subject. First, at any time during a Lisp session, a /default package/ is defined and it can be discovered via the global symbol [[http://clhs.lisp.se/Body/v_pkg.htm][=*package*=]]: #+BEGIN_SRC lisp :session pcl :exports both *package* #+END_SRC This is to be read as: the /value/ of the /symbol/ =*package*= is the package whose /name/ is the string ="COMMON-LISP-USER"=. The types =symbol= and =package= are system-defined, disjoint sets of machine objects. A symbol is frequently used as a convenient intermediate object, which allows the user to reach some object of arbitrary nature, its /value/. Here, =*package*= is a symbol, the value of which is the package named ="COMMON-LISP-USER"=. Technically, a symbol contains a pointer to its (possible) value, here a package which happens to be the package used ordinarily in a simple Lisp session. There are no facilities to refer to a package object /directly/. However, one can use its /name/ and the function [[http://clhs.lisp.se/Body/f_find_p.htm][=find-package=]] to obtain a reference: #+BEGIN_SRC lisp :session pcl :exports both (find-package "COMMON-LISP-USER") #+END_SRC Conversely, given a package, the function [[http://clhs.lisp.se/Body/f_pkg_na.htm][=package-name=]] returns its name: #+BEGIN_SRC lisp :session pcl :exports both (package-name (find-package "COMMON-LISP-USER")) #+END_SRC The example was given to illustrate that the functions =find-package= and =package-name= are the inverse of each other, defining a 1-1 correspondence between the packages /currently/ defined in the environment and their respective names, some character strings. Because of rules not of interest here, it is common and convenient to use only uppercase character strings to name packages. If a name does not correspond to any current package, the =find-package= function returns the symbol =nil=, usually displayed uppercase: #+BEGIN_SRC lisp :session pcl :exports both (find-package "COMMON-LISP-user") #+END_SRC which illustrates that a package name is case-sensitive. Frequently, only a few packages are in use, and because of their importance, descriptive names are used. These descriptive names can be a little long, and it is possible to define and use various /nicknames/ for these names. For example, the ="COMMON-LISP-USER"= package usually has the nicknames ="CL-USER"= and ="USER"=. The function [[http://clhs.lisp.se/Body/f_pkg_ni.htm][=package-nicknames=]] can be used to retrieve the list of nicknames (strings) for a package, which doesn't include the name of the package. #+BEGIN_SRC lisp :session pcl :exports both (find-package "COMMON-LISP-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (package-nicknames (find-package "COMMON-LISP-USER")) #+END_SRC If a package is to be unambiguously identified by its name or one of its nicknames, two different packages must have disjoint sets of /name + nicknames/. * Tracking a symbol. The Lisp symbols are very particular when compared to the identifiers of other programming languages. They persist throughout the runtime of an environment and are an important component of the power of Lisp. A package is referenced via its name; the same goes for a symbol, but the process is much more complex. The first and most common method to create or "to /allocate/ a symbol" consists in just writing it down and passing it to the Lisp reader using the standard Lisp conventions. In a Lisp environment, there is a set of "pre-defined" symbols ready to be used. To caution the reader about the complexity of the subject, we begin with a seemingly strange example: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (find-symbol "X" "COMMON-LISP-USER")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (ignore-errors x)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (find-symbol "X" "COMMON-LISP-USER")) #+END_SRC The function [[http://clhs.lisp.se/Body/f_find_s.htm][=find-symbol=]] is analogous to the function =find-package=: given some /name/, a character string, it looks for a symbol with this name. Remember the correspondence between packages and names? A similar correspondence exists between symbols and names, but it's more involved. In particular, a name is /never sufficient/ to determine a symbol, a /package/ is also necessary. This is why the =find-symbol= function requires /two/ arguments, the name of the symbol and the package with respect to which the search is to be done. At the beginning of our story, no symbol with the name ="X"= existed in the default package ="COMMON-LISP-USER"=, and the return value of =find-symbol= is negative, =nil=. (We will explain the second =nil= value in a moment.) Then we pass the Lisp reader the one character expression '=x='. Lisp evaluates it according to its /read-eval-print/ cycle. First, Lisp /reads/ the expression and notices that the symbol =x= is used. It examines the current package ="COMMON-LISP-USER"=, and does /not/ find such a symbol. Before doing anything else, Lisp /allocates/ (creates) this symbol, the right terminology being: "Lisp /interns/ a symbol =x= of name ="X"= in the package ="COMMON-LISP-USER"=." To refer to this symbol, the pair made of /its/ name ="X"= and the (not /its/!!) package ="COMMON-LISP-USER"= are necessary, but it would be cumbersome to have to write the expression =(find-symbol "X" "COMMON-LISP-USER")= every time this symbol is used. To simplify the process of referring to a symbol the designers of Lisp have organized the workspace as follows: if a symbol is read, its character string is capitalized, giving its /name/, and the default package is assumed. In other words, the text ='x'= is roughly equivalent to the text =(find-symbol "X" "COMMON-LISP-USER")=. How convenient! ('Roughly', because '=x=' may trigger the allocation of a symbol. In contrast, =find-symbol= never allocates a symbol; but see the function =intern= later.) In the second expression of our example, once Lisp has "understood" that the user intends to use the symbol =x=, and observing that such a symbol is not present in the package ="COMMON-LISP-USER"=, it allocates a symbol of name ="X"= and /interns/ it in the package ="COMMON-LISP-USER"=. More specifically, a pointer referring to the just allocated symbol is added to the list $InS(p)$ of the internal symbols of the package $p$. Note also that it is common in Lisp to input the symbols in lowercase letters, and Lisp prints them in uppercase. This might be convenient for distinguishing input from output in an interactive session, but requires some clarification. In particular, there is no difference between the symbols =nil= and =NIL=, whereas the strings ="nil"= and ="NIL"= are different. #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (find-symbol "nil" "COMMON-LISP-USER")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (find-symbol "NIL" "COMMON-LISP-USER")) #+END_SRC The fruitless search for a /symbol/ with the lower-case name ="nil"= (a /string/) returns a double negative answer, but it is the second =NIL= that is meaningful! Compare that with the second expression which has the same first return value, but a different second one. The symbol =NIL= also plays in Lisp the role of the boolean /false/, which creates an ambiguity: if the answer is the symbol =NIL= is that a negative answer or is it a positive answer made of the symbol =NIL=? The ambiguity is resolved by the /second/ return value. In the first case, the second =NIL= /confirms/ that the first =NIL= is to be interpreted as the boolean /false/. In the second expression, the second return value =:INHERITED=, not a negative, indicates that the symbol =NIL= has been found, and that its /accessibility/ via the package ="COMMON-LISP-USER"= is /inherited/, a point to be explained later. In other words, the function =find-symbol= returns a genuine symbol /and/ its accessibility via the package argument, or a pair of =nil='s if no symbol is found. The logician can deduce that an accessibility cannot be the symbol =nil=! Let us return to the one character expression '=x='. The read part of the read-eval-print loop cycle reads the symbol =x= and interns it in the package ="COMMON-LISP-USER"=. The expression is /read/ and must now be /evaluated/. The evaluation of a symbol consists in looking for a /value/ of this symbol. To this end a value pointer in the symbol's internal representation is examined. In this case it is found that, at this time, there is no value for the freshly allocated symbol =x=. Therefore the /eval/ step of the /read-eval-print/ cycle generates an error, terminating the cycle with an informative error message. In any case, a symbol of name ="X"= is now present in the package ="COMMON-LISP-USER"=, as confirmed by the value of the last expression. The symbol's accessibility via the ="COMMON-LISP-USER"= package is /internal/, which is to be explained later. * Examining a symbol. A symbol is a Lisp object with a rich internal structure and there are several functions for exploring symbols and their properties. Let us examine in detail the symbol =x= allocated in the previous section. #+BEGIN_SRC lisp :session pcl :exports both (symbol-name 'x) #+END_SRC The [[http://clhs.lisp.se/Body/f_symb_2.htm][=symbol-name=]] function returns the /name/ of a symbol; it is analogous to the =package-name= function for a package. The Lisp function =symbol-name= is crucial for our subject, and we denote it as a mathematical function $sn: Sm \longrightarrow St$, a function which in general is not injective: several symbols may have the same name, this is our main subject. Note that in the previous expression the symbol =x= has been quoted ='x= to prevent its evaluation. More precisely, ='x= is an abbreviation for the expression =(quote x)=, where [[http://clhs.lisp.se/Body/s_quote.htm][=quote=]] is a /special/ function which /does not/ evaluate its argument, and it returns this argument as is. Hence, the previous expression is equivalent to: #+BEGIN_SRC lisp :session pcl :exports both (symbol-name (quote x)) #+END_SRC Here, the non-quoted =x= does not generate an error. If instead you do not quote the symbol =x= (implicitly via ='=, or explicitly via =quote=) it will be evaluated before being used and generate an error: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (ignore-errors (symbol-name x))) #+END_SRC The package which /owns/ a symbol, its /home package/, can be determined with the function [[http://clhs.lisp.se/Body/f_symb_3.htm][=symbol-package=]]. #+BEGIN_SRC lisp :session pcl :exports both (symbol-package 'x) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (find-symbol "X" "COMMON-LISP-USER")) #+END_SRC As we have just seen, a symbol produces a name and a package (with the help of =symbol-name= and =symbol-package=). Conversely, given a name and a package, =find-symbol= produces the symbol with this name /via/ this package. The reader might think that there is a correspondence between symbols and pairs of strings and packages: $$Sm \longleftrightarrow St\times P$$ However, the situation is more involved: for the beginner, this might be a convenient point of view, but in fact such a point of view is wrong. If you intend to fully understand the complex relationship between strings, packages and symbols, please forget this tempting but terribly misleading correspondence. The goal of the next sections is to establish the correct point of view. For completeness we finish this section with a brief glance at the other data possibly stored in a symbol. A symbol can have a value. You may remember that in our Lisp session the symbol =x= does not (yet) have a value. This can be established by the predicate [[http://clhs.lisp.se/Body/f_boundp.htm][=boundp=]]: #+BEGIN_SRC lisp :session pcl :exports both (boundp 'x) #+END_SRC The main method to give a symbol a value is the [[http://clhs.lisp.se/Body/m_setf_.htm][=setf=]] function (a macro). In most Common Lisp implementations, the =setf= macro may be used directly, in which case the default status of the argument symbol is /variable/. In strict ANSI implementations such as SBCL here, a =defvar= statement is required to explicitly define this status. Other possible statuses are /constant/ (via =defconstant=) and /parameter/ (via =defparameter=). #+BEGIN_SRC lisp :session pcl :exports both (defvar x) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (setf x '(4 5 6)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both (boundp 'x) #+END_SRC Note that the first argument of =setf=, the symbol =x=, is not evaluated. Otherwise an error would be generated, since the symbol =x= does not yet have a value. After the =(setf ...)= expression has been evaluated, the symbol =x= does have a value, in this case, the list =(4 5 6)=. This value would now be the result of the evaluation of the expression =x=. It can also be obtained with the function [[http://clhs.lisp.se/Body/f_symb_5.htm][=symbol-value=]]. (In this tutorial, we do not study the possible difference between, on the one hand, the simple /evaluation/ of the symbol =x= asked for by the expression ='x'= and the /evaluation/ of =(symbol-value 'x)= on the other. In our elementary examples there is no difference between the two. Possible differences might come from different variable scopes, such as /lexical/ or /dynamic/, but this subject is not studied here.) #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim x #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (symbol-value 'x) #+END_SRC Don't forget to quote the symbol! See the following error: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (ignore-errors (symbol-value x))) #+END_SRC A symbol can also have a /functional value/. This does not mean that the value we just discussed could be a function (which it could be!). It means that /besides/ the previous value, a symbol can also have an /additional/ value, a function, which can be used independently of the "ordinary" value of this symbol. For example the function $n \mapsto 10 - n$ could be recorded as the /functional/ value of the same symbol, and this function could then be used as shown below. #+BEGIN_SRC lisp :session pcl :exports both (defun x (n) (- 10 n)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both (x 3) #+END_SRC Both values coexist in the symbol without any interference: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (symbol-value 'x) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (symbol-function 'x) #+END_SRC and can be used jointly and independently: #+BEGIN_SRC lisp :session pcl :exports both (x (first x)) #+END_SRC You might already suspect that the position of =x= just after a left parenthesis =(x ...)= makes Lisp extract the /functional/ value and not the "ordinary" value. This is correct, but not the subject of this tutorial. Finally, we mention only in passing the following: 1. A symbol can have a functional value while not having an ordinary value. 2. An ordinary value can be a functional object, usually then called via =funcall= or =apply=. 3. A symbol also contains a property list or =plist=, with many possible uses, but not at all studied here. The internal structure of the [[http://clhs.lisp.se/Body/t_symbol.htm#symbol][symbol]] =x= might be visualized thus: #+BEGIN_EXAMPLE +-----------------+ | Name o-|-----> "x" +-----------------+ | Value o-|-----> (4 5 6) +-----------------+ | Function o-|-----> (lambda (n) (- 10 n)) +-----------------+ | Package o-|-----> # + ----------------+ | Property list o-|-----> NIL +-----------------+ #+END_EXAMPLE * But what the hell is a package? Now we attack the heart of our subject.\\ **Fact 1:** A package $p\in P$ maintains four fundamental lists: - A list $InS(p)$ of its internal symbols. - A list $ExS(p)$ of its external symbols. - A list $ShgS(p)$ of its shadowing symbols. - A list $U(p)$ of its used packages.\\ The first important fact toward understanding the subject is this: the first three lists are lists of /symbols/ or, more precisely, lists of pointers (machine addresses) aimed at the symbols. Every symbol has a /unique/ "existence": a symbol is a unique machine object in the environment, but /several/ packages may /see/ the same symbol, that is, include the machine address of this symbol. In particular these lists are not lists of symbol names (strings). There is another mechanism that allows Lisp to reach a symbol given its name. Every symbol can be accessed directly via a suitable package, and several packages can in general play this role for the same symbol at a given moment in a Lisp session. For example, all the [[http://clhs.lisp.se/Front/X_Symbol.htm][standard Lisp symbols]] in the ="COMMON-LISP"= package typically are accessible via any package. No duplicate symbol (more precisely, no duplicate pointer) is to be found in each of these lists. Also the relations $InS(p)\cap ExS(p)=\emptyset$ and $ShgS(p)\subset InS(p)\cup ExS(p)$ must be satisfied, for reasons to be studied later.\\ **Fact 2:** Given the current set of packages, assumed /coherent/, and given the current state of all the fundamental lists of these packages, the function =find-symbol= can unambiguously determine /whether/ a symbol of a given name (some string) is /accessible/ via some package; if so, this symbol is unique.\\ The importance of the package argument of =find-symbol= could make the reader believe the function =symbol-package= will be useful for various tests. This is not the case: All this function returns is the "birth place" of the symbol, technically called its /home package/, which says almost nothing about the current status of this symbol with respect to any package. All it says is that at least it is /interned/ in this package. In particular, a symbol can be accessible via several packages, possibly different from its home package. Here is a simple illustration: #+BEGIN_SRC lisp :session pcl :exports both (symbol-package 'nil) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (find-symbol "NIL" "COMMON-LISP")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (multiple-value-list (find-symbol "NIL" "COMMON-LISP-USER")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (eq (find-symbol "NIL" "COMMON-LISP") (find-symbol "NIL" "COMMON-LISP-USER")) #+END_SRC The /home package/ of =nil= is ="COMMON-LISP"=, and this symbol of name ="NIL"= is accessible via ="COMMON-LISP"=, but also via ="COMMON-LISP-USER"=, though not with the same status. It is /external/ in the first case and /inherited/ in the second case, points to be examined later. The value (=T=) of the last expression /proves/ that both symbols, determined by =find-symbol= via different packages, in fact are the same. Fact 2 can be made a little more precise: At any given time during a Lisp session, a set $P$ of packages and a set $Sm$ of symbols are defined. The various lists of symbols $InS(p)$, $ExS(p)$ and $ShgS(p)$, and the list $U(p)$ of packages used by a package $p\in P$ together define without any ambiguity which symbols are accessible via the package $p$. In other words, there is a well defined function $\rho$: $$ \rho: P \longrightarrow [St \longrightarrow Sm \cup \{\mathtt{nil}\}] $$ We chose the letter $\rho$ for reader, because this function is mainly used by the reader. Our "mathematical" function $\rho$ is nothing but an avatar of the Lisp function =find-symbol=. Let $p\in P$ be a package and $st\in St$ be a string. Then: $$ \rho(p)(st) = (\mathtt{find-symbol}\; st\; p) $$ The left-hand side uses the standard mathematical functional notation, and the right-hand one uses the Lisp notation. If no symbol is found for a pair $(st, p)$, the symbol (!) =nil= is returned and the second value of =find-symbol=, the symbol =nil= again, tells the user that the symbol search did not return anything. A package $p$ is nothing but a way of defining the function $\rho(p):St\longrightarrow Sm\cup\{\mathtt{nil}\}$. In other words, the four fundamental lists of a package determine what character strings appear as the names of the symbols accessible via this package. This approach is described in detail now. * Internal symbols. The first fundamental list of a package is the list of its /internal symbols/. It is a pity that no standard Lisp function can produce this list, but here is a simple solution to the problem: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (defun package-internal-symbols (package) (let ((rslt)) (do-symbols (s package) (when (eq (second (multiple-value-list (find-symbol (symbol-name s) package))) :internal) (push s rslt))) rslt)) #+END_SRC We can list now all the internal symbols of a package: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (package-internal-symbols "COMMON-LISP-USER") #+END_SRC In fact, only the first ten internal symbols are displayed. We see in particular the symbols =package-internal-symbols=, =s= and =rslt= used in the definition of our function. Some are rather esoteric, strictly speaking illegal: at the beginning of a Lisp session, the ="COMMON-LISP-USER"= package should in principle be void of symbols. To make our experiments more understandable it is better to allocate and use toy packages. The [[http://clhs.lisp.se/Body/m_defpkg.htm][=defpackage=]] function (a macro) allocates a package of a given name. #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (defpackage "P1" (:use "CL")) #+END_SRC A package ="P1"= is now present in our environment, without any internal symbols. #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (package-internal-symbols "P1") #+END_SRC (The symbol =nil= also represents the empty list.) Let's allocate a few symbols and make them internal in the package ="P1"=! The current default package is ="COMMON-LISP-USER"= and we switch it to ="P1"= like this: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (in-package "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" *package* #+END_SRC From now on, the default package is ="P1"=. If we write down a symbol then it will be understood by the Lisp reader as accessed via the package ="P1"=, that is, unless we use a special syntax. Let us create a list of three symbols and assign it to the symbol =my-list=, and then examine the internal symbols of the package ="P1"=: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (setf (symbol-value 'my-list) '(symb1 symb2 symb3)) #+END_SRC The reader probably would prefer =(defvar list)= followed by =(setf list ...)=, which is rejected by some Lisp implementations, in particular SBCL, because in principle the symbol =list=, owned by the package ="COMMON-LISP"=, is "locked". We use the symbol =my-list= instead. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (ignore-errors (package-internal-symbols "P1"))) #+END_SRC Oops! What happened? We've successfully used the function =package-internal-symbols= earlier and suddenly it seems to have disappeared from the environment. The point is that it was /defined when/ ="COMMON-LISP-USER"= was the default package. Let us examine the situation! #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "COMMON-LISP-USER")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "P1")) #+END_SRC A symbol with the given name is found in both packages. Before encountering the undefined functional value error, Lisp ( more precisely, the Lisp reader) had interned a /new/ symbol with the name ="PACKAGE-INTERNAL-SYMBOLS"= in the current default package, ="P1"=, for the sole reason that it was present in the source text. Let us verify that both symbols are different! #+BEGIN_SRC lisp :session pcl :exports both :package "P1" (eq (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "COMMON-LISP-USER") (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "P1")) #+END_SRC The symbols are different. Compare that with the status of =nil=, where the same symbol =nil= was accessible via ="COMMON-LISP"= and via ="COMMON-LISP-USER"=. In our present situation both symbols are /different/, and so are their respective properties. For example, the first symbol does have a functional value and the second does not, which can be tested with the function [[http://clhs.lisp.se/Body/f_fbound.htm][=fboundp=]]x: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" (fboundp (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "COMMON-LISP-USER")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" (fboundp (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "P1")) #+END_SRC This is the first time we have seen the double-colon =‘::’= notation: When the symbol =package-internal-symbols= was searched via ="COMMON-LISP-USER"=, the answer was positive but the returned symbol was a little lengthy: =COMMON-LISP-USER::PACKAGE-INTERNAL-SYMBOLS=. The current default package is ="P1"=, via which /this/ symbol is not accessible. However, we may access it using the double-colon notation: #+BEGIN_CENTER =package-name::symbol-name= #+END_CENTER This is one of the methods which allows the user or Lisp to refer to a symbol not accessible via the default package. The Lisp reader is told: "We mean the symbol =symbol-name= via the package =package-name=." In contrast, for the symbol with the same name but interned in the different ="P1"= package, because ="P1"= is the current default package, the double colon notation is not necessary. The longer notation lets us access a symbol interned in the ="COMMON-LISP-USER"= package and its functional value as follows: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (cl-user::package-internal-symbols "P1") #+END_SRC The =cl-user::= prefix tells Lisp to look for the symbol via the ="CL-USER"= package (Remember that ="CL-USER"= is a nickname for ="COMMON-LISP-USER"=?) and, because of its position just after a left parenthesis, to use its functional value. Currently, there are four internal symbols in the package ="P1"= and they are returned in an arbitrary order. By the way, what about the symbol =list=, to which we assigned a list (!) of symbols? #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (find-symbol "LIST" "P1")) #+END_SRC The symbol is accessible via the package ="P1"= but with a different status, the status /inherited/. We will examine this situation later. In fact, the symbol =list=, one of the most important symbols in Lisp, is already present in the package ="COMMON-LISP"= at the beginning of a session, and by default accessible from any package. We must mention also that our function $\rho$, or the function =find-symbol= if you prefer, is (almost) injective /with respect to/ $st$. Let $p_1$ and $p_2$ be two packages, $st_1$ and $st_2$ two strings. Then if $\rho(p_1, st_1) = \rho(p_2, st_2)\neq\mathtt{nil}$, then $st_1 = st_2$. Pay attention to the fact that the equality is an equality between /symbols/. This is a consequence of $\mathtt{symbol-name}(\rho(p, st)) = st$; in other words, there is no possibility of nicknames for a symbol name. In particular, there is a 1-1 correspondence between the internal symbols of a package and their respective names. * Importing a symbol. Instead of using the lengthy notation =cl-user::xxx= to make an internal symbol in ="CL-USER"= accessible from ="P1"=, we may opt to /import/ this symbol in the package ="P1"=. However, in doing so there is another problem due to a name conflict, a point which will be examined in detail later: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (ignore-errors (import 'cl-user::package-internal-symbols "P1"))) #+END_SRC The problem is a "collision" between two /different/ symbols, which cannot coexist as internal symbols in the /same/ package ="P1"= with the /same/ name. Importing the symbol internal in ="CL-USER"= into the package ="P1"= is not compatible with the current presence of /another/ symbol in ="P1"= with the same =symbol-name=. We must first [[http://clhs.lisp.se/Body/f_uninte.htm][=unintern=]] the accidental, pre-existing symbol in ="P1"= to "make room" for the symbol from ="CL-USER"=. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (unintern 'package-internal-symbols "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (import 'cl-user::package-internal-symbols "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (package-internal-symbols "P1") #+END_SRC The Lisp terminology is rather misleading: in fact, it is not at all the symbol which is imported, but the connection /string/ $\mapsto$ /symbol/. More precisely, let $p_1$ and $p_2$ be two packages, $st$ a string and $sm$ a symbol. If $\rho(p_1)(st) = sm$ holds before the importation, then, after the importation, both relations $\rho(p_1)(st) = sm$ /and/ $\rho(p_2)(st) = sm$ are satisfied. In particular, the first relation remains valid. It is not really an importation; it is a /copy/ in the package $p_2$ of the connection $st \mapsto sm$, available in the package $p_1$. Furthermore, nothing is modified about the symbol itself, except /possibly/, but rarely, its home package, as will be examined later. It is tempting for a beginner to think the home-package of the symbol is being modified. Not at all! Another erroneous interpretation would be to think a /copy/ of the symbol to be imported is made "inside" the target package, with all the corresponding ingredients of the symbol, value, functional value, and so on. Wrong again! A copy is made, but only of a /pointer/ toward our symbol found in $InS(p_1)$ and copied in $InS(p_2)$. The [[http://clhs.lisp.se/Body/f_import.htm][=import=]] function has two arguments, the /symbol/ to be imported and the /target package/. The symbol must be well defined, that is, taking account of the current state of the environment. Here, before the import, the symbol is accessible only via ="CL-USER"=, and it is mandatory to use the notation =cl-user::...= (and the symbol must also be quoted to preclude its evaluation). After the import, as can be seen in the example, the symbol can be used functionally from the package ="P1"= without the prefix notation. Although the imported symbol is in the list of the internal symbols of the package ="P1"=, it is not quite the same as before. The next mandatory exercise consists in comparing the original symbol and the imported one. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (eq 'cl-user::package-internal-symbols 'package-internal-symbols) #+END_SRC The function =eq= proves that both notations name the same symbol. Here's another proof: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (eq (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "CL-USER") (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "P1")) #+END_SRC If you remember the =symbol-package= function, the next test is revealing: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (symbol-package (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "CL-USER")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (symbol-package (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "P1")) #+END_SRC The /unique/ symbol accessed by two different paths has of course a unique home package, namely its /birth place/, the ="CL-USER"= package. Please, do not pay too much attention to this symbol package; it is just an artifact of the session's history and does not have any real interest. You must focus on the chain $\mathit{package}\rightarrow[\mathit{string}\rightarrow \mathit{symbol}]$ and not much else. There is no real reason to be concerned about the map =symbol-package=: $\mathit{symbol}\rightarrow\mathit{package}$. * Name conflicts. If poorly understood, the introduction of Lisp packages to a Lisp project will sooner or later lead to /name conflicts/. There is nothing mysterious about them and there are standard "home remedies" for dealing with them.\\ *Warning.* A name conflict can only be generated if two /different/ symbols $sm$, $sm' \in Sm$ end up with the same name: $sn(sm) = sn(sm')$. A name conflict occurs if these symbols $sm$ and $sm'$ become accessible via the same package $p$, which is a forbidden situation, since it would introduce ambiguity to the =find-symbol= function. On the other hand, the fact that some symbol (singular) is accessible via /different/ packages never generates a name conflict by itself. Lisp constantly is on the lookout for possible name conflicts and stops with an error if the evaluation of an expression causes a name conflict. Some errors are /continuable/, which means that Lisp is able to propose a conflict resolution via =unintern= or =shadowing-import=. Remember the use of =unintern= in the previous section? What happens if an imported symbol has the same /name/ as a symbol already accessible in the target package? In mathematical language, let $p_1$ and $p_2$ be two different packages, $st$ some string, and $sm_1$ and $sm_2$ two different symbols, internal in the respective packages $p_1$ and $p_2$, with same name $st$. In other words, assume $\rho(p_1)(st) = sm_1$ and $\rho(p_2)(st) = sm_2$, $sm_1 \in InS(p_1)$ and $sm_2 \in InS(p_2)$, and $sm_1\neq sm_2$. A situation of this sort could be the following (Remember the current default package is ="P1"=!): #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (defvar cl-user::symb4) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (setf cl-user::symb4 111) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (defvar symb4) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (setf symb4 111) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (eq 'cl-user::symb4 'symb4) #+END_SRC The evaluation of the first expression interns a symbol of name ="SYMB4"= in the package ="CL-USER"= and gives it the value 111, while the second expression interns a /different/ symbol, again of name ="SYMB4"=, in the package ="P1"=, the current default package, and also gives it the value 111. As a result two symbols have been allocated, installed somewhere in the environment, with the same value of 111. Also the connections $\rho(\mathtt{"CL-USER"}): \mathtt{"SYMB4"}\mapsto \mathtt{cl-user::symb4}$ and $\rho(\mathtt{"P1"}): \mathtt{"SYMB4"}\mapsto\mathtt{p1::symb4}$ have been installed, that is, the addresses of both symbols have been pushed to the /respective/ lists of /internal/ symbols of the packages ="CL-USER"= and ="P1"=. The =eq= comparison confirms that the symbols just allocated are different. Two symbols with the same name ="SYMB4"= are now present in our environment. This is possible because these symbols are interned in different packages. Each one is accessible via the appropriate package. Both symbols are live and can be freely used: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (setf cl-user::symb4 (+ symb4 cl-user::symb4)) #+END_SRC What happens if we import =cl-user::symb4= into the package ="P1"=? Let's try! #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (ignore-errors (import 'cl-user::symb4 "P1"))) #+END_SRC The importation is rejected, for it would not be possible anymore to coherently define $\rho(\mathtt{"P1"})(\mathtt{"SYMB4"})$. We will see later that another process called /shadowing/ allows the user to suppress the symbol =symb4= already present in ="P1"= and to authorize the importation of the symbol present in ="CL-USER"=. * The function =intern=. **Fact 3:** Let $p$ be a package and $(sm_1,...,sm_n)$ the list of its internal symbols. If $p$ is the default package, any of these symbols can be directly used without the double colon notation ='::'=. If $p$ is not the default package, these symbols can be used thanks to the notation $p::sm_i$. All these internal symbols have different names. Nothing prevents several packages from having the same symbols in their lists of internal symbols. This is possible thanks to the =import= function which can work only if no name conflict is generated.\\ To be complete on this matter, we now to consider the Lisp function [[http://clhs.lisp.se/Body/f_intern.htm][=intern=]]. It is used almost exactly as the function =find-symbol= with just a difference: if no symbol is found, then a symbol is allocated and interned in the package referred to in its second argument. Compare the following: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (find-symbol "SYMB5" "P1")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (intern "SYMB5" "P1")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (intern "SYMB5" "P1")) #+END_SRC We start without any =symb5= symbol in ="P1"=, which is confirmed by the double negative answer of =find-symbol=. If instead we use the function =intern=, Lisp notices that such a symbol is absent from ="P1"= and therefore allocates a symbol of name ="SYMB5"= and /interns/ it in the package ="P1"= as an internal symbol. The second value =nil= is an indicator of the allocation just made. If we repeat the same expression, which is not an error, then Lisp signals that such a symbol is already present and returns its status, which also informs us of its previous presence. A little weird! This definition of the function =intern= guarantees the coherence of the set of internal symbols of packages. A symbol can be allocated only through this function =intern=. It requires two arguments, a string $st$ and a package $p$. In a sense, the function =intern= first calls =(find-symbol st p)= with the same arguments. If a symbol is found it is returned without modification and no allocation of a new symbol takes place, which otherwise would create a name conflict. On the other hand, if a corresponding symbol is not found, it is allocated and interned in the package explicitly or implicitly quoted, and this new symbol certainly does not generate a name conflict. In this way, the uniqueness of the names of internal symbols of a package is enforced. This does not prevent two different symbols of the environment from having the same name, but the packaging system is organized such that they cannot be accessed via the same package. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (package-internal-symbols "P1") #+END_SRC In practice, the Lisp reader constantly uses the =intern= function, for example when reading an expression. If it detects the presence of a symbol, the colon notation allows it to decide if a package is explicitly given, otherwise it must use the default package. The name of the symbol, alone or package-qualified using the colon notation, then unambiguously determines the symbol. If present, it is used, otherwise it is allocated and then used. This is exactly the role of the =intern= function. * Moving a symbol between two packages. Up until now, we have seen no means of modifying the home package of a symbol. In fact, it is possible but may seem a little contorted. It is almost without any practical interest except as an opportunity to use this as a vehicle to discover other facts. First, it is possible to /unintern/ a symbol. For example, we can unintern the symbol ="SYMB1"= from the package ="P1"=. What happens is precisely that this symbol is removed from the list of the internal symbols of ="P1"=. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (unintern 'symb1 "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (package-internal-symbols "P1") #+END_SRC What is modified is the /list/ of internal symbols in the package ="P1"=. The symbol =symb1= itself has not been modified at all, except possibly its home package. In particular the symbol is not deallocated or destroyed, and therefore always present somewhere in the environment. But it's become inaccessible as such, for a symbol most often is accessible only via a package, the default one or some other explicitly given. The symbol =symb1= was interned only in the package ="P1"=, so that it appears now to be unreachable! If a symbol is truly unreachable, the Lisp garbage collector, always silently working in the background, will establish that fact and free the corresponding part of memory for reuse. Except that our symbol remains reachable, for it was the first element of the list assigned to the symbol =my-list=. Let's have a look at its value! #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim my-list #+END_SRC Our symbol is still present in the list but in a little different form. To be sure, it is still a symbol: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (type-of (first my-list)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (symbol-package (first my-list)) #+END_SRC No home package, i.e., our symbol is "homeless"! The prefix ='#:'= indicates that this symbol is homeless. The =unintern= function had this effect because the package argument of =unintern= was the package which /owned/ the symbol. Otherwise the home package would not be modified. Although our symbol has become homeless, it continues to be an ordinary symbol, for example, we can assign it a value: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (set (first my-list) 444) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (symbol-value (first my-list)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (eval (first my-list)) #+END_SRC Unlike =setf= (macro), the =set= function evaluates its arguments, so that before the assignment, the expression =(first my-list)= is evaluated, which returns our mysterious symbol =#:symb1=. Once that's done, the =set= function assigns (the value of) the second argument to it, as is confirmed by the other expressions. It is even possible to /import/ a homeless symbol in a package, but then the target package becomes the home package of the imported symbol. The importing package can be different from the original home package. As a result our symbol is "moved" into another package. This explains the new form of the value of =my-list=. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (import (first my-list) "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (symbol-package 'cl-user::symb1) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim my-list #+END_SRC Our symbol could also have been imported in other packages before being uninterned from its home package. The status of our symbol with respect to other packages would not have been modified. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (import 'symb3 "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (eq 'cl-user::symb3 'symb3) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (unintern 'symb3 "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (eq 'cl-user::symb3 (third my-list)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (import 'cl-user::symb3 "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (symbol-package 'cl-user::symb3) #+END_SRC Here we import =symb3= in the package ="CL-USER"= before uninterning it from the package ="P1"=. The symbol is then accessible via the package ="CL-USER"= or via the list =my-list=. The evaluation of the expression =(eq ...)= proves that both paths lead to the same address. When the first =(import ...)= expression is evaluated, the symbol does have a home package and that it is not changed. When it is uninterned from the package ="P1"=, it becomes homeless. Lisp does not take account of the fact it is internal in other packages. In theory, this would allow it to select one of these packages as an "emergency home" package, but the choice would be quite arbitrary? Furthermore this would force Lisp to examine the list of all other packages and all their accessible symbols, which would be very expensive! Here, despite the symbol being internal and therefore interned in the package ="CL-USER"=, it becomes homeless after evaluating the =(unintern ...)= expression. Finally we import it in the package ="CL-USER"=, where, in fact, it is already present. According to the ANSI standard (see the following note), in such a situation, its status remains unchanged. In particular its home package remains =nil=. A little esoteric, huh? Don’t worry: the home package does not have any real role for the Lisp user. ----- *Note:* The [[http://clhs.lisp.se/Body/f_import.htm#import][passage]] in the [[http://clhs.lisp.se/Front/index.htm][Common Lisp HyperSpec]] reads like this: /"If the symbol is already present in the importing package, import has no effect."/ It appears that the behavior of SBCL and some other compilers is different: The output is =#= not =NIL=. ----- We will prove later that if a symbol $sm$ has a home package $p$, then $sm$ is certainly present in $p$, that is, internal or external in this package. The "converse" is false: a symbol may be present in several packages $p_1,\ldots , p_k$ while not having a home package. This might sound a little weird, but you shouldn't lose much sleep over it: except for the internal efficiency of the Lisp interpeter or compiler, the notion of a home-package is almost useless for the programmer. * External symbols and used packages. The analysis of this notion of an internal symbol was a little lengthy, but, in this way, the general structure of the packaging system is now very clear, and it will be a lot easier to understand the other concepts. Two fundamental lists maintained in the packages play a "dual" role in the packaging system, the list of /external/ symbols and the list of /used/ packages. The following function gives the list of the external symbols of a package. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim *package* #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (defun package-external-symbols (package) (let ((rslt nil)) (do-external-symbols (s package) (push s rslt)) rslt)) #+END_SRC The evaluation of the first expression is just to confirm that the default package remains ="P1"=. Let us see what the lists of external symbols of the packages ="P1"=, ="CL-USER"= and ="COMMON-LISP"= are: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (package-external-symbols "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (package-external-symbols "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (subseq (package-external-symbols "COMMON-LISP") 0 10) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (length (package-external-symbols "COMMON-LISP")) #+END_SRC There are no external symbols in ="P1"= and ="CL-USER"=, but 978 such symbols in the package ="COMMON-LISP"=. (Only the first ten are being displayed.) A point of concern might be the following: these symbols are accessible via the package ="COMMON-LISP"= and yet the double-colon notation =common-lisp::simple-error= was not used. The explanation has two "dual" parts: 1. The symbol =simple-error= is external in the package ="COMMON-LISP"=. 2. The package ="COMMON-LISP"= is in the =package-use-list= of the package ="P1"=: #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (multiple-value-list (find-symbol "SIMPLE-ERROR" "COMMON-LISP")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (package-use-list "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (package-nicknames (find-package "COMMON-LISP")) #+END_SRC ="CL"= is the nickname of the package whose official name is ="COMMON-LISP"=, and which is not to be confused with ="COMMON-LISP-USER"=. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (eq (find-package "CL") (find-package "COMMON-LISP")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (eq (print (find-package "CL")) (print (find-package "CL-USER"))) #+END_SRC Applied to a package $p$, the Lisp function [[http://clhs.lisp.se/Body/f_pkg_us.htm][=package-use-list=]] returns the list of all the packages used by $p$. If $p_1$ uses $p_2$, then it is said that $p_2$ is used by $p_1$. Whether this relation is reflexive can be doubted (What does "$p$ uses $p$" mean?), and it is neither symmetric ("$p_1$ uses $p_2$" does not imply "$p_2$ uses $p_1$") nor transitive ("$p_1$ uses $p_2$" and "$p_2$ uses $p_3$” does not imply "$p_1$ uses $p_3$"). The oriented graph of the /use/ relation is totally arbitrary. * First toy examples. Let's start over with two "fresh" packages ="P1"= and ="P2"=. #+BEGIN_SRC lisp :session pcl :exports both :package "P1" :results value verbatim (in-package "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (delete-package (find-package "P1")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (defpackage "P1" (:use "CL")) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (defpackage "P2" (:use "CL")) #+END_SRC We switch the default package to ="P1"=: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (in-package "P1") #+END_SRC and we assume that, /in this environment/, we define the functions =package-internal-symbols= and =package-external-symbols= as explained before. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (defun package-internal-symbols (package) (let ((rslt)) (do-symbols (s package) (when (eq (second (multiple-value-list (find-symbol (symbol-name s) package))) :internal) (push s rslt))) rslt)) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (defun package-external-symbols (package) (let ((rslt)) (do-external-symbols (s package) (push s rslt)) rslt)) #+END_SRC What about the internal symbols of ="P1"=? #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (package-internal-symbols "P1") #+END_SRC We see the symbols naming our functions and also the symbols =rslt= and =s= used to define them. All the other symbols used in their definitions are in fact external in ="COMMON-LISP"= and absent here. We move now to the package ="P2"= from where we cannot access the symbol =package-internal-symbols= directly. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (in-package "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (multiple-value-list (find-symbol "PACKAGE-INTERNAL-SYMBOLS" "P2")) #+END_SRC We have already seen the use of the ='::'= notation to make reference to the symbol at once. We can also /import/ the symbol, in which case it becomes directly accessible from ="P2"= permanently: #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (import 'p1::package-internal-symbols "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (package-internal-symbols "P2") #+END_SRC Another solution requires /two/ steps: 1. Make the symbol =external= in some package, which is the role of the Lisp function [[http://clhs.lisp.se/Body/f_export.htm][=export=]]. 2. Declare that the current package /uses/ the package where the symbol is external. Let us use this method to make the symbol =package-external-symbols= directly visible from ="P2"=. #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (export 'p1::package-external-symbols "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (use-package "P1" "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (package-external-symbols "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (package-external-symbols "P1") #+END_SRC An =export= expression has the same form as an =import= one: =(export= $sm$ $st$ =)=, where $sm$ is a symbol and $st$ is a string naming a package. The symbol $sm$ must be readable from the current default package, for us ="P2"=, which explains the (required) prefix ='p1::'=. The argument $st$ is a string naming the package where the symbol $sm$ is to become external. Again, the terminology is somewhat misleading: the function =export= does not export anything. All it does is to make some connection $st \rightarrow sm$ "exportable" to other packages via =use-package= expressions. The 978 external symbols of ="COMMON-LISP"= contain all the standard Lisp constants, variables and functions. Because an allocated package is always initialized with the package ="COMMON-LISP"= in its =package-use-list=, these basic Lisp objects will be accessible from any package. If we remove ="COMMON-LISP"= from the =use-package-list= of the current default package, via =unuse-package=, the situation becomes a little tricky. ----- *Note:* Before setting the saw to the branch on which we are sitting, we have to take one precaution: We can't let go of =LET=. Otherwise the interaction between Emacs and SLIME will cease to function. #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (cl::import 'cl::let "P2") #+END_SRC ----- #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (unuse-package "COMMON-LISP" "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (cl::multiple-value-list (cl::ignore-errors (+ 2 2))) #+END_SRC The notation for the external symbol =t= in the package ="COMMON-LISP"= has become =COMMON-LISP:T=, for ="COMMON-LISP"= is no longer used by ="P2"=. Since the symbol =T= is external in ="COMMON-LISP"=, the simple colon notation is sufficient. However, the most elementary operations are not easily accessible anymore, for example, the addition function ='+'=. This is not a good position to be in. Let us go back to the previous environment! #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (cl::multiple-value-list (cl::ignore-errors (use-package "COMMON-LISP" "P2"))) #+END_SRC This does not work since the symbol =use-package= can't be reached easily either, and we /must/ use the prefix notation. #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (cl::multiple-value-list (cl::ignore-errors (cl::use-package "COMMON-LISP" "P2"))) #+END_SRC But here is a new problem: earlier, when when we tried to use the symbol ='+'= from ="P2"=, Lisp detected that this symbol is not accessible from ="P2"= and therefore interned a fresh symbol of name ="+"= in ="P2"=. This was legitimate then, but it is now incompatible with ="COMMON-LISP"= having a different symbol with the same name ="+"=, if ="P2"= were to use ="COMMON-LISP"=. The same goes for =use-package=. We must unintern these symbols from ="P2"= to resolve the conflict. Let's not forget to use the colon notation for =unintern=, which is otherwise unreachable! #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (cl::unintern '+ "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (cl::unintern 'use-package "P2") #+END_SRC ----- *Note:* The following is necessary to keep the interaction between Emacs and SLIME going. Otherwise, this would be another conflicting symbol. #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (cl::unintern '*default-pathname-defaults* "P2") #+END_SRC ----- #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (cl::use-package "COMMON-LISP" "P2") #+END_SRC Phew, we are back in a state where Lisp is usable! * Importing an external symbol. Recall that a symbol $sm$ is /accessible/ from a package $p$ if $\rho(p)(st) = sm$ for $st = \mathtt{symbol-name}(sm)$. In that case, the possible return values from =find-symbol= are as follows: 1. If $sm$ is one of the internal symbols of $p$ the second value returned by =find-symbol= is =:internal=. 2. If $sm$ is one of the external symbols of $p$ the second value returned by =find-symbol= is =:external=. 3. If $sm$ is an external symbol of a package $p'$ used by the package $p$ the second value returned by =find-symbol= is =:inherited=. We have seen how to import a symbol from a package $p'$ into a package $p$. The example given used a symbol $sm$ internal in the package $p'$, which then became also an internal symbol of the package $p$ . This also works if $sm$ is external in the package $p'$, regardless of $p$ using or not using $p'$. (... assuming that no name conflict is generated.) Let's continue our journey in ="P1"=! #+BEGIN_SRC lisp :session pcl :package "P2" :exports both :results value verbatim (in-package "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (intern "SYMB1" "P2") "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB1" "P2")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB1" "P1")) #+END_SRC We intern a symbol =p2::symb1= in ="P2"= and make it external immediately. As it stands this symbol is not accessible via ="P1"=. The fact that it is external in ="P2"= does not prevent us from importing it in ="P1"=. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (import 'p2:symb1 "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB1" "P1")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB1" "P2")) #+END_SRC Once the importation is done, the symbol is accessible via ="P1"= with the status /internal/. The /external/ status with respect to ="P2"= is unchanged, which underlines that the status of a symbol does not depend on anything in the symbol itself. It is the /package/ which must be examined to determine the status of a symbol /with respect to this package/. Curiously, the last =find-symbol= returns the symbol =symb1=, without the prefix ='p2:'=. Lisp prints a symbol according to the following rule: #+BEGIN_QUOTE First, Lisp determines whether a symbol is accessible via the current default package, in which case it is printed without using the colon notation. Otherwise, Lisp examines the home package, where the symbol is certainly internal or external, and it is printed with the double or simple colon notation. The only exception to this rule appears to be a /keyword/, that is, a symbol of the package ="KEYWORD"=. (Though the ANSI definition is somewhat incomplete for the ="KEYWORD"= package, we can assume that it is forbidden to change anything about the scope of keywords.) Keywords are printed simply as =':xxx'=, without any prefix. Finally, if the symbol is homeless, it is printed as =#:xxx=. #+END_QUOTE The take-home message here is that an importation does not change anything for the relevant symbol; it is just a matter of updating a list of internal symbols in some package, that is, a list of respective pointers toward these symbols. Installing a new signpost in San Francisco that reads "New York" does not change anything in New York. Once more: #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (eq (find-symbol "SYMB1" "P1") (find-symbol "SYMB1" "P2")) #+END_SRC A novice could believe that adding ="P2"= to the =package-use-list= of ="P1"= could generate a name conflict, because the symbols =p1::symb1= and =p2::symb1= would become both accessible via ="P1"=. This is not the case: both notations represent one and the same symbol, and there is no ambiguity. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (use-package "P2" "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB1" "P1")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB1" "P2")) #+END_SRC Note that because our symbol is internal in ="P1"=, the function =find-symbol= reports its status as =:internal=. This is in a sense preferred to the status =:inherited=, which after all would be possible. A name conflict occurs only if two /different/ symbols with the same name could be accessible via the same package, possibly via another package used by the first one. Two symmetrical illustrations: #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (intern "SYMB2" "P2") "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (intern "SYMB2" "P1") ) #+END_SRC This is not an error: The second =intern= /does not/ allocate a new symbol, a symbol of name ="SYMB2"= being already accessible via ="P1"=, which uses ="P2"=. In particular the second =intern= does not import the pre-existing symbol. We have to come up with something else to really allocate a new symbol. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (ignore-errors (import (make-symbol "SYMB2") "P1"))) #+END_SRC The function [[http://clhs.lisp.se/Body/f_mk_sym.htm][=make-symbol=]] allocates a /homeless/ symbol of a given name, that's its definition. A homeless symbol cannot cause a name conflict, because it is /not/ accessible via any package. Nothing prevents us from trying to =import= this symbol in ="P1"=, but this generates a name conflict between two different symbols: the freshly allocated =#:symb2= and the symbol external in ="P2"= accessible via ="P1"= which uses ="P2"=. Here's another way to cause a name conflict: #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (intern "SYMB3" "P1")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (intern "SYMB3" "P2")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (ignore-errors (export 'p2::symb3 "P2"))) #+END_SRC The package ="P1"= uses the package ="P2"=, but this does not prevent them from having two different internal symbols with the same name ="SYMB3"=, for ="P1"= does not see the symbol internal in ="P2"=. If we try to make this symbol external in ="P2"=, a name conflict arises. Another sort of name conflict can occur between two used packages: #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (intern "SYMB4" "P2") "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (intern "SYMB4" (defpackage :p3)) "P3") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (ignore-errors (use-package "P3" "P1"))) #+END_SRC Compare that with: #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (intern "SYMB5" "P2") "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (import 'p2:symb5 "P3")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (unexport 'p3:symb4 "P3") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (use-package "P3" "P1") #+END_SRC Two symbols of name ="SYMB5"= are external in ="P2"= and ="P3"=, but the second one is in fact the same as the first one: it has been imported and immediately made external in ="P3"=. This time the =use-package=, which makes the external symbols of ="P2"= /and/ ="P3"= visible from ="P1"=, does not generate a name conflict, since we are dealing with two different paths which lead to the same /symbol/. This works as long as we do not forget to first =unexport= the symbol ="SYMB4"= of ="P3"=, i.e., to downgrade its status from /external/ to /internal/. * A symbol can shadow another symbol. In the matter of name conflicts, the Lisp creators could have adopted another strategy and decided that the symbols /present/ in a package, that is, /internal/ or /external/ in this package, have precedence over /inherited/, possibly different, symbols. However, they reasoned that this could easily generate true programming errors because of possible perverse collisions of identifiers, and turn into a source of obscure bugs. Nevertheless, in line with the general spirit of Lisp, if a programmer really wants to use such a strategy, she can do it on a case-by-case basis and explicitly describe the intended exemptions. This is the role of the [[http://clhs.lisp.se/Body/f_pkg_sh.htm][=package-shadowing-symbols=]] list, which is maintained in every package $p$ and denoted by $ShgS(p)$. This list is empty initially. $ShgS(p)$ can contain symbols present in the concerned package, internal or external. These symbols will automatically /shadow/ any other competing symbol, such as symbols exported in other used packages and with the same =symbol-name=. The function [[http://clhs.lisp.se/Body/f_shadow.htm][=shadow=]] takes a symbol name (or a list of symbol names) and a package name, allocates a symbol with this name if it is not /present/ in this package, and records it in the shadowing list of the package. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (shadow "SYMB6" "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (package-shadowing-symbols "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (intern "SYMB6" "P2") "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (eq 'symb6 'p2:symb6) #+END_SRC Once the symbol =p1::symb6= is on the shadowing-list of ="P1"= the existence of a different external symbol with the same name in the used package ="P2"= does not generate a name conflict. The last =eq=-test proves both symbols are really different. Along the same lines, the function [[http://clhs.lisp.se/Body/f_shdw_i.htm][=shadowing-import=]], as clearly indicated in its name, imports a symbol and immediately adds it to the shadowing list, which could resolve a possible name conflict. For example the *different* symbols =p2:symb4= (external in ="P2"=) and =p3::symb4= (internal in ="P3"=) are in our environment; the package ="P1"= uses both packages ="P2"= and ="P3"=, but =p3::symb4= is internal in ="P3"= and therefore not in conflict with =p2::symb4=. Remember also the current default package is ="P1"=. A simple importation of =p3::symb4= in ="P1"= generates a conflict, while a =shadowing-import= adds the imported symbol to the shadowing list, shadowing the symbol =p2:symb4= and resolving the name conflict. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB4" "P2")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB4" "P3")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (ignore-errors (import 'p3::symb4 "P1"))) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (shadowing-import 'p3::symb4 "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (package-shadowing-symbols "P1") #+END_SRC If necessary, =p2:symb4=, shadowed by the internal =symb4= in ="P1"=, remains accessible via the colon notation. The evaluations of the following expressions assign the same symbol value to both symbols as is confirmed by the subsequent '=' form. Evaluating the next =eq= expression proves that both /symbols/ are different. In contrast, because =p1::symb4= was imported from ="P3"=, the last =eq=-test is affirmative. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (defvar symb4) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (defvar p2:symb4) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (setf symb4 (setf p2:symb4 111)) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (= symb4 p2:symb4) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (eq 'symb4 'p2:symb4) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (eq 'symb4 'p3::symb4) #+END_SRC It is even possible to add an /inherited/ symbol to the shadowing list, maybe to declare a precedence with respect to another future competing symbol in another package intended to be used later. This experiment assumes three different packages, where one of them is not yet used. We first "detach" for example the package ="P3"= from ="P1"=, leaving ="P2"= used by ="P1"=. (Remember ="P1"= is the current default package.) #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (unuse-package "P3" "P1") #+END_SRC We then install two new external symbols with the same name in ="P2"= and ="P3"=: #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (intern "SYMB7" "P2") "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (export (intern "SYMB7" "P3") "P3") #+END_SRC As we have seen already with essentially the same example, if ="P1"= now decides to use ="P3"=, the two ="SYMB7"= symbols collide: #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (ignore-errors (use-package "P3" "P1"))) #+END_SRC In some context, it could happen that ="P1"= has no reason to use =p3:symb7= directly, and instead would like to give precedence to =p2:symb7=. This can be achieved with a shadowing import. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB7" "P1") ) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (shadowing-import 'symb7 "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (use-package "P3" "P1") #+END_SRC The =shadowing-import= refers to =p2:symb7= implicitly, which is external in a package used by ="P1"= and therefore inherited by ="P1"=. We see how the use of =shadowing-import= solves the name conflict, which otherwise would be raised by =use-package=. Let us review our symbols as seen now from our three packages: #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB7" "P1")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB7" "P2")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (eq 'symb7 'p2:symb7) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (multiple-value-list (find-symbol "SYMB7" "P3")) #+END_SRC #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (package-use-list "P1") #+END_SRC The statuses of =symb7= have not changed in ="P2"= and ="P3"=, but seen from ="P1"=, the symbol is now /internal/. This is to satisfy the rule that any shadowing symbol in a package must be internal or external in this package, and also a consequence of the implied importation. * A mathematical definition of the packaging system. We recall that there are three general sets we must study: - The set $St$ of /strings/ - The set $P$ of /packages/ - The set $Sm$ of /symbols/ There is one important difference: the set $St$ is the /infinite/ set of all the possible strings, while $P$ and $Sm$ are /finite/ sets made of the packages and symbols previously /allocated/ in the current Lisp session. We use the notations $pn$ to denote the function =package-name= $pn: P \rightarrow St$ and $sn$ to denote the function =symbol-name= $sn: Sm\rightarrow St$. The map $pn$ is /injective/ for the packages; the function $pn: P \rightarrow pn(P)$ defines a 1-1 correspondence between the packages and their respective names. Here, we ignore the potential nicknames of a package, which is not essential for our discussion. Every package $P$ contains four /fundamental/ lists: - The list $InS(P)$ of its internal symbols - The list $ExS(P)$ of its external symbols - The list $ShgS(P)$ of its shadowing symbols - The list $U(P)$ of its used packages More precisely, these lists are lists of /pointers/ to the mentioned objects, or, if you prefer, of the machine addresses of these objects. It is important to understand these lists are made of (pointers to) /symbols/, not of their names. Each of these lists is /without any duplicates/. The fundamental lists defined in a package are the source of a relatively extensive terminology, which is detailed now. The set of symbols /present/ in a package $p$, denoted by $PrS(p)$, is defined as the union $PrS(p) := InS(p) \cup ExS(p)$: a symbol is present in a package if it is internal or external in this package. This union is /disjoint/, the intersection $InS(p) \cap ExS(p)$ must be empty. Also the relation $ShgS(P) \subset Pr(P)$ is required. A shadowing symbol of the package $p$ must be internal or external in this package, i.e., a shadowing symbol is present. Let $q$ be a package used by the package $p$. The symbols of $q$ /shadowed/ by a symbol of $p$ are: $$ ShdS(q, p) = \{sm \in ExS(q)\;\mathbf{\underline{st}}\;\exists sm' \in ShgS(p)\;\mathbf{\underline{st}}\; sn(sm) = sn(sm')\} $$ In other words, a /precedence/ relation is defined by the list $ShgS(p)$: if ever a symbol $sm' \in ShgS(p)$ and a symbol $sm\in ExS(q)$ have the same name, then, with respect to the package $p$, $sm'$ takes precedence over $sm$, which means $sm$ is /shadowed/. If $sm'$ is present in $p$ but not shadowing, with $sm'\neq sm$ and $sn(sm') = sn(sm)$, then a name conflict would be raised, which is forbidden. The inherited symbols of a package $p$, denoted by $InhS(p)$, are defined as: $$ InhS(p) := \Big\{ \bigcup_{q\in U(P)}(ExS(q) − ShdS(q,p)) \Big\} − PrS(P) $$ A symbol is inherited in a package $p$ if it is external in a /used/ package $q$, not shadowed by a shadowing symbol of $p$, and not present in $p$. The union in the preceding formula is not necessarily disjoint: a symbol can be external in several packages simultaneously, and without causing a name conflict. A symbol could be simultaneously present in $p$ and external in a package $q$ used by $p$. That is why the the last term, $−PrS(P)$, is necessary to define unambiguously what an inherited symbol is. Finally the accessible symbols $AccS(p) := PrS(p) \cup InhS(p)$ are the symbols which are present or inherited. The very definition of $InhS(p)$ implies that this union is disjoint. In this way, the /status/ of a symbol with respect to a package $p$ via which it is accessible is unambiguously defined as a keyword =:internal=, =:external= or =:inherited=. For a package $q\neq p$ there is no relation whatsoever between the status of some symbol $sm$ with respect to $p$ and its status with respect to $q$. *Exercise:* Show that the sixteen pairs of possible status combinations can be realized! It is important to be clear about possible intersections (overlaps) between the lists defined above. Let $p$ be some package and $q$ another package used by $p.$ If $sm\in ExS(q)$ is an external symbol of $q$, the symbol $sm$ /may/ also be present in $p$, in which case its status is internal or external, but not inherited. In particular, this does not generate a name conflict, because it is the same symbol which is visible in two different packages via the same name, and a symbol has only /one/ name! Nevertheless its status /with respect to/ $p$ is not inherited. The status of a symbol is so defined deterministically, even if it is present in two interacting packages by one using the other. Of course the same symbol has status external in $q$. Following the same logic, if $q,q'\in U(p)$, the intersection $ExS(q)\cap ExS(q')$ is not necessarily empty. The Lisp system maintains the /coherence/ of these lists, which is defined as follows: *Definition.* A set of packages is /coherent/ if, for every package $p$, the function =symbol-name= defines a 1-1 correspondence $sn: AccS(p) \rightarrow sn(AccS(p))$. This map is by definition surjective; it is injective if two different symbols accessible via $p$ have different names. In this way the function =find-symbol= is free of ambiguity. It searches first in the present symbols, and finally in the external symbols of the used packages. It is not necessary to compute the list of shadowed symbols since a shadow/ing/ symbol, which is also a present symbol, is found first. The main purpose of the coherence rule is to prevent users from designing too contorted situations in name scopes, wich are frequently the source of subtle and difficult to find bugs. Several symbols with the same name may coexist in an environment. The most common way to access a symbol consists in using its /name/, but /different/ symbols may have the /same/ name. In fact, its symbol name is not sufficient to access a symbol, a package must also be specified. This can be the default package, in which case the direct notation without any colons is possible, or the package name can be explicitly used to prefix the symbol name using the one- or two-colon notation. Of course, the symbol so accessed must be defined in a /deterministic/ manner. The definition just given says that a /unique/ symbol of some name is accessible from some package. The main function in the packaging organization is a map: $$ \rho:P \rightarrow [St \rightarrow ((Sm\cup\mathtt{nil})\times Stt)] $$ where $Stt$ is a constant set of four symbols, the Lisp symbol =nil=, and three keywords, the possible statuses of an accessible symbol: $$ Stt = \{\mathtt{nil}, \mathtt{:internal}, \mathtt{:external}, \mathtt{:inherited}\} $$ Because of the assumed coherence of the package configuration, the function $\rho$, an avatar of the function =find-symbol= of a package $p$ and a string $st$, determines a /unique/ symbol named $st$ and accessible via $p$, and also its status. It is possible that the search fails, in which case the function returns the pair =(nil,nil)=. (More precisely =(values nil nil)=.) The second =nil= is a kind of a null status: "There is nothing accessible via $p$ under the name $st$." The relation $\rho(p)(s) = \mathtt{(find-symbol}\;p\;st\mathtt{)}$ is always satisfied. It is now relatively simple to understand the various Lisp functions for handling packages and symbols, in particular what name conflicts might be generated. ** (intern $st\;p$) This function first evaluates =(find-symbol= $st\; p$=)=. If a symbol is found, /both/ values of =(find-symbol ...)= are returned. If no symbol is found, then Lisp /allocates/ (creates in memory) a new symbol without any value or function value. The =symbol-package= is initialized to $p$ and this symbol is added to $InS(p)$. In this case, =(values= $sm$ =nil)= is returned, the second value =nil= being a code for saying a new symbol has just been allocated. No name conflict can be created this way. This function is used implicitly by the Lisp reader when it encounters a symbol in an expression. If the colon syntax is not used, the default package is used for $p.$ Otherwise the (capitalized) package name of $p$ preceding the colon(s) is read, followed by the (capitalized) =symbol-name= $st$ after the colon(s). Finally, =(intern= $st\; p$=)= is evaluated, which yields the symbol so /read/ by the Lisp reader. In this way, the symbol can be either a pre-existing one or one freshly allocated as a consequence of this reading. ** (unintern $sm\;p$) A symbol $sm$ is /interned/ in a package $p$ if it is present in this package. It is /present/ in package $p$ if $sm\in PrS(p) = InS(p) \cup ExS(p).$ Uninterning a symbol asks for this membership to be "revoked." If $sm\notin PrS(p)$, nothing is changed and =nil= is returned. Otherwise $sm$ is removed from $InS(p)$ or $ExS(p)$, and also from $Shg(p)$ if $sm$ was shadowing with respect to $p$; in this case, =unintern= returns =t=. In the last case, when uninterning a shadowing symbol, a name conflict can be generated: it is possible for symbols $sm'\in ExS(q)$ and $sm''\in ExS(q')$ to have the same name while being /different/ symbols in the respective packages $q$ and $q'$ used by $p$, and that this collision was resolved by the symbol $sm$ shadowing both. If this happens, the =unintern= request is rejected. Also, if a symbol $sm$ is uninterned from a package $p$ and $p$ is the home package of $sm$, then the home package of $sm$ is modified and becomes =nil=, that is, the symbol becomes homeless. This is the only case where the symbol itself is modified by =unintern=. The same symbol could be external in another package used by $p$, in which case the symbol /remains/ accessible via $p$. Otherwise, if the home package was =nil= or another package, it is not modified. ** (import $sm\; p$) This is a request for the symbol $sm$ to be added to $InS(p)$. Lisp first looks for a possible name conflict. This happens only if a /different/ symbol $sm'\neq sm$ with the same name ($sn(sm) = sn(sm')$) is already accessible via $p$. If so, the importation is rejected. If no name conflict is detected, Lisp proceeds as follows: If the symbol is already /present/, it is left /unchanged/ in $InS(p)$ or $ExS(p)$. Else, if the symbol is /inherited/ because of a package $q$ used by $p$, $ExS(q)$ is left unchanged and $sm$ is added to $InS(p)$ (not $ExS(p)$!). If $sm$ is not accessible via $p$, it is simply added to $InS(p)$. In the specific case of an imported homeless symbol, the =symbol-package= of the symbol is updated to $p$. This is the only case where a symbol object is modified by an importation. If the imported symbol is already owned by a package, whatever its home package, it is not modified. If the importation succeeds, the symbol will always be /present/ in the target package, even if it was previously accessible only via a used package. ** (export $sm\; p$) The terminology "export" is not really appropriate, "externalize" would be more precise. The user of an =export= expression asks Lisp to record the symbol $sm$ in $ExS(p).$ This is allowed only if $sm$ is accessible from $p.$ In other words, the =export= function cannot /add/ an accessible symbol to $p.$ In view of the rule $InS(p)\cap ExS(p) = \emptyset$, if present in $InS(p)$, the symbol $sm$ is also erased from that list. If $sm$ is external in a package $q$ used by $p$, its status in $q$ remains unchanged, but $sm$ is nevertheless added to $ExS(p)$. Remember that the =use-package= relation is not transitive. The fact that $sm\in ExS(q)$ makes $sm$ accessible via $p$ using $q$ but not via $p'$ using $p$. If it is desired that $sm$ be accessible via $p'$, it is possible to make it external in $p$ by an =export= expression. Another solution would be to import $sm$ in $p'$, an operation which doesn't involve $p$. After exporting it, the symbol $sm$ becomes accessible from any package $q$ using $p$, that is, if $p\in U(q)$. This can generate a name conflict with a possible /different/ symbol $sm'$ of the same name, which is present in $q$ or accessible via $q$ in another package $p'\in U(p)$, that is, if $sm'\in ExS(p')$. If this happens, the /export/ request is rejected. ** (unexport $sm\; p$) The symbol $sm$ must be accesible via the package $p.$ If the status of $sm$ is /external/, it is downgraded to /internal/. Otherwise it remains unchanged, even if it is /inherited/. ** (shadow {$sm$ or $st$} $p$) It is intended to /import/ the symbol $sm$ in $p$, to record it in the shadowing list $Shg(p)$ of the package $p$, and also to /resolve/ any name conflict possibly raised by the importation. This time the first argument must be a /symbol/, a pre-existing one or a symbol just allocated because of its presence in the source statement. The symbol is imported (See the function =import=.) and added to the shadowing list $Shg(p)$ of $p$ if not yet present in this list. This addition to the shadowing list solves the possible name conflicts with symbols inherited from packages used by $p.$ It is possible that a /different/ symbol /sm'/ with the same name was /present/ in $p$ beforehand. If so, such a symbol is /uninterned/ from $p$, that is, erased from the lists $InS(p)$, $ExS(p)$ and $Shg(p)$. The difference between =shadow= and =shadowing-import= is clarified by the following toy example. We start a new Lisp session, allocate a package ="P"= and experiment with symbols named ="A"= and ="B"= in the packages ="CL-USER"= and ="P"=. #+BEGIN_SRC lisp :session pcl :package "P1" :exports both :results value verbatim (in-package "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (defpackage "P" (:use :cl)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (setf my-list '(a b p::a p::b)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (shadow 'p::a "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (shadowing-import 'p::b "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (package-shadowing-symbols "CL-USER") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (eq 'a 'p::a) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (eq 'b 'p::b) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim my-list #+END_SRC If we =shadow= the symbol =p::a= in ="CL-USER"=, the symbol =a= remains different from the symbol =p::a=. In contrast, if we =shadowing-import= the symbol =p::b= in ="CL-USER"=, then the symbols =b= and =p::b= are equal. In a sense, the expression =(shadow ’p::a "CL-USER")= is cheating because the presence of ='p::'= has no role whatsoever, for only the name of the symbol =p::a= is used and this name is the same as the name ="A"= of =a=, that is, because of the default package, =cl-user::a=. The symbol defined by ="A"= via the package ="CL-USER"= remains unchanged, in particular, it remains different from the symbol =p::a=. In contrast, if =shadowing-import= is used, an =import= operation is induced, so that the symbol =p::b= becomes directly accessible via ="CL-USER"=. The previous symbol $\rho:($="CL-USER"=$)($="B"=$)$ has therefore been /uninterned/ and is now unreachable by =find-symbol=. However, it is still visible in the list named =list= as the homeless symbol =#:B=. The conclusion of our test is this: - The symbol =cl-user::a= is unchanged but added to $ShgS($ ="USER"= $)$. - The symbol =cl-user::b= is almost unchanged, but no longer accessible via ="CL-USER"=. The only change is its =symbol-package= becomes =nil=, because the symbol has been uninterned from its home-package. - The symbol =p::a= remains unchanged; only its name has been used for something which does not really concern it. - The symbol =p::b= remains unchanged, except that it is now directly accessible via ="CL-USER"= and now printed as =B= in the value of =list=. ** (use-package $q$ $p$) Unless $q$ is already in $U(p)$, the package $q$ is added to the =package-use-list= of $p.$ Lisp checks beforehand for possible name conflicts. A symbol $sm \in PrS(p)$ could collide with a /different/ symbol $sm'\in ExS(q)$ if they have the same name $sn(sm) =sn(sm')$. If $sm = sm'$, no name conflict is generated. One cannot resolve such a name conflict by adding $sm'$ to $ShgS(q)$, hoping to give the precedence to the new inherited symbol: shadowing operates in a uni-directional manner, from a using package toward a used package. If one does so by a =shadowing-import= of $sm'$ into $p$, the symbol $sm$ would be coherently uninterned, erased from $PrS(p)$, and in a sense replaced by $sm'$. One can prepare the precedence of $sm$ over $sm'$ before the =use-package= by previously =shadow=ing $sm$ in $p.$ Also Lisp on its own can propose to /unintern/ the symbol $sm$ in a /continuable/ error in order to resolve such an observed name conflict. Another type of name conflict could be raised if another package $q'$ is already used by $p.$ It is then possible that two /different/ symbols $sm \neq sm'$ with the same name $st$ are external, respectively in $q$ and $q'$, in which case $\rho(p)(st)$ is ambiguous. If so, evaluating the =use-package= expression would generate an error. On the other hand, the intersection $ExS(q) \cap ExS(q')$ may be non-empty: a symbol external in both packages does not generate any ambiguity via $p$. As usual, it is always important to remember the lists such as $ExS(q)$ and $ExS(q')$ are lists of /symbols/, not strings. A name conflict can be generated only by /different/ symbols having the same =symbol-name=. If no name conflict is detected, the package $q$ is added to $U(p)$ so that the new set of accessible symbols via $p$ is: $$ \mathrm{new-}AccS(p) = \mathrm{old-}AccS(p) \cup (ExS(q) − ShdS(q, p)) $$ ** (unuse-package $q$ $p$) If the package $q$ is in $U(p)$, it is removed from this list. ** A bug in package-internal-symbols? While preparing these notes, the author thought for a moment to have encountered a mysterious bug. The minimal model for this bug goes as follows: #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (unuse-package "P2" "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (unuse-package "P1" "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (delete-package "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (delete-package "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (defpackage "P1" (:use :cl)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (defpackage "P2" (:use :cl)) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (use-package "P2" "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (export 'p2::a "P2") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (import 'p2::a "P1") #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (package-internal-symbols "P1") #+END_SRC The point is that the standard Lisp function =do-symbols= runs through all the symbols /accessible/ via ="P1"=, but it does not check for possible duplicates. Really? It seems possible that a symbol can appear more than once in $AccS($="P1"=$)$!? This does not seem compatible with the exclusion of duplicates in general in $InS(p)$ and $ExS(p)$, and also the rule $InS(p) \cap ExS(p) = \emptyset$. Yes, but a – one !– symbol can be simultaneously in $InS(p)$ and $ExS(q)$ for a package $q$ used by $p$, which is the case here for our symbol =p1::a= = =p2:a=. The evaluation of =(do-symbols (s p) ...)= makes =s= run through all the internal symbols of $p$, then through all the external symbols of $p$, and finally through all the external symbols of $q$ for every package $q$ used by $p$. It happens that our symbol is in $InS($="P1"=$)$ because it was imported there, and also in $ExS($="P2"=$)$ where it was initially defined. Didn't the programmer of =package-internal-symbols= check the status of the symbol, so that the (fake) second symbol should have been rejected? No, not really, and it is interesting to see the path followed by Lisp: only the symbol-name and the examined package are used, so that when Lisp computes =(find-symbol (symbol-name s) package)=, it obtains /the/ symbol, in fact the only one in this story, together with its status with respect to the =package= argument. In our case that is the package ="P1"= and the symbol's status (with respect to ="P1"=) is =:internal=, even if at the same time $s \in ExS($ ="P2"= $)$. This illustrates some of the hidden traps and pitfalls that one might encounter as a result of different packages seeing the same symbol in various ways. Remember: no name conflict can be caused by one symbol alone. A name conflict happens only if /two/ different symbols with the same name are "visible" by the same package. Nothing of this sort happened here, where /one/ symbol is visible by several packages. To overcome this weakness in the implementation of =package-internal-symbols=, using the Lisp function =delete-duplicates= is sufficient. #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (defun package-internal-symbols-dedup (package) (let ((rslt)) (do-symbols (s package) (when (eq (second (multiple-value-list (find-symbol (symbol-name s) package))) :internal) (push s rslt))) (delete-duplicates rslt))) #+END_SRC #+BEGIN_SRC lisp :session pcl :exports both :results value verbatim (package-internal-symbols-dedup "P1") #+END_SRC ** Symbol-package revisited. Throughout this tutorial, we kept insisting that this function is not really interesting. To be complete, we prove that a symbol $sm \in PrS($ =(symbol-package = $sm$ =)= $).$ In other words, if the symbol -package of the symbol $sm$ is $p$, then $sm$ is present in $p$, either internal or external. When a symbol is allocated (necessarily via =intern=), it is in particular interned in the package which is the second argument of =intern= so that it becomes a member of $InS(p)$. It may be exported in this package, which would make it a member of $ExS(p)$, but it remains present. The symbol can be added to $InS(q)$ for other packages $q$ via =import= or =shadowing-import=. It can also then be "upgraded" to $ExS(q)$ in those packages, but the membership in the initial package $PrS(p)$ remains unaffected. The only way to change this situation is using =unintern= with the package argument being $p.$ Then the symbol becomes homeless. In turn, this can be changed with =import= and =shadowing-import=, in which case the symbol is interned in a possibly new package, and the story continues along similar lines. Finally, unless a symbol is homeless, it is always present in its home-package. This also shows that, as was mentioned in the introduction, the =symbol-package= qualifier of a symbol does divide the set of /interned/ symbols into packages. It is impossible to have two different interned symbols with the same symbol-name and also the same symbol-package, for they would be interned in the same package, with the same name, and then the set of packages would not be /coherent/. An interned symbol is internal or external in its symbol-package, and certainly visible via this package. However, it may be visible from many other packages, and this information is not stored in the internal representation of the symbol. If it is external, it is accessible via the packages which use its package. The list of these packages can be determined by the standard Lisp function =package-used-by-list=. But a symbol can also be imported by other packages, which in turn may decide to make (the pointer to) this symbol external, and so visible from other packages. *Exercise:* Try to define a function =symbol-accessible-from= with one argument, a symbol, that returns the list of all packages where this symbol is accessible. Do not forget the contorted particular case of =common-lisp:nil=! Hint: Have a look at the standard Lisp function [[http://clhs.lisp.se/Body/f_list_a.htm#list-all-packages][=list-all-packages=]]. An arbitrary number of homeless symbols may have the same name, as illustrated by: #+BEGIN_SRC lisp :session pcl :exports both :results output (eq (print (make-symbol "Z")) (print (make-symbol "Z"))) #+END_SRC When =make-symbol= is called, Lisp allocates a /new/ homeless symbol with the prescribed name, even if another one with the same name has been already allocated. You see /both/ homeless symbols allocated above are printed in the same way, yet they are /different/. ** Practical uses of the packaging system. Many of the somewhat esoteric technicalities detailed in this tutorial have no direct use in concrete programming. On the other hand, they can help to understand and demystify the behaviour of your Lisp system. A particularly irritating fact exhibited by some systems is the following: When you merely install, not load, a source file in the editor of your Lisp IDE, these systems immediately scan for symbols appearing in these files and intern them in a separate package. Sometimes this is the package indicated in an inital =(in-package "XXX")= of the file, or still more inappropriately, the standard ="CL-USER"= package, or also the current default package, even if the initial =in-package= references another package. Often this later leads to symbol collisions which are sometimes hard to identify. A developer who meets this kind of mysterious symbol collisions is advised to search first in this direction. In this case, you should make independent micro-experiments to study the actual behavior of your IDE when you install a file in your development environment. Probably in the early days of Lisp, this was done to improve the efficiency, but on modern systems, it is difficult to see compelling reasons to justify such a strategy. Most symbols of a Lisp file in fact are symbols with lexical scope, a scope limited to a small part of the code, typically /inside/ a function, a =let= form, a =do= form and so on. These references are converted by the compiler into raw machine addresses, and these symbols have no real life at runtime. The most important symbols that have a real life at runtime are those that name functions, those defined through a =defun= form and the like. If a large program is divided in several files, then some decisions must be taken about the status of such a symbol. Usually, a correspondence "file $\leftrightarrow$ package" is maintained. Several Lisp systems require that a file begins with an appropriate =(in-package ...)= expression, declaring that all the symbols used in this file are to be understood as present in this package, except when something else is a consequence of decisions taken outside this file or even in this file. In the ANSI definition of [[http://clhs.lisp.se/Body/m_in_pkg.htm][=in-package=]], the referenced package must already exist in the environment. Beginning a file by: #+BEGIN_EXAMPLE (defpackage "MY-PACKAGE") (in-package "MY-PACKAGE") #+END_EXAMPLE is tempting but does not obey the requirement about the first instruction of a file being the appropriate =(in-package ...)=. The solution is to have a separate file in the project which is called aptly =packages.lisp= and which contains all the necessary =(defpackage ...)= expressions for the project. The =defpackage= form is flexible and can contain also all the main information about a package, in particular the list of its external symbols and its used packages. (See the ANSI definition of [[http://clhs.lisp.se/Body/m_defpkg.htm#defpackage][=defpackage=]].) This file is a good foundation for structuring the "public" documentation, explaining the role of these packages and their external symbols. Along the same lines, it might be worthwhile to have an ="EXTERNAL"= package which contains all the external symbols of the application. In this way, the other packages may use this package systematically, giving to this package a status similar to the status of the ="COMMON-LISP"= package. This does not prevent another package to import some symbols of other packages, symbols whose role is too specific to justify to make them "globally" external. ** Shortcuts. There are several ways for the developer to shorten the Lisp expressions dealing with the packaging system. To be rigorous in our explanations and close to the deep structure of Lisp, in this tutorial, we used a string identifier everywhere for a package. A symbol could be used instead, its name then being used as the intended string. For example, =(defpackage :my-package)= is equivalent to =(defpackage "MY-PACKAGE")=, for the name of the keyword =:my-package= is used, and because of the capitalization rule of the symbols applied by the Lisp reader. The name of =:my-package= is ="MY-PACKAGE"=, which through the =defpackage= expression becomes a package-name by following the standard rule which asks for an uppercase name. Also, each time a package argument is needed, frequently the second one, it can be omitted if this package is the current default package, which justifies this terminology. When using these shortcuts, do not forget about maintaining maximum readability of the source code!